Wow it sucks to be single! Especially with a child...there are so many creeps out there and my response will warn you of a couple of types.
Ok, my first impression was for you to "go for it" but before I do that, I have to ask a serious question. Can you contact this guy any time day or night? When I re-read your post and see that he can't stay out late, and that he works a lot, and that he has only seen you a couple of times, i have to ask if you are sure he is not married. If I were you, I would check this out first and foremost and please tell me you haven't slept with him!
If you determine that he is not married, and that he indeed wants to spend time with you whenever you are available, then here is my advice...
Just do it! (but not "it")
I think it is great to take things slow. The crazy schedule thing may be God's way of slowing you down and being able to have you keep this guy if he is worth it. The very first thing to remember is to wait as long as possible before beginning a very physical relationship...even though you have been talking a lot and may feel you know him pretty well, just hold off!
That said, your little girl is very young and very resilient at this age. I think you can leave her three nights a week with your parents and see this guy at least once a week to begin with. She will have a balance of love from you, your parents and her father. I know this is an odd way of looking at it, but you are potentially planning for her future...a future living in a loving family will be so much better for you all, and would ultimately increase the time you would be able to spend with her (if you married and were able to stay home for example). I am not saying that you should not have an education or career for yourself, or that you should marry to get out of a situation, I am saying that I know how hard it is to be the sole bread winner and parent of a child, just scraping by!
On a more sour note (but I used to be a federal agent and know what I am talking about...as other posters mentioned there are pedafiles out there who get involved with women with children so they can get at the children later. NEVER become too complacent about this and NEVER make the mistake of thinking it can't happen to you or your child. Whenever you get involved with a man, always monitor your child's behavior around him and if you leave them alone together, make a habit of showing back up unannounced, earlier than expected. Sorry gals, it is a sad fact of life!