I don't understand what seems to be an all or nothing attitude. Your choice sounds like I either see more of him or not see him at all. Actually this sounds like a good arrangement since you know it's better to wait to start dating until more time has passed. By not spending lots of time with him you can work at keeping this as a friendship.
I suggest that you make better boundaries. Stop with the middle of night visits. I suggest doing a good job at work Is much more important than a sexual relationship. You want more companionship so set it up for that. Ihaven't talked with him about finding a way to do that, do so now.
I've dated many men over the years and worked in a predominantly male profession. I suggest that the reason he's busy or has other plans during the day or early evening may be that he wants a predominantly sexual relationship. I suggest that he could make more time for you if he had an incentive to do so. When you go along with seeing him mostly in the middle of the night he is telling you sex is more important than other activities.
Or it may be that he hasn't thought this thru. I suggest you take the initiative to start a conversation about meeting each other's needs. Tell him you want more companionship and plan together to make that happen. If he's not willing to find a way to do that, I suggest that being fun to be with him is not enough. You want a more balanced relationship and that's good.
I have dated men with children although none had primary custody. Do his children live full time with him? If so, when do they have time with their mom? Could he have someone, such as parents, sibs, or male friends who have their children some of the time? If you don't have the same days off, could one of you change days off? Sounds like he works an evening shift from which he's free during the middle of the night? Or does he have other plans during the evening on his days off?
I suggest that you have at least the "weekend" days to be together in the evenings. Either.one of you could have a babysitter. Or perhaps the espouses would be willing to have the kids an extra night once in awhile.
If you're expecting to see him more often than once a week it is unreasonable. When I was single and dated single men without children we mostly only got together an average of once a week until we decided to be exclusive. Even then it wasn't much more than twice a week unless we worked the same shift with the same days off.
I suggest that you are in charge of your life. It's your responsibility to find a way that is satisfactory for you. You can say no to the nights starting at midnight and ask for more daytime/evening time. It may be that you cannot change this pattern. In which case you have to decide to either accept it without hard feelings or move on.
I did accept a couple of my relationships as they were but continued to want more and they ended because I became resentful. Not worth it.
I, again, suggest that getting enough sleep is of high priority because of your job and your children. I've gone without sleep so that I could be with a boyfriend and it took a toll on my work and the rest of my personal life.