Sigh... I just went through this very thing with one of my best girlfriends. I expect some gals will yell at you, but I'm not going to. I'm going to say, "keep your hat on, girl. Old boyfriends are just delicious memories - not reality. Focus on the reality of what you're proposing."
Like anger is a warning that something is wrong, but not necessarily a mandate to act out angrily, feelings for another man are definitely a clue that something's amiss in your relationship, but not necessarily a mandate to act on your longings.
First, you have three children whose lives will be forever changed by your decision. The minute you had kids, like it or not, you put their needs right up there with your own, and sometimes ahead of your needs. Act as much on their behalf as on your own. On to the question at hand...
You've been with someone for 6 years. You've heard of the 7 year itch. A lot of relationships get to a point of boredom. It's up to you to talk to your man about how the two of you can put a little romance and fire back into the relationship. You owe it to your man, your children, and yes, to yourself, to give it a shot. You have no idea what would happen if you tried to get back together with someone for whom you have feelings, memories, but no real knowledge. By contrast, you said you have kids with your current man, that he's fathering not only the baby you had together, but the two children you brought with you, and that you're still friends! You didn't say your current man abuses you or that you have been fighting nonstop for six years, or that there are any serious reasons why you should get out of the relationship. Before you take a huge gamble with everyone's lives, you owe it to all of you to try to fix your own relationship.
And what about those old feelings, anyway? Well, every woman has an old flame, old feelings, and maybe even some remorse. That's a nice treasure in your memory bank. But all that good feeling does not add up to a successful relationship. It can add up to a lot of fantasizing when there's a void in your current relationship though. Please try to remember it's all fantasy - nothing is as perfect as fantasy, and nothing is as difficult as reality. Please try your best to fix your current relationship before you jump ship. You really don't want to look back to this point five or six years from now, and hate yourself for what you've done.