J.M.
Oh I get that sometimes too. I think it's guilt : ) This is why I think that: Normally when my daughter went with someone else, even my mom, I was totally at the end of my rope, and thinking "if I don't get a break I am going to go insane!" and then as soon as she was gone I thought "she's so cute. I can't believe that I was so exasperated, what if something terrible happens and I never get to tell her that I love her again!" I sometimes feel the same way after they go to bed after a _long_day.
Don't really have any advice as to how to make it go away, but you're not alone.