I thought you were a stay at home mom. You're not and so I've edited my post.
If you're asking if you have to do chores and tasks on the week end I say definitely not. When I worked I tried to leave one week end day free of chores and planned to do something fun. And now that I'm retired I nearly always leave the entire weekend free of chores so that I can spend the time with my grandkids. I don't even do the dishes.
I can see how, when you do that, your husband might think you do nothing or very little all week. I don't understand why he wouldn't want you to spend the time with him and the kids on the weekend rather than do chores. Perhaps I'm not understanding that part of your post, either.
I would ask him why he thinks you do little around the house when he's gone and why it's important to him. I suggest that there is an issue beneath it that needs to be addressed. If he's wanting the house to be neater or certain things done then he should be helping. Have you had an in depth calm and rational conversation about what he's thinking and how the two of you can resolve this issue?
You say "it seems like he doesn't think I do much around the house." Perhaps he doesn't mean to sound that way. Some people have a poor way of wording comments so that they sound critical without them meaning them to be.
Perhaps he doesn't not feel appreciated for the work he does? I think I'd try praising him more for what he does. I'd also make a comment from time to time about how you appreciate being able to stay home with the kids and how you know it's the pits to be gone all week. Tell him you miss him. Put the focus on the good things he does and get it off what you do.
And, with him, have an agreement to spend time together as a family when he is home. That way, he'll be more involved and have more of a stake in making happy memories. Perhaps he'll spend less time focused on what you do the rest of the week.
My work away from home daughter and her husband, who have 3 children and another on the way spend one weekend day doing chores all together and one week end day doing family activities.