If he isn't sitting straight or close to it, then he needs to work on his core muscles. How is his crawling? If he isn't crawling, then he needs to find another way to build them. You should look at how he holds his body. If he has weak muscles on one side, then he may need physical therapy. I don't know if this has anything to do with Ergo or not, my babies didn't want to be worn, but, I do know that one weak side in toddler-hood can have lasting effects for life.
I personally would take him to the doctor for it. If the doctor has ANY concern, or says there is something, but let's wait and see, then I would recommend going to a physical therapist for an evaluation. Remember, they specialize in muscle movements and the doctor doesn't. The doctor should be willing to give you a referral if you need one. But, if not, I believe that everyone should be able to have an evaluation from a PT without a referral, but I guess it is up to the insurance company.
At this age Early Intervention can help as well, just call your local school district, but they usually start with the doctor I think.
If you are concerned, then don't let it pass. I say this from experience. My son was born with Torticullis, (weak neck muscles on one side), and Scoliosis. The doctors and physical therapists explained a lot to me about his muscles, and the one big thing they said is if the two sides aren't equal the person will favor one, and the longer they do it, the worse it will be. You can see adults who just look kind of "off"....and this is one of the main reasons why: Imbalance in muscle development.
Good luck with this and I hope it goes away very soon all on its own.