My daughter was born 8 weeks early 8 years ago. If I remember correctly, you should gage their developmental milestones based on how old they would be from what their birth date should have been. So, if he was born 3 weeks early and he is 4 weeks old, he is still really only about 1 week old. So base his activities on that for the first 6 months to a year and if he seems to have great leaps of development than even better. Don't compare what your son is doing to any other babies his age. They are all different. My neice walked at 9 months, and my son walked at 13 months. They are both healthy, happy, normal(?) teenagers now.
Also, talk to your Pediatrician and/or their nurse. My old one gave me a developmental sheet at each checkup that discussed what the average range of activities and developmental skills should be at each age.
For you, relax. Have fun with your son, enjoy this time where he is completely dependant on you. Enjoy watching him sleep, marveling at every new experience he faces. Babies are smart and resilient. If he's hungry he'll eat; if he's tired, he'll sleep; and if he needs you he will let you know. Babies grow up way too fast. Good luck!!