L.,
It is often that way when we lack for our family what is natural for us to give to others. I had more patience with other people's kids than I had with my own when I taught. Part of it was the fact that I was alot more invested in my own kids so things got to me more. The other part is that I had expended all the gifts God gave me on others and had nothing left for my own. So I quit working to stay home. That's not the answer for everyone, but I felt like it was a good one for me.
To find places where you connect with him, I would just sit and watch him. Find out what he is interested. What toys does he gravitate to the most. What t.v. shows does he watch the most. Does he like sports, fishing, reading. Does he like to draw? What if you decided to make a book of the stories you have told him? Maybe he would like to pick the ones to put in it. Maybe he would like to help you do the art work. I would find something you could work on and do just a little bit each day so that he had something to look forward to with you.
It's those little times and small things that he will remember. They are things you can do every day instead of on special occasions. Tell him everyday a specific thing that you think is cool about him. For instance, "I think it is so cool how you make friends. You are a really good friend and that makes me proud." "I love how your brain works. You are really creative and that makes me smile." "You have a really big heart. I see it in how much you love your dad. I love that about you." He may push it away because he is a boy and embarrassed, but he will hold those things so close to him. He will never forget that you said them.
Ask for his help with things. Let him know that you have confidence and that you couldn't do without him. Ask his opinion. Let him know that you value his thoughts on something. Leave him note on his pillow or in his backpack telling him how proud you are of him and that you want him to have a great day.
Pull him into you in small ways a little bit at a time. At first it might seem forced or awkward, but the more you do it the easier it will be. Before you know it, he will like how it makes him feel so much that he will be seeking you out.