I think you've got your answer...it's probably the right time to start sending a birthday card and politely declining; around this age, your daughter needs to be more involved with children her own age.
I've dealt with exactly the same thing with my daughter, whose "best friend" is the girl next door who is three years older than she. This girl has gigantic parties with TONS of children (family friends, etc) so my daughter usually ends up playing with some of the younger children who come, not even her friend.
Since your daughter doesn't have that option (no younger siblings to play with, etc) I think you should probably give a very honest and polite explanation why you aren't coming...because you're right, the birthday child should be allowed to do what they want that is appropriate for their own age. I believe it would be considerate of your friends to understand that!
Good luck!