L.A.
My father passed away when I was 23. So that I still felt connected on Father's Day, our family would get together and talk about him. We would share special stories and it always ended up being very theraputic and memorable, full of some sadness and a lot of laughter.
Perhaps you can make a small appearance with your mom, but then go out and have a little remembrance party for his mom. Maybe make him a special photo in a frame or even an album of him and his mom together during his childhood. You can even ask her family members and friends who knew her when she was young to share some stories of her youth in letter form.