I don't want my son at his "grandmas"?

I don't want my son at his "grandma's" because i feel odd that she adopted her son (my baby's deadbeat sperm donor who is in prison) when he was a few months old so they aren't really my son's grandparents. Blood relation might not be important to them but it is to me. I feel like they are using my baby to pretend like he's theirs because they couldnt have one of their own. They also seem very awkward around kids and this woman is very unmotherly. Should i just keep my son with me because they make me really uneasy?