Right now, your milk should make up the majority of your baby's diet. It's not unusual for older breastfed babies to prefer waiting for mom, and take just enough milk to get by until they can breastfeed. Is your son getting plenty of opportunity to nurse when you're together? Some breastfed babies of working moms will "reverse cycle" - do most of their feeding during the evenings and at night, and eating very little during the day. You might want to encourage your baby to nurse more when you're together by waking him to nurse when you get up in the morning and then nursing again when you drop him off at Grandma's. You can also wear him in a front carrier or sling so he can nurse freely when you're home during the evening. And consider sleeping with your baby so he can nurse at night frequently.
It's normal for weight gain to slow down in an older baby. This may not be a problem at all. By 6 months, he's only likely to gain 2-4 ounces a week (and it's not abnormal for weight gain to slow even more or stop for a short time). It's also normal for babies to drop % lines on the growth charts. If your doctor is using the CDC growth charts, they aren't particularly accurate for an exclusively breastfed baby. Plotting his weight on the WHO growth charts (which use the normal growth of exclusive breastfeeding for 6 months, followed by continued breastfeeding and a gradual introduction of solids) might be reassuring. Is your baby becoming more mobile? If he's crawling or cruising, his weight gain might have slowed because he's burning more calories than he was a couple of months ago.
Finally, solid foods are almost always lower in calories and nutrients than breast milk. And adding solids to his diet is likely to reduce the amount of milk he receives, so he might end up getting even fewer calories. If he refuses milk from Grandma, you could have her offer solid foods instead. But you don't want to replace his nursing with solids when you're with him!
How is Grandma feeding him? He might be unhappy because he feels like he's got no control of the process. When bottle feeding, he has to use his jaws and tongue to stop the flow of milk so he can swallow. If he feels like he's being flooded with milk, he might refuse to eat simply to protect himself. Feeding with a slow-flow nipple, feeding more upright, and following his cues when to slow the milk flow all might help. Or try using a soft-spouted sippy cup instead so he can feed himself.