Most couples recognize the fact that one was without their family one year and that it is their turn the next year. It sounds like he has settled into this routine and now just relies on the fact that you go there every year. You need to put your foot down on this one, because if you wait a few more years to get your family into the rotation, he will be less and less likely to go willingly. You should sit him down and explain to him, that his feelings of wanting to go to his family's home instead, is what you have been living for 4 years. Only in your case you haven't gotten to be with yours at all. You need to tell him that the only fair way to do this is to alternate Christmases. Even if you 2 did his family last year, and do something just the 2 of you this year, that next year is for your family, then his etc...... You just need to remember to show him he wouldn't like it if he didnt see his family on Christmas for 4 years.
and If you rotate thanksgiving too, you can have Thanksgiving at one, then Christmas at the other, then the next year you switch the holidays, so you both get to see your family during the holidays.
Good luck! Be strong and dont give in! You and your family are just as important as his.