Why is it rude to wear a hat at the dinner table?

I know that it IS rude, and I never do it. But i was talking about it with a friend the other day and neither of us could figure out where that gesture came from. For that matter, why does taking your hat of in front of a lady show respect?
14 answers 14