I think it depends on the part of the building you are in (for example, I would try to dress business casual if you will be near your supervisors or higher-ups in the company), how long you will be there, etc. I would wear something casual but tasteful. I think you are basically correct, and I think in an ideal world, we should all be free to just wear our sweats around on our days off (seriously, I do feel that way!) BUT there are always hard-nosed people peeking around, so it is better to be safe than sorry. Wear something that is neat, not revealing, and just put on a casual hairstyle and a little makeup, whatever fits your style. I think the key is a PULLED TOGETHER look no matter how casual or dressy.