If it's true, then no it is not wrong. If you don't get him out of your house, you risk your wife cheating on you. Not a good thing.
I'm a straight man, but I've never dated or married yet. If I had a wife, and I thought someone else living with me might have a crush on her, I'd kick him out and ask questions later. If I had a wife, and someone had the potential of stealing her away from me, I would become very jealous.