Lets entertain his logic for a moment.
On some day at some time this man clearly said, if you are homosexual you are not welcome. I think we can all agree that this was the message.
I take it as you did not list any other things he said were out, he is okay with people who abuse drugs, either legal or illegal, he also says nothing about men who beat their wives or children and no to mention child abusers or people who lie or steal or kill or rape or paedophiles or are unfaithful to their wives/husbands.
Clearly all of those must be okay in the eyes of his faith as this pastor has no history of rejecting any of those sortss of people. Ask him to share a list of such people he has outed to is church or maybe can he share the preaching he has done in the past condeming those people. .
Therefore you have a dilemma.
You can refuse to join a place of worship that supports the about people or you can join him and support them.
You see this is the flaw in all of these religions, not a single one of them runs around condemning men who rape their daughters for example, but two mature aged men who love each other, they cant come.
Me personally I long ago decided I would rather enjoy the company of homosexuals and lesbians than shre my time with paedophiles and drug abusers etc.