You might be gay or bi - you aren't sure yourself yet. if you are those feelings were always there, even if you hid them from yourself. It is also possible you just like sex with males - but are romantically and physically attracted to women as well - in which case you are marginally bisexual, or even straight (yes I know, people who know less of the world or choose to know less will insist I'm crazy, but I'm not). Be honest with yourself. You said you can't lose him again. Is that because you can't imagine being without him again? If it is, then you need to ask yourself whether or not you are in love with him. If you are, then the healthiest thing you can do is accept yourself and love him back. You will be happier. EVEN if you don't feel like you will because of the "people" who will "reject" you - you really will. On the other hand if he's just a friend and its just the sex, that's ok too.
What may not be ok is the pot. I know, its legal in many cases now - but you both need to be able to feel what you feel and love what you love WITHOUT any mind altering substances - alcohol, pot, whatever. THAT TOO will ultimately make you happier.