Daddy fetish sounding thing, add in the always thought you were asexual thing, and flags go up.
Did you have a dad in your life? If not it could be nothing more then searching for one, that approval type thing.
If you do and the realtionship is decent enough, there could be some sort of abuse buried in your early years. Not by dad mind you, although they are often the first person thought of. Buried in your mind but at the same time, imprinted there forever. Or, maybe you just never meet anyone who actually appealed to you, and now you do.