Damn, what is up with these questions, lately? Never enough information in the question to answer it. In general, if the bio dad wants to spend time with the son OUTSIDE of your home, then that should not be a problem for anybody.
HOWEVER, you are remarried now and both you AND YOUR NEW HUSBAND have a right to privacy in that marriage. Having the ex around can be a HUGE problem, as well as an invasion of your new husband's privacy.
So the only way it would be 100% OK and your husband would be wrong to object is:
If the ex arranges for his own transportation to/from wherever your son is going, and your son has OTHER transportation that does not involve your current husband.
If your ex expects to visit your place to pick up his son, that is WRONG. Your visitation agreement (while not addressing the therapist appointment specifically) SHOULD have specified a public place to meet (not at your house!) for the ex to pick up your son, for his scheduled visitation. That arrangement should be applied to non-scheduled visitation, also. Along with that, if your ex is transporting your son, he should not be dropping off your son at your place, either.
If YOU (yourself) are involved with transporting your ex, that is a HUGE NO-NO, as it is again a violation of your privacy as well as the privacy of your new husband.
Also, as the therapist appointment is not an official visitation, then your son should not be delayed (not even by 15 seconds) in getting to the therapist appointment or returning home from the therapist appointment due to the fact that his father is there. So, for example...bio dad goes to therapist appointment with son. Afterwards, he offers to take your son to lunch at TGIFriday's. WRONG!!!!!
What you need to get straight in your head is....
Aside from visitation rules spelled out in black and white on paper...
The activities of your ex should not affect YOU, your HUSBAND or your SON, to deviate from your normal schedule, at all. If these appointments are being used as an excuse to spend more time with your son OUTSIDE of the appointments,
Then your current husband is absolutely, 100% correct that it needs to be written into the visitation agreement.