You are more likely to get hypothermia in cold water, which can lead to unconsciousness and then drowning or death. Also, even mild hypothermia can restrict your blood vessels and make it hard to swim and stay afloat. This can also lead to drowning.
Overall, yes it’s more dangerous, but it’s less dangerous if it’s in a pool or some other monitored body of water that is easy to get out of.
But if it’s in the ocean especially, or a dark lake, it’s very dangerous
However, I went scuba diving at Lake Phoenix which is pretty cold, but I was wearing a dry suit during the scuba part. They also had an obstacle course in the water, including inflatables and slides and things to climb and a body-rocket you jumped off, that I and my spouse did while wearing just a regular swimsuit. It was freezing of course, but they gave you life jackets, so we couldn’t really drown, and also it was daylight out so it was easy to see.
So sometimes it’s safe, but it can easily be dangerous
I do believe that diving in Coldwater can have long-term benefits to help your immune system. It probably hurts it in the short term but helps in the long term