It is a good question and one I have never seen satisfactorily answered.
People have validated the assumption by using computer models but that is not really good enough. Imagine if I thought my dice was loaded and always produced a six. Then, instead of doing a real test, I make a computer model that contains everything I believe. Then when you ask me about my dice I get you to run the model and see what it says. When it says "six" I have not really proved anything, have I?
We have a climate that we know has changed by several degrees in a decade in the past and now we are looking for smaller increases over a longer time.
We do know that there is some man-made warming because the scientists add it in the form of "adjustments". Strangely, the graph below no longer appears on the NOAA site. Perhaps it was too incriminating?
Also, in Australia, where some people keep an eye on what the official scientists are saying, they notice that certain places have long past temperatures that are now falling. How can that be?
If most corrections cool the past or warm the present then we have a source of anthropogenic global warming right there!