Yes, it is. It may also get you fired. Why? Because you are clearly throwing gas on the fire. You aren't saying anything to diffuse the situation but instead are invalidating the customer's emotional state and their complaint. You are trivializing them and trying to make them seem small and petty for their reaction.
What's more, saying that is profoundly stupid, a completely illogical argument. How people usually find you has absolutely nothing to do with the situation at hand, making your assertion argumentatively what is called a "straw man." A "straw man" is a logically fallacious and distracting argument one puts forward that one can easily win and thus make it seem one has won the other argument that is actually being argued. Since the other person can't disprove how other people usually find you because they have no basis to argue against that, you easily win that argument, and in so doing, you are trying to translate that win into making it seem like you've won the unrelated but actual argument that is at hand when, in actuality, how people usually find you has absolutely nothing to do with the argument at hand.
Even if they're calling you "rude," if that's the argument, the fact that 20 other people didn't think you were rude to them yesterday or the day before or even earlier today doesn't mean you're not being rude now. Rather, since proof is in the pudding, this mouthiness, this sass you're giving them, actually proves their point that you are being rude and anything but likeable and are giving that customer no reason at all to be happy to see you.
You are not at work for your own emotional satisfaction. You are at work to serve the customer in front of you right now and create the most positive experience you can for that customer so that they become more likely to come back and become a regular customer. Saying to an angry customer, "Gee that's odd, most of the customers really like me a lot and are usually happy to see me," is doing the EXACT OPPOSITE of your job, what your employer is paying you to do. Your job is to diffuse that anger and turn it into a positive experience, not rebuff it with a bunch of lip.