This matter could well be the exception to the general rule that "good fences make good neighbors".
A fence/wall resting directly on the property line should belong equally to both parties, regardless of who erected it initially. The status of those 3 inches in the middle depends on your local regulations. A local government official should be able to answer that.
As a property owner, my greater concern would be that this wall was initially constructed to be a breeze barrier, not a retaining wall. But by filling in his garden, he is using it as a retaining wall. The construction standard for a retaining wall should be quite a bit higher than that for a breeze barrier. I wonder if its up to local code. If not, then this wall could easily give way and his garden would then be in yours. And it would be his responsibility, entirely, to make the matter right. Perhaps an appropriate local official needs to be called out to inspect the wall for compliance with local codes. If it isn't sufficiently strong, then your neighbor could well be ordered to either replace the wall appropriately, and at his expense solely, or to restore the original grade of his garden. I doubt he'd be too happy with either choice. And how much might it cost him just to fight such an action?
Not that you necessarily must go through with all that. But if your neighbor wishes to be hard nosed, then you can play that game as well.
In the meantime, you should be able to paint or whatever on that portion of the wall that sits on the boundary, wherever the legal boundary turns out to actually be. And I certainly would see no problem with putting plantings on YOUR property. Should they grow over onto his, then he has every right to trim them back IF he can find a way to do so without coming onto your property.
Oh, and if you just want to get his goat, send him an invoice for the maintenance of those 3 inches for the past 37 years. While this wouldn't be seriously given or received, perhaps you could add a note on the invoice that you would be happy to sit down over a cup of tea and work this out agreeably and as neighbors. Maybe it would be just enough to break the stalemate.
Good luck.