Because you've proven no connection between the person harmed and the person paying restitution.
I'll grant that if I broke your window, I should pay you. But, if someone who looks a bit like me, broke the window of someone who looks a bit like you...there's no reason for me to pay you. You have to prove that *I* did something that caused *you* harm. So, you have to show three things: an act occured, you suffered, and I was the cause.
Your analogy only works if I accept at least one of the following is true:
1) All white people have oppressed all black people.
2) Guilt for crimes, and victimhood from crimes are passed down genetically.
I see no scientific or logical or moral reason to accept either.