I was never good at math in high school. Starting at Algebra II, I simply didn't understand it. Somehow I memorized enough to get through trig, and in college I got through calculus (with an A, even.) Now I'm homeschooling my daughter. She is struggling with Algebra I, but I find that math now comes very easily to me and it now makes sense. My husband was on the math team in high school and could have been a math major in college (he became a CPA instead), and he runs math problems in his head. He simply couldn't get down on our daughter's level to teach her what comes so easily, almost intuitively, to him. With her, we've switched to a tutor who uses a very different type of math program (Math-U-See) which is both tactile and visual, and my daughter is understanding math much better. I don't think she'll ever be a "natural" at the higher, more abstract maths, but I think she'll be able to do enough to get her through some basic college math.
To sum up what I'm trying to say is, 1) sometimes later in life you may understand and be able to do math problems that stump you now and 2) sometimes a different math curriculum or even a different teacher/tutor can help you have an "ah-ha!" moment when you're trying to grasp a new concept.
There is one thing for sure: Don't stop until you're completely sure you CAN'T go any farther. As long as you're making SOME progress, well, you're making some progress. I think that just the effort to understand math helps your brain make neural connections. Math is EXTREMELY developmental, and honestly some people are better "wired" to understand math at an earlier age than others. Still, while you may never be an "Olympian" at math, you probably could get good enough to pass Calculus in college.