Pretty well everything is toxic in excess, it is just a matter of what constitutes excess. Usually, we consider toxicity in terms of ingestion and inhalation when discussing metals, because they do not readily absorb through the skin (although some of us suffer dermatitis from contact with certain metals such as nickel in jewelry, it won't kill you so isn't "toxic" per se).
Concentration limits in soils or water are generally defined in terms of human sensitivity to the elements upon ingestion (the primary route to addition of metal to the bloodstream and the body system overall). Elements with low concentration limits would be mercury and selenium, say, whereas zinc and manganese would be examples of elements that we can tolerate at higher concentrations. Gold, of course, is very unreactive and is generally seen as a low toxicity metal, although even that idea has to be taken with caution (not true in detail, just true in general).
This is a gross generalization though. The form of the element matters. Cr+6 (hexavalent chromium) is a lot more dangerous to humans that chromium metal or trivalent chromium, as an example. Copper is generally seen as biologically more hazardous ("toxic") than lead, actually. Adding copper sulfate to a water bath will suppress bacterial or algal growth quite efficiently, much better than lead chloride salt addition would work. However, lead is pretty nasty to brain function so debilitates without killing. Which is worse?