Hunting animals that stand out from the crowd because of their impressive horns or lustrous manes could lead to extinction, according to a study.
Research predicts that removing even 5% of high-quality males risks wiping out the entire population, for species under stress in a changing world.
Animals prized by trophy hunters for their horns, antlers or tusks usually have the best genes, say UK scientists.
Removing these could push a species over the edge, they warn.
There is intense global debate over trophy hunting. Some argue that it should be banned or restricted, while others say it can provide valuable revenue for conservation.
Dr Rob Knell of Queen Mary, University of London, who led the research, said the assumption that so-called selective harvesting is not especially threatening to a population of animals does not take into account recent work.
''Because these high-quality males with large secondary sexual traits tend to father a high proportion of the offspring, their 'good genes' can spread rapidly, so populations of strongly sexually selected animals can adapt quickly to new environments,'' he said.
''Removing these males reverses this effect and could have serious and unintended consequences.''
Human hunting is different from natural predation in that big-game trophy hunters target large animals, usually males.
They may be awarded prizes for killing animals with exceptionally large antlers, horns or manes.
And illegal poaching of animals such as elephants for the ivory trade also targets animals with the biggest tusks.
Using a computer simulation model, the scientists were able to predict the impact of selectively targeting males on the basis of their secondary sexual traits.
''If the population is having to adapt to a new environment and you remove even a small proportion of these high quality males, you could drive it to extinction,'' said Dr Knell.
''You're removing the genes from the population that would otherwise allow the population to adapt.''
In the past, human hunting has led to the extinction of many animals, from the zebra-like Quagga, which was once common in Southern Africa, to the Tasmanian tiger of mainland Australia and Tasmania.
Hunting is still legal in many countries; trophy hunting takes place over a larger area in Sub-Saharan Africa than is conserved in national parks.
In the US and Canada, there is also a lucrative trophy hunting industry, for the likes of deer and big-horn sheep.
Some argue that revenue from trophy hunting can support conservation efforts and local livelihoods.
The scientists said age restrictions that allow males to breed before being removed could reduce the impact of trophy hunting.
This is already recommended with some species, such as lions.
"When properly regulated trophy hunting can be a powerful force for conservation which is why we're suggesting a different management approach as opposed to calling for a ban," said Dr Knell.
The study is published in Proceedings of the Royal Society B: Biological Sciences.
Follow Helen on Twitter.