It's not about total population or how much the elite can earn. It begins with how many average people have access to sport infrastructure, trainers, good diets, school/college sports programs, local competition, etc.
Sure some American athletes can be poor by American standards, but it's still incomparable to those African countries.