

I think there are two more questions that need to be answered first, before being able to tell whether we should prioritize the many.
First question is what is the ultimate goal behind prioritizng the many? Happiness of the population? Infinite growth? To conquer the stars? Depending on what the goal is, there are occasions where minorities should be the focus if we want to approach the goal the fastest.
Example is moon landing: The amount of resources that was spent on “simply” building a rocket, space suits & equipment, and send a couple of humans over there was prioritizing the few. Despite a lot of people watching with curious eyes, it did not benefit the many’s needs much. There were several goals here: Being before the USSR, explore the unknown, satisfying shareholders, and more. By the many working hard to send the few, we approached all these goals faster than if we would allocate some of these resources towards the many’s needs, like health (prime days of smoking cigarettes).
The second question is what timeframe are we talking? Is it long-term or short-term success we’re aiming for? Because in many cases, if we want long-term success fast, the many are those who should “suffer”.
Example is where the long-term goal is the glorious evolution of mankind: In one way, we downprioritize the few who are those born with defects, either by culling them or by ensuring they do not make offspring. In another way, we downprioritize the many who are on- or below-average intelligence/capabilities. But then we get the question of how we quantify the few/many; where do we draw the line? And as we get more smart/capable humans, the average constantly shifts - what is the concrete goal?
Suffice to say that this is written without emotion, as that makes this discussion the soup it really is: Ethics, benevolence, discrimination, etc., as you mentioned.
Just to counter your arguments a bit: