The basic point is this: legacy admissions are much, much more pernicious than almost any affirmative action policy, even on the assumptions of anti-affirmative action supporters (assuming, at least, that anti-affirmative-action supporters are motivated by better motives than "GO WHITE GUYS", which I assume most are). If you focus your attention on attacking affirmative action policies without attacking legacy admissions more, then you are in all likelihood a racist, ignorant, or stupid.
(Which isn't to say if you oppose both you have any of those qualities. I probably still disagree with you: I think there may well be a role for legacy admissions and affirmative action policies in a high-quality university. But disagreeing with me on that is totally fine (at least in the sense that I won't consider you a horrible person if you do so). Disagreeing with me about being cool with legacy admissions but not cool with affirmative action means, well, you very probably fit one of the three perjoratives I gave above).