“Representatives shall be apportioned among the several states according to their respective numbers, counting the whole number of persons in each state.”
It says “the whole number of persons.” Not citizens. Not voters. Not legal residents. Not white people. Not English speakers. Not Republicans. (Or Democrats.) Persons.
Because it deals with mathematics — though not with as much specificity as, say, setting the minimum age for people who can be president — this is one part of the U.S. Constitution that should be least susceptible to any interpretation that might benefit one faction over another.