The reason I think polygenicity is important in this case is that it means there is a huge mutational target that natural selection has to keep an eye on. The constant production of new mutations in sperm and egg cells, the fact that so many of them could affect intelligence, and the fact that they will tend to do so negatively, should, in my opinion, make it harder to push intelligence consistently upwards, when new mutations will constantly be pulling it back down.
Again, I would argue this is a different situation to many other traits. For any trait, new mutations are likely to degrade, rather than improve, the developmental program and biological pathways underlying it. But for some traits, the “goal” of that program is to hit a species-optimal set point. Mutations affecting that program could mean you miss high or miss low – there’s no reason to expect to go one way or the other, really (as far as I can see).
For intelligence, following my argument above, the goal is to hit the maximal level possible. New mutations will thus not just replenish genetic variation affecting the trait (in either direction, as in standard models of stabilising selection); they will tend to push it downwards.
Now, maybe someone will tell me why that actually doesn’t matter, but it seems to me that this will tend to oppose any efforts of directional selection to push intelligence upwards in any given population. Whether that is true or not (or the size of the effect it could have) may depend on how much the trait is dominated by the effects of rare mutations. Various lines of evidence suggest that the collective influence of such mutations on intelligence is very substantial.
That is from Kevin Mitchell. I do not feel qualified to judge his claims, but nonetheless found the discussion of interest.