Racist or not, Mailer is certainly part of the mindset that keeps the mainstream media -- whether it's journalism or the entertainment industry -- so lily white. If you're a person of color and get in, then of course it must be because you were some affirmative action case.
I've experienced this in my career, taking a job at a big newspaper and having most people assume I was an intern from the minority internship program. Even at other jobs, there is always a subtle undercurrent that the few non-white people on staff were there only because there was some mandate to hire minorities.
Sure, sometimes a minority candidate gets hired when they're not ready for the job. I've seen it. I've also seen a lot of white people who aren't qualified get hired as well. Having a diverse workforce is generally good, but it has to be done right, with good hiring decisions that don't fall into the affirmative action trap that Mailer is complaining about.