Sometimes dystopia comes disguised as a consumer choice

Politics always thrives on scapegoats and villains, whom both Left and Right blame for our nagging sense of unfreedom. Often, however, our chains turn out to be freely chosen—or rather, the result of a million choices that add up to social conditions no one quite intended and few would choose. One of the most vivid illustrations of the dynamics by which an entire society may be ensnared by the rational choices of its members is provided by the 1997 film Gattaca. 

The movie tells the story of Vincent Freeman, a “faith child” in a world that, like our own society, no longer walks by faith, but by sight—specifically, by ubiquitous biometric surveillance. The surveillance begins before birth, as nearly all parents have their embryos screened for defects, and, if need be, enhanced, to ensure that they only give birth to genetically optimal offspring. The wealthiest families are generally able to ensure the best outcomes, raising children who are taller, stronger, smarter, and more beautiful—almost guaranteed social and professional success—while the poorest (or those with religious scruples) are left with “de-gene-rates,” subject to the full range of human frailty. The resulting stratification of society is reinforced at every stage by DNA testing to ensure that only the fittest advance into and among the elite.

The bitter irony of the film is that no one forced this dystopia on anyone—it arose through the free movements of the market, in response to an increasingly affordable new technology: one that individuals rationally adopted, until suddenly they were trapped and could no longer unchoose. 

It is this tendency that I have taken to calling the “Gattaca Effect”: a particularly pernicious form of collective action problem driven by the advantages conferred by new technologies.

Of course, on some level, almost every successful innovation tends to compel broad-based adoption. If a new technology confers a real competitive advantage, it will soon become the only viable way of operating in the industry: Lumber companies don’t really get to choose whether to use chainsaws or axes anymore. But the constraint is a narrow one, limited to a particular domain of economic activity in which few competing values are endangered by the push for efficiency. It does not reshape society as a whole, and ordinary people can get on just fine without a chainsaw if they’d rather.

The “Gattaca Effect,” by contrast, is a form of what philosopher Ivan Illich calls “radical monopoly.” In a regular monopoly, a single provider gains a stranglehold over the provision of a product; in a radical monopoly, that product or technology gains a stranglehold over society itself, such that it is difficult to live without it, and society is forced to reorganize around it. Electricity and automobiles have followed this trend, in America at least; more recently the smartphone, and perhaps soon the self-driving car. 

As this range of examples shows, radical monopoly is not necessarily a bad thing on balance. Most of us are quite happy with our ancestors’ choice to adopt and adapt to electricity. Few would want to go back to horse-drawn carriages. The smartphone is perhaps a closer case, on one hand delivering all manner of conveniences, communication modes, and knowledge, on the other hand creating “a mobile labor camp in which we voluntarily intern ourselves,” in Byung-Chul Han’s striking phrase.

If the full Gattaca Effect arises when individually rational decisions to optimize via a certain technology create a new equilibrium that restructures society in a way that is bad for most participants, but which they can no longer escape, how might this happen? Why would people choose such a thing? Two distinct mechanisms, which can have disastrous consequences when combined, are on clear display in the film.

The first mechanism is that of incommensurable goods and measurability bias. We will often make trade-offs sacrificing very real, fundamental human goods that are hard to quite get our hands around, in pursuit of more easily measurable goods. Human freedom is a fundamental good, but a devilishly hard one to quantify. The sense of being master of one’s own fate, of having responsibility for one’s own life, of taking joy in one’s achievements or blame for one’s failures—these are fundamental to our sense of well-being, but profoundly eroded in a world of genetic determinism. Why? Because the benefits—being healthier, wealthier, and wiser—are so much more vivid, salient, and measurable. 

In a wonderful scene, Vincent’s parents are meeting with the geneticist planning the conception of his younger brother Anton. When the geneticist describes all of the conditions he has screened for, including “premature baldness,” the parents interrupt, concerned: “We were wondering if we should leave some things to chance.” No, the geneticist admonishes, “You want to give your child the best possible start. Believe me, we have enough imperfections built in already. Your child doesn’t need any additional burdens.” What parent would want to burden their child? 

Nor does this measurability bias end with the initial decision. Once chosen, its logic infects everything. As the film goes on, we learn that companies, always looking for good proxies for talent rather than having to go through grueling interviews, have normalized screening applicants based on genome; police forces, trying to narrow down suspects, check all DNA in the vicinity of the crime for any “de-gene-rates.” In a tug-of-war between the measurable and the unmeasurable, the measurable always wins out.

The ratchets of responsibility norms go in only one direction. Vincent’s parents inhabit a world in which goods such as “leaving some things to chance” are still intelligible, known intuitively from personal experience. Once abandoned, though, they can be lost almost beyond recall, and it can seem almost unthinkable to sacrifice control for such sentimentality. Within the past generation, as the smartphone has made it possible to geolocate children at all times, parents have found themselves unable to leave anything about their teenagers’ movements to chance. Who would ever shut down a useful and safety-enhancing feature for the sake of some amorphous notion of autonomy, ensuring that what once might have seemed the unavoidable and not uncommon missteps of an adolescent will instead look to the community like reckless endangerment by an adult? 

There’s a second act to this tragedy: After we have too easily sacrificed the deeper, less quantifiable goods, we often find even the quantifiable ones elusive. Why? Because most of them are the result of positional competition. When Lance Armstrong decides to start doping in the Tour de France, he gains a meaningful advantage—for a while. Then others pile in. Once everyone starts doping, the advantages evaporate and everyone is back where they were—with their own integrity, and their sport’s, destroyed, and their risk of heart disease much elevated. The first parents to genetically optimize their offspring give that child a leg up. Once everyone has an IQ of 165, they are left fighting each other for the same positions as before. 

* * *

Similar dynamics are already playing out with AI. The first students who decide to sacrifice their integrity and their intellectual growth by using AI to cheat on their homework will in fact capture a GPA boost and perhaps, briefly, better college or employment prospects. But the advantage will disappear as soon as the behavior becomes the norm, leaving only the long-term harms, now more broadly shared. Many of AI’s otherwise legitimate uses face the same challenge. With the proliferation of personal AI agents, booking your favorite restaurant becomes easier than ever, until everyone deploys their own agents, and the restaurants invest in their own agents to deal with the flood of incoming requests. AI-generated résumés now flood AI-generated and -monitored job postings, yielding a dehumanizing and inefficient labor market.

Never before has a technology promised so many efficiency gains in so many domains, if only we consent to reordering our lives and social structures around it. Indeed, because AI capabilities are advancing most rapidly in verifiable, measurable skills, the technology will create an immense pressure to reorient products, processes, and workflows around quantitative metrics where AI excels, sacrificing in the process all those things that, in the words of the old Mastercard ad, “money can’t buy.”

Left to our own devices, most of us choose the easier way, the faster way, the way that confers some measurable advantage for ourselves or, as in Gattaca, for our children. When choosing otherwise also means falling behind, we quickly run out of people willing to do the harder thing for the lower payoff. Left to our own devices, we will fall victim to herd dynamics that may leave all of us wondering how we got there. 

But we do not need to leave ourselves to our own devices, either literally or figuratively. We have both the right and the responsibility to make collective decisions about our technological future through public policy: taking the time to decide whether and how we do want society reshaped before Gattaca Effects settle the matter for us. We need not be able to predict the future to discern the plausible effects of powerful AI on our common life, ask which possible futures represent improvements, and take action in the present to make those futures more likely.

Abraham Lincoln described the legitimate object of government as “to do for a community of people, whatever they need to have done, but can not do, at all, or can not, so well do, for themselves—in their separate, and individual capacities.” In an era of unprecedented technological evolution, individuals feel increasingly powerless to resist or even steer the immense forces reshaping their lives. It is the urgent task of our leaders to ensure that progress in processors and algorithms leads to progress in the human condition, rather than a world optimized for ends that none of us would choose. 

Brad Littlejohn
Brad Littlejohn is program director and policy advisor at American Compass.