You won't understand it when you're older
What is the human condition of automating our thinking?
I remember the COVID scenarios from the spring of 2020. The real outcome was always going to lie somewhere on a continuum, but the proposed scenarios came as discrete options: many die and the virus burns out; the virus keeps mutating, many still die and restrictions last for years; and finally the one that sounded least dramatic, that it is going to be tough for a while and then the virus lingers in society forever as something we have adapted to. The last one is (roughly) where we ended up. In the discussion about AI, I feel we are missing that option. The studies in which experts and superforecasters estimate the odds of AI destroying us1 age incredibly fast, and that consensus has shifted within the last months alone. We can and should act to steer away from the worst cases, but I read these forecasts mostly as signs of the times in which they were made, later to be studied by historians.
However, if this will be the case, AI will not linger at the edge of society the way a virus may. Instead, it might “disappear” into it. Mark Weiser opened his famous 1991 Scientific American article with “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”2, where his examples were writing and electric motors. We stop noticing such technologies because they are everywhere and our attention is limited. These technologies evolve from tools into infrastructure we depend on. AI is headed the same way, into the background of society, and possibly to become its dominant background.
Right now we are in the honeymoon phase (or, for some, the inverse honeymoon phase), in which we are supposed to get used to AI and somehow recalibrate our humanity around it. The natural question is what is society’s immune system “against” AI; institutions, norms, law (Press X to doubt), education? All of these presuppose people who understand what they are regulating, teaching or judging. Of everything that immunity has to protect, I feel understanding is the most exposed.
Not understanding is, in itself, ancient
Try this before continuing: explain, step by step, how a zipper works. In the study that coined the term illusion of explanatory depth, people rated how well they understood everyday devices like these, then tried to write the explanations, and their ratings dropped considerably3. Writing down an explanation is just one concrete way of finding out how little we really know. This is not something to be afraid of in itself, because it is ancient. Leonard Read’s 1958 essay I, Pencil is narrated by a pencil that claims “not a single person on the face of this earth knows how to make me”4, yet the pencil “appears” for the use of all of us, reliably and even these days, very cheaply.
Human intelligence has always been collective. We trust others to be able to explain things to us, and we do not sharply distinguish between the knowledge in our own heads and the knowledge in the community that we could go and ask for5. Where that community lives has shifted enormously, in the “first transition” with the internet and search engines and now with AI. In software not understanding has been the case for a long time. Large websites, large commercial systems, and software we put our lives upon, like the systems that fly aircraft, are most likely not understood as a whole by any single person. Note that I am not saying that is necessarily bad. Instead, we have accepted such a societal deficiency a long time ago, because it was confined to a small enough circle of things that we could still feel to be in control.
Not understanding something also doesn’t stop us from using it, but it may very well stop us from using it safely. For example, Aspirin was sold from 1899, and its mechanism was only described by John Vane in 19716. For decades we used a drug we couldn’t explain, at scale and mostly without regret. Sometimes this went badly as well; Tetraethyl lead was added to gasoline from the 1920s on, and a 2022 estimate puts the resulting loss from childhood exposure in the present-day US population alone at 824 million IQ points7. So why would AI be any different from these? I think for two reasons: there may be nobody left to ask, and there may be no time to catch up.
Who you gonna call?
Historically, it has been safe not to understand, because there was always someone to ask. For most of human history that someone was older. Margaret Mead turned this into a taxonomy of cultures by who learns from whom8 (one could make it continuous, as the share of society’s “workings” each of us learns from elders, peers or the young, but even in a discrete form it is useful):
| Culture | Who learns from whom | The future… |
|---|---|---|
| Postfigurative | the young from the elders | repeats the past |
| Cofigurative | everyone from their peers | is shared among contemporaries |
| Prefigurative | the elders from the young | is a world the elders never knew |
| ? | nobody from anybody | is understood by no one |
Mead argued that the technological surge after the Second World War had pushed the whole world into the prefigurative mode: “nowhere in the world are there elders who know what the children know”. This also partially catalyses the feeling that the young are somehow crooked: the more prefigurative the society is, the more elders are watching people live in a world they never knew. The old person with the smartphone is the prefigurative culture in miniature, and it is why I call them the good version: the phone is opaque to them, but their grandchild can still show them how it works.

I am afraid AI might launch us into the fourth row. The implicit assumption behind there being only three modes of figurativeness is that there is always somebody who understands, and that if you do not, you can go and ask. As AI enters its mass age and most things come to be done with it, the world as a whole may become opaque. Note, that the reason would not be just that people do not understand AI; most of us don’t already. Rather, it would be that people don’t understand the things that are built with AI. Black box models run by black box entities are something one can still choose not to use, but once society itself is built with them, we may end up not understanding the results either. This, I currently feel, is the primary threat AI poses to modern society. It is deceptively easy to accept the definitions and solutions AI gives you, because they seem to work and you feel you are getting things done, but if you don’t also get the understanding of why they work, you are treading on thin ice.
The shift, then, is from nobody understands the whole to nobody understands enough of single parts either. E. M. Forster wrote this down in 1909 in The Machine Stops, where humanity lives underground with every need served by the Machine: “The better a man knew his own duties upon it, the less he understood the duties of his neighbour, and in all the world there was not one who understood the monster as a whole. Those master brains had perished.”9 The maintenance of the machine survives, but understanding does not. Science has had small previews already decades ago, even without AI: for example, the four colour theorem was first proven by a computer brute forcing different configurations, and it was debated for years whether that counted as a proof at all10.
No time to catch up
Mead’s prefigurative world still had someone who understood, because the young could keep up. If the world changes faster than generations replace each other, even the young cannot, and everyone becomes the elder already within the years in which their own cohort is supposed to be the one running society. Consider a 25-year-old today. How long until they are as lost in a society run on mass AI as an average 75-year-old is today with a smartphone? 20 years? Ten? Hopefully never?
In 2016, AlphaGo’s 37th move in its second game against Lee Sedol was first taken by commentators for a mistake, simply because humans could not understand it; it turned out to be a game changer11. That single move could be studied afterwards and understood, and there is evidence that human Go got better after AlphaGo12. Thus, in the good case, incomprehension is a phase rather than a state, provided there is ample time to study. In the bad case changes of this kind happen in society so fast, that there will not be enough time to deeply comprehend them, and at some point we will simply give up.

The human condition and cost of the lack of understanding
From a human point of view, not understanding is just fine until you have to understand something you cannot, while knowing that you must. Consider living in a society, performing functions (what we would call working today), driven by forces that you cannot understand, put forward by arguments you cannot understand either. Such is a Kafkaesque The Trial at the level of the society, and in it you retain no control. In my eyes, the loss of understanding is fundamentally equivalent to the loss of control.
So far AI has mostly affected knowledge workers, because it is an immensely powerful brain handicapped by having no body, and knowledge work needs no hands or feet. That can change within years: AI labs have been hiring for robotics, humanoid robots have had a major push over the last two years, and there are already shops in China run by robot assistants13. I would be extremely surprised if robots powered by these models did not spread to basic physical jobs within the next ten years, and I do not think compute will stop them, since the best models keep getting distilled and quantized into ones that run fast on surprisingly little hardware. As soon as running a robot is cheaper than employing a human, the robot takes the place, because corporations (as well as other entities) are driven by profit; it is a lower energy configuration of a dynamical system, so we will keep taking humans out of the societal loop wherever we can. This has happened throughout the history of automation and is nothing new. There will probably be companies that make a point of still hiring humans, but I expect that to be a niche.
What automation has long done is remove the simple, repeatable tasks that teach you the basics of what you are doing, leaving the human to oversee the machine. Lisanne Bainbridge called this one of the ironies of automation: monitoring does not build skill, and yet the more advanced the system, the more crucial may be the contribution of the human operator when it fails14. Understanding is a maintenance cost, and when we automate to cut costs, we naturally cut that one too.
The loop also runs the other way. Do you already do things not because they are mandatory but because software demands them? Do you search-engine-optimize your online presence? Do you make your social media posts to get as many likes, reactions, and visibility? The technology that offers you the infrastructure to publish strongly encourages you to do so. The measure has become a target, since you wouldn’t need to do it otherwise1516.
The problem is that the systems we build this way also get more complex and more strongly coupled, like the financial markets, and often highly optimized, tightly coupled complex systems are also very fragile: small perturbations propagate far, as if the system had a long correlation length17. If AI turns out to be the ultimate complexity-adding solution for society, it can make society less stable as well, and a fragile system is exactly the one in which somebody eventually has to step in and understand it. When NASA engineers wanted to study the Saturn V’s F-1 engine again around 2012, they had to take a stored engine apart and scan it, because much of what it took to build one had lived in the heads of engineers long retired18. Luckily, that was one engine, with the hardware still sitting in a warehouse. Have we committed the current state of the societal repo?
It also is not just about losing your job, even though that is the primary concern of many authors. Imagine a society in which you never have to do any mundane task and can focus on whatever you like. You might not comprehend what is going on, but you might still be relevant for reasons we do not fully understand: as a consumer, as a voter, or because, in our human-centric view, a society is legitimate only if it is run by us humans (this point of view will likely also change). The next level of not understanding could be becoming irrelevant altogether, which some consider an existential catastrophe19. I do not want to go there, because this post is about understanding, but the same effects that erode our ability to see through things erode our relevance. Democracy has always assumed that citizens can, at least in principle, understand the rules that govern them. What if the mundane tasks were, in fact, the ones you wanted to do?
Am I doomposting?
Is this then really a problem, or is it just what society has been through over the decades and millennia? It may not be new in kind, but the strength of it, the opaqueness and the prevalence, are probably much greater. It may also be that the change comes slower than we think, as those who see AI as a normal technology argue20, but based on what we are seeing right now I would not put too many of my eggs in that basket.
The strongest version of the counter-case is probably Whitehead’s, from 1911. In An Introduction to Mathematics he writes: “Civilization advances by extending the number of important operations which we can perform without thinking about them.”21 Thinking costs resources. Not having to think about how to write code, or about whatever we will consider “grunt work” right now, is not necessarily a dystopic, inhuman society. Seen historically, it might be exactly what we call progress. I am not arguing that it is, but there is certainly a group of people who believe it, and who believe moreover that it is natural: a feedback loop in which AI is more efficient than humans and humans not understanding everything is the natural price of the next step in this progression.
I don’t know if understanding is fundamental to human existence or to human progress, but I do claim there is an important difference between being able to perform operations without thinking and living in a world without understanding it. By intuition and experience you do many things without thinking about them, but you could still understand them if you wanted to, and implicitly, Whitehead’s civilization keeps that option open. The question for the coming decades is whether you still can, even if you want to. How that plays out depends on how deeply AI becomes ingrained in society. If it manages to fade into the background, most of us will call it progress, and the lack of understanding will be accepted, despite occasional objections.
Another obvious reply is that AI can be a teacher, and already is. It can do things for you, but it can also explain them to you, and in fact I consider that to be one of the greatest things to use it for. There are studies suggesting that people learn less from AI doing the work than from doing it themselves22 (who would have known), although in the same study a tutor built to guide rather than answer seems to avoid much of the harm, and an AI tutor built on the same teaching practices as an active-learning class has even outperformed the class itself23. So some kind of local comprehensibility should be retainable; the system as a whole is opaque, but if you poke hard enough at a small enough point, you can start understanding a small part again.
You will understand nothing and you will be happy
Finally, Asimov’s short story The Feeling of Power (1958) is set in a future where computers do all the calculation, and a low-level technician rediscovers how to do arithmetic by hand24. The irony is in that the rediscovered understanding is valued only as a cheap replacement for the machines, since for some reason, in missile guidance, humans are cheaper and more expendable than computers. The story turns society on its head, since the intrinsic value of human understanding has been lost.
So what will understanding become, if the world turns opaque to those of us who do not want to spend our time understanding the systems AI keeps building? A hobby, a luxury, even a status symbol? In a sense that is what it already is. If you can explain how a modern AI system works (the AI itself, not what it has built), it is probably because it interests you enough to be a hobby. It is a luxury, because getting there takes time and usually a university education, in which case it is also a status symbol. But if basic understanding about basic things becomes any of these, who will be able to afford living in a society in which they retain control? And should the next generation understand the world it lives in, or is that a demand no generation has ever met for its full lifetime? I hope, that the elder with the smartphone, who still has a grandchild to ask, is not the optimistic vision of our future.
See also the continuation post: Proof of understanding
E. Karger et al., Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament, https://forecastingresearch.org/research/existential-risk-persuasion-tournament. ↩︎
M. Weiser, “The Computer for the 21st Century”, Scientific American 265(3), 94–104 (1991), doi:10.1038/scientificamerican0991-94. ↩︎
L. Rozenblit and F. Keil, “The misunderstood limits of folk science: an illusion of explanatory depth”, Cognitive Science 26(5), 521–562 (2002), doi:10.1207/s15516709cog2605_1. ↩︎
L. E. Read, “I, Pencil”, The Freeman (1958), https://fee.org/articles/i-pencil/. ↩︎
See S. Sloman and P. Fernbach, The Knowledge Illusion: Why We Never Think Alone (Riverhead, 2017), https://www.penguinrandomhouse.com/books/533524/the-knowledge-illusion-by-steven-sloman-and-philip-fernbach/, and J. Hardwig, “Epistemic Dependence”, Journal of Philosophy 82(7), 335–349 (1985), doi:10.2307/2026523. ↩︎
J. R. Vane, “Inhibition of prostaglandin synthesis as a mechanism of action for aspirin-like drugs”, Nature New Biology 231, 232–235 (1971), doi:10.1038/newbio231232a0. ↩︎
M. J. McFarland, M. E. Hauer and A. Reuben, “Half of US population exposed to adverse lead levels in early childhood”, PNAS 119(11), e2118631119 (2022), https://doi.org/10.1073/pnas.2118631119. ↩︎
M. Mead, Culture and Commitment: A Study of the Generation Gap (Natural History Press, 1970), https://www.goodreads.com/book/show/6549202. ↩︎
E. M. Forster, “The Machine Stops”, Oxford and Cambridge Review (1909), https://en.wikipedia.org/wiki/The_Machine_Stops. ↩︎
K. Appel and W. Haken, “Every planar map is four colorable”, Illinois Journal of Mathematics 21(3), 429–567 (1977), doi:10.1215/ijm/1256049011 and doi:10.1215/ijm/1256049012. ↩︎
“AlphaGo versus Lee Sedol”, Wikipedia, https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol. ↩︎
M. Shin, J. Kim, B. van Opheusden and T. L. Griffiths, “Superhuman artificial intelligence can improve human decision-making by increasing novelty”, PNAS 120(12), e2214840120 (2023), doi:10.1073/pnas.2214840120. ↩︎
For example, the robot-run shops opened in Hefei in August 2026: People’s Daily Online, “Hefei in Anhui launches robot-run stores in commercial districts” (27 August 2026), https://en.people.cn/n3/2026/0827/c90000-20492934.html. ↩︎
L. Bainbridge, “Ironies of Automation”, Automatica 19(6), 775–779 (1983), doi:10.1016/0005-1098(83)90046-8. ↩︎
Langdon Winner called this “reverse adaptation”, human ends adjusted to fit the available technical means: L. Winner, Autonomous Technology (MIT Press, 1977), https://mitpress.mit.edu/9780262730495/autonomous-technology/. ↩︎
Goodhart’s law is named after C. A. E. Goodhart, “Problems of Monetary Management: The U.K. Experience”, Papers in Monetary Economics Vol. I (Reserve Bank of Australia, 1975), reprinted in Monetary Theory and Practice (Macmillan, 1984), pp. 91–121, doi:10.1007/978-1-349-17295-5_4. ↩︎
C. Perrow, Normal Accidents: Living with High-Risk Technologies (Basic Books, 1984), https://press.princeton.edu/books/paperback/9780691004129/normal-accidents. ↩︎
L. Hutchinson, “How NASA brought the monstrous F-1 ‘moon rocket’ engine back to life”, Ars Technica (2013), https://arstechnica.com/science/2013/04/how-nasa-brought-the-monstrous-f-1-moon-rocket-back-to-life/. ↩︎
J. Kulveit et al., “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development”, arXiv:2501.16946 (2025). ↩︎
A. Narayanan and S. Kapoor, “AI as Normal Technology”, Knight First Amendment Institute at Columbia University (2025), https://knightcolumbia.org/content/ai-as-normal-technology. ↩︎
A. N. Whitehead, An Introduction to Mathematics (Williams & Norgate, 1911), ch. 5, https://www.gutenberg.org/ebooks/41568. ↩︎
H. Bastani et al., “Generative AI without guardrails can harm learning: Evidence from high school mathematics”, PNAS 122(26) (2025), https://doi.org/10.1073/pnas.2422633122. ↩︎
G. Kestin et al., “AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting”, Scientific Reports 15 (2025), doi:10.1038/s41598-025-97652-6. ↩︎
I. Asimov, “The Feeling of Power”, If: Worlds of Science Fiction (1958) ↩︎
Comments
Comments run on GitHub Discussions, so you need a GitHub account to post.