In This Post:

• More AI Use Leads to Less Critical Thinking, Obviously?

• Playing the Trust Game with AI

• Learning to Trust AI isn’t a Choice – It’s Inevitable

• So We Miss A Few Errors [Shrug]

• The Merits of The ‘Fuck It’ Approach

• Finding your Inner CEO

  ---

In the year 2012, humanity reached its intellectual peak. This is not my opinion. This is a fact reported by the Financial Times. We have been sliding towards Idiocracy ever since. That, and another new study by Microsoft lends scientific support to one of our principal fears around the increased use of AI: that it might deaden our critical thinking skills as we increasingly just accept whatever conclusions a chatbot is burping out. This suggests a particular problem for architects. While many architects would claim ‘design thinking’ as the source of their superpowers, I would respectfully disagree. It is your ability to marry design thinking with critical thinking that makes you truly magical. And if you didn’t do it well, you probably couldn’t be an architect anyhow.

Design thinking, without criticality, leads to architectural fantasy and the kind of aloof, shape-driven architecture that marginalizes the influence of architecture as a serious profession. Critical thinking, without design, leads to sterility in the built environment, and the kind of world that only engineers could live in. Which is why the death of critical thinking has me so worried – it can only push architecture further into the realm of unrestrained form-making and unbuilt, spectacle architecture.

Share

The ‘critical thinking is dying!’ bell has been ringing for so long, from so many places, that I hardly ever pay attention anymore. We’ve collectively re-understood the existence of a divergent opinion as evidence that its author wasn’t thinking critically. I mean, how could they be, if their conclusions were different than mine?

But the evidence is in, and the Microsoft study was particularly troubling because of what it said about AI – it’s not that AI robs us of critical thinking skills, it's that we willingly hand over the labor of critical thinking when using AI.

In their study, Microsoft surveyed 319 knowledge workers who use GenAI tools regularly at work, collecting 936 real-world examples of GenAI use to determine when those workers deployed their critical thinking skills, and when they just let the AI do its thing. That feels like overkill. They could have just polled the laziest worker and would have drawn the same conclusion: "Well, duh." When people grow confident that AI is performing correctly and generating accurate answers to questions & problems, they lose the imperative to check its work, and the whole thing becomes a self-reinforcing cycle:

More AI Use Leads to Less Critical Thinking, Obviously?

Microsoft's study bore out this intuition with some rigor, and had two other, more interesting findings:

1. Critical Thinking vs. Confidence: Higher confidence in GenAI correlates with less critical thinking, while higher self-confidence correlates with more critical thinking (specifically, in the use of critical thinking to evaluate the problem independently, and to check the AI's work), suggesting that domain experts reflexively use more criticality when querying AI about their own field.

2. With increased AI use, the nature of 'work' changes - from task execution to task 'stewardship' - guiding and monitoring AI outputs.

The design & conclusions of the study were problematically static for me. It seemed that the relationship between relative confidence, trust, and error checking, would evolve dynamically, based on how the User was using AI, and how the AI was responding. Like a game. Whatever mistrust I had of AI could be gradually obviated by successive, error-free performance on its part, which would lead to more trust on my part, and thus less error-checking.

But I found myself insecurely wondering: is this even a problem? We trust machines all the time. How often do you perform an operation on a calculator, and then write it out longhand to see if the calculator is doing the math correctly? Why should we be so suspicious about the things that AI tells us? Because of hallucinations? OK, granted. But assume AI is right 97% of the time. If you had a colleague who gave correct answers to 97% of all the questions you ever asked of him or her, how diligently would you doublecheck their work? Were our suspicions around AI strictly related to its newness? Were we paradoxically bothered by its near-human competence? Most importantly: when do we begin to fully trust AI, and is our trust, at that point, justified? Or dangerous?

Out of curiosity, I decided to build my own game theory model and test it out.

Playing the Trust Game with AI

The structure of my model was simple: a User asks an AI a question, and the AI responds. They play for 100 successive games. The AI can either get the answer right, or wrong, but there are actually three outcomes:

1. Every correct answer by the AI results in a 'trust boost', resulting in the User applying less critical thinking the next game.

2. Every wrong answer that's detected results in a 'trust penalty', resulting in the User applying more critical thinking the next time, increasing the likelihood (but not guaranteeing) that they would find any subsequent errors.

3. Any wrong answer that goes undetected basically behaves like a right answer, because the User doesn't know it's wrong i.e. trust goes up, and less critical thinking is applied on subsequent trials, resulting in less error detection.

Whether or not an error is detected depends on the level of critical thinking the User applies in evaluating an LLM's answer on that particular game. That, in turn, relies on the level of trust that's been built up progressively between the User and the LLM. I began the simulation with a baseline level of human trust standing at 500/1000, and a baseline likelihood of a user to detect an AI error at 95%, and then asked the model to play out 100 games.

I repeated the simulation many times, playing around with the ratio between the 'trust boost' and the 'trust penalty' (e.g. the 'trust ratio'). I had begun at 1:2. An answer that was perceived as correct (whether it was or not) received a 1 point bump in trust, while an answer that was perceived as incorrect received a 2 point trust penalty. I modified the simulation to see what would happen at trust ratios of 1:1, 1:4 and 1:10, reflecting different levels of caution.

Learning to Trust AI isn’t a Choice – It’s Inevitable

The results were unsettling. I spent about an hour building this model, another hour playing with it, and then a few sleepless nights staring at the ceiling, thinking about its implications.

The main takeaway was that no matter how I manipulated the variables, the human/AI relationship eventually ended up in largely the same place: humans largely trusting AI to get it right, and largely being unawares when AI got it wrong. Assuming the AI starts out as being accurate most of the time1, repeated correct answers build trust with the User, and therefore lead to a decline in the User's application of critical thinking to the answer he or she was given. That decline in critical thinking means that the User is less able to detect errors in the future. Rinse and repeat.

After a certain number of games, every simulation eventually stabilizes to a ‘trust equilibrium’, representing the overall trust that the User has in the AI’s responses. It’s never completely 100%, because humans aren’t wired that way, and the AI will continue to make mistakes that, even at low levels of error detection, keep the User’s trust level in check. The simulation balances out when the trust penalty and the trust boost offset each other’s effects over the long term.

Over the next few sleepless nights, I tried to imagine what variables I could tweak to avoid this depressing result. As it turns out, none of the variables particularly affected the overall outcome.

The ‘trust ratio’, for instance, barely mattered. at a trust ratio of 1:2, 'trust' eventually stabilizes around 98.8% of maximum (meaning, we really trust A.I.). At that level of trust, our error detection is 31.3% - we only ever catch a third of the errors that AI makes, anyhow. At 1:4, 'trust' stabilizes at 96.9% of maximum. So, a doubling of the User's cautiousness only leads to a minor reduction in the eventual overall trust. You'd think that the solution is just to raise the trust penalty sufficiently high that it keeps our error detection high, and we can catch the errors that AI makes! Maybe if the trust penalty were 1:10? As it turns out, that leads to more errors detected in the short term, but still ends up with the same result – it just takes longer.

The relationship between critical thinking and error detection didn’t seem to matter, either. I modeled it alternately as a linear, logarithmic, polynomial, and sigmoidic relationship, just to be sure. Nope. In all cases, our trust ascends asymptotically to some really high number (like 99%) and our deployment of critical thinking, and hence error detection, descends proportionally.

The only thing that did seem to matter was the relationship between trust and critical thinking. If trust could rise, but we could maintain high levels of critical thinking, then the user remained persistently capable of detecting the AI’s errors. But that hardly makes sense from a human psychological perspective. The whole point of trust, as a concept, is that you don’t have to inspect the result.

So We Miss A Few Errors [Shrug]

After a few days, I concluded that maybe this eventuality wasn’t all that bad. I reckon it's kinda like those little disclaimers people now put on their emails 'Sent from my mobile device, please forgive any spelling or typographical errors.' A generation ago, it would have been unforgivable to send out professional communication with typos in it - it would have been a clear sign that you weren't a professional. But nowadays, we all just kinda look the other way, because we're all using the technology. We're all aware that mobile keyboards are small, and typos are eady to make. The convenience we enjoy via mobile devices outweighs any consternation we have about poor grammar and speling. If we're all using AI, and we know that AI is going to be 99.99% error free, and we know that we, as humans, will only catch 1/3rd of the 1/1000 errors it makes, why wouldn’t' we all just adapt to say 'fuck it.' Maybe, AI hallucinations are the new typos.

The Merits of The ‘Fuck It’ Approach

Architects aren’t really known for saying ‘fuck it’, at least with respect to their designs. But there’s a salient detail of the model that I omitted up until now: in modeling these games, I specified that the AI playing the games would be 97% accurate.2 Out of 100 queries, it would only be wrong 3 times, on average.

More importantly, I modeled the AI so that its rate of accuracy was constant. Which is also a lie. The performance of frontier LLMs seems to be doubling every six months.3 If we accept 97% as a current accuracy rate, frontier LLMs will exceed the baseline Massive Multitask Language Understanding (MMLU - a broad test of knowledge and problem-solving ability) scores of human experts within the next month, and achieve 99.99% accuracy within five years.

Now, honestly, if your AI is only wrong once out of every ten thousand times, would you apply rigorous critical thinking to its results? Would anyone? Should anyone? At that point, you’re not saying ‘fuck it’ out of a lack of concern – trusting the AI to do its thing is a logical, responsible response to performance you’re seeing.

Finding your Inner CEO

The Microsoft paper didn't really have any interesting ways to deal with this conundrum. The researchers opined that AI could be designed in order to get us to think more critically, which sounds terribly circular. Asking a machine to think up ways that it could stop us from having to rely on it to think things up?

Maybe what we need isn’t a solution, but rather a different perspective. The reality is that we’re moving into a different kind of work – where we’re not gathering information, we’re just checking to see whether AI is getting it right. We’re not solving problems, per se, we’re editing the solutions that AI dreamt up.

Now, in the above paragraph, substitute “AI” with “junior staff member” and you’ll encounter an experience familiar to anyone who has ever risen through the ranks of management. Every rising manager encounters that point where you have to delegate. Every rising manager has to at some point sit down with themselves and accept the reality that your people will occasionally get things wrong. But the results you get from delegating to a productive, well-functioning team far exceed the results that you can achieve on your own, or through micromanagement. Every rising manager has to at some point shift their focus: from getting every detail of their own work perfect, to helping their team collectively strive towards perfection.

I predicted as much in my April 2023 article In the Future, Everyone’s an Architect. Fun fact: the original title for that piece was ‘Everyone an Architect, Every Architect a CEO.’ And there was some more content in earlier drafts about how architects would eventually see themselves as the CEOs of teams of AIs.

So maybe that’s the positive spin on this whole ‘death of critical thinking’ business. In their paper, Microsoft researchers understood ‘critical thinking’ as being applied to a task. To undertake a task uncritically sounds inherently bad. But it’s perfectly normal, okay, and productive to delegate your critical thinking on a task to a subordinate, if you’re a CEO. It’s one of my favorite things to do, as a manager. Every manager has a few of those employees to whom you can give anything. You know the ones I’m talking about. Your go-to. The person to whom you can hand off a task, and you never have to think about it again. Something I learned later, as a manager, was that sometimes people walk in the door with that sort of quality, but in other cases, that quality can be drawn out of people, if you give them the right kind of mentorship. I think that’s where we are. We have to train our AIs to be the sort of staff we need. We have to stop thinking about this AI transition like architects, and start thinking about it like CEOs.

Share