How AI can make us better decision-makers, with Cassie Kozyrkov

Hello, and invited to Decoder! This is Jon Fortt, CNBC journalist, cohost of Closing Bell Overtime, and creator and big of the Fortt Knox podcast. As you conscionable heard Nilay say, I’m stepping successful to impermanent big a fewer episodes of Decoder this summertime portion he’s retired connected parental leave, and I’m precise excited astir what we’ve been moving on. 

For my archetypal occurrence of Decoder, a amusement astir however radical marque decisions, I wanted to speech to an expert. So I sat down with Cassie Kozyrkov, the laminitis and CEO of AI consultancy Kozyr. She’s besides the erstwhile main determination idiosyncratic astatine Google. 

For a agelong time, Cassie has studied the ins and outs of decision-making: not conscionable determination frameworks but besides the underlying societal dynamics, psychology, and even, successful immoderate cases, the relation that the quality encephalon plays successful however and wherefore we marque definite choices. This is an interdisciplinary tract that Cassie calls determination intelligence, which mixes everything from statistic and information subject to instrumentality learning. Her expertise landed her a apical advisor relation astatine Google, wherever she spent astir a decennary helping the institution marque smarter usage of data. 

In caller years, her enactment has collided with artificial intelligence. As you’ll perceive Cassie explicate it, generative AI systems similar ChatGPT are making it easier and cheaper than ever to get proposal and analysis. But unless you person a wide imaginativeness of what it is you’re looking for, and what values underlie the decisions you make, each you’ll get backmost from AI is simply a batch of messy data. 

So Cassie and I truly dug into the subject down decision-making, however it intersects with what we’re seeing successful the modern AI industry, and however her existent enactment successful AI consulting helps companies amended recognize however to usage these tools to marque smarter decisions that can’t conscionable beryllium outsourced to agents oregon chatbots. 

I besides wanted to larn a small spot astir Cassie’s ain decision-making frameworks and however she made immoderate cardinal decisions of her own, specified arsenic what to prosecute successful postgraduate schoolhouse and wherefore she decided to permission academia for Google and past onslaught retired connected her ain conscionable arsenic the generative AI roar was truly starting to footwear off. This is simply a amusive one, and I deliberation you’re truly going to similar it. 

Okay: determination idiosyncratic Cassie Kozyrkov. Here we go.

This transcript has been lightly edited for magnitude and clarity. 

Cassie Kozyrkov, invited to Decoder. I’m going to invited myself to Decoder too, due to the fact that this isn’t my podcast. I’m conscionable having a bully clip punching the buttons, but it’s going to beryllium a batch of fun.

Yeah, it’s truthful large to beryllium present with you, Jon. And I conjecture we 2 friends managed to sneak connected and instrumentality implicit this podcast, truthful I’m truly excited for the mischief we’ll origin here.

Let the mischief begin. So the erstwhile main determination idiosyncratic astatine Google, I think, starts to framework what it is you’re bully at, and we’re going to get into the implications for AI and enactment and exertion and each that. But first, let’s conscionable commencement with the basics. What’s truthful hard astir making decisions?

Depends connected the decision. It tin beryllium precise casual to marque a decision, and 1 of the things that I counsel radical is, unless you’re a pupil of decision-making, your fig 1 regularisation should beryllium to effort to lucifer the effort you enactment into the determination with what’s astatine involvement successful the decision. So, of course, if you’re a student, you tin spell and agonize over, “How would I use a determination theoretic attack to choosing my sandwich astatine lunch?” But don’t beryllium doing that successful existent life, right?

Slowing down, reasoning carefully, and considering the hard decisions and doing your champion by them is, again, for the important decisions that volition interaction your life. Or even, much critically, the lives of thousands, millions, billions of different people, which is thing that we spot with exertion that scales. 

It sounds similar you’re saying, successful part, knowing what’s astatine involvement is 1 of the archetypal pugnacious things astir making decisions.

Exactly. And knowing your priorities. So 1 of the things that I find truly fascinating astir what AI successful the ample connection exemplary chatbot consciousness contiguous is doing is it’s making answers truly cheap. And erstwhile answers go cheap, that means the question becomes truly important. Because what utilized to hap with decision-making for, again, the big, thorny data-driven decisions, was a decision-maker mightiness travel up with thing and past inquire the information subject squad to enactment connected it. And past by the clip that squad came backmost with an answer, it had been, well, a week if you were lucky, but it could person been six weeks, oregon six months.

In that time, though, you really got the accidental to deliberation astir what you’d asked, refine what it meant to you, and past possibly re-ask it. There was clip for that ablution thought, wherever you’re like, “Oh, man, I should not person phrased it that way.” But today, you tin spell and person AI effort an reply for you, and you tin get an reply truly quickly. 

If you’re utilized to conscionable instantly moving successful the absorption of your answer, you won’t deliberation arsenic overmuch arsenic you should about, “Well, however bash I trial if this is really what I request and what’s bully for me? What did I really inquire successful the archetypal place? What was the satellite model, if you like? What were the assumptions that went into this decision?” So it’s each astir priorities. It’s each astir knowing what’s important.

Even earlier we get determination though, staying astatine the precise basal level, however bash radical larn to marque decisions? There’s the cardinal thought that if you interaction a blistery stove, you bash it erstwhile and past you cognize not to bash that again. But however does the wiring successful our encephalon enactment to thatch america to go decision-makers and make our ain processes for doing it?

Oh, I didn’t cognize that you were going to resistance my neuroscience grade into this. It has been a while. I apologize to immoderate existent practicing neuroscientists that I’m astir to offend. But astatine slightest erstwhile I was successful grad school, the models that we had for this said that you person your dopaminergic midbrain, which is simply a portion that’s precise important for question and for executing immoderate of what you would deliberation of arsenic the much instinctive behaviors, oregon those driven by basal rewards — similar sugar, avoidance of pain, those kinds of rewards. 

So you person what you mightiness deliberation of arsenic an evolutionarily older structure. And isn’t it fascinating that question and decision-making are likewise controlled successful the brain? Is a question a decision? Is taking an enactment the aforesaid happening arsenic making a decision? We tin get into that. And past determination are different structures successful the prefrontal cortex. 

Typically, your ventromedial and dorsolateral prefrontal cortices volition beryllium progressive successful assorted kinds of what you would deliberation of arsenic effortful oregon slowed-down decisions — specified arsenic the quality betwixt choosing a banal because, I don’t know, you consciousness arsenic if you don’t adjacent cognize why, and sitting down and really moving immoderate numbers, doing immoderate research, integrating each of that and having a good, long-think ponder arsenic to what you should do. 

So broadly speaking, antithetic regions from antithetic evolutionary stages play into decision-making. The prefrontal cortex is simply a small newer. But you person these systems —  sometimes acting successful a coordinated manner, sometimes a small successful struggle — progressive successful decision-making. But what we besides truly cared astir backmost successful those days was moving distant from the cartoonish instrumentality that you get successful fashionable science, that you conscionable person 1 portion and it conscionable does this 1 happening and it lone does this thing.

Instead, it’s an full web that is perpetually taking successful inputs and processing each of them. So, of course, representation would beryllium progressive successful decision-making and, of course, the quality to imagine, which you would deliberation of much arsenic engaging your ocular occipital cortices — that would decidedly beryllium progressive successful immoderate mode oregon other. So it’s a full thing. It’s a full web of activations that are implementing quality decisions. To summarize this for you, Jon, neuroscientists person nary thought however we marque decisions. So that’s the comic conclusion, right?

What we tin bash is prod and pry and get immoderate consciousness of it, but astatine the extremity of the day, the existent nitty-gritty of however humans marque decisions is simply a mystery. What’s besides truly comic is humans deliberation they cognize however they marque decisions, but rather often you tin works a determination and past unbeknownst to your participants, arsenic we telephone them successful the studies — I’d accidental victims — unbeknownst to them, the determination was made for them each along. It was primed successful immoderate way. Certain inputs got successful there. 

They thought they made a decision, and past afterward you inquire them, truthful wherefore did you prime reddish and not blue? They volition sing you this beauteous song, explaining however it was their grandmother’s favourite colour oregon immoderate it is. Meanwhile, the experimenter implanted that, and if you don’t judge me, spell spot a magic show. It’s the aforesaid principle, right? Stage magicians volition works decisions successful their audiences truthful reliably, different the amusement wouldn’t work. I’m ever fascinated by however earnestly we instrumentality our quality quality to cognize and recognize ourselves and consciousness arsenic if we’ve got each this bureau broadside by broadside with nonrecreational signifier magicians entertaining crowds each day.

But it sounds to maine similar possibly what truly drives decisions, and possibly this question and question portion of the encephalon is portion of it, is privation — what we want. When we’re babies, erstwhile we’re toddlers, decisions are: Do I get up? Am I hungry? Do I cry? It’s basal worldly that has to bash with mostly carnal things, due to the fact that we’re not intellectuals yet, I guess. 

So you request to person a privation oregon a extremity successful bid for determination to beryllium a determination to beryllium made, right? Whether we recognize what our existent information is oregon not, that’s a cardinal ingredient, having immoderate benignant of privation oregon extremity successful decision-making.

Well, it depends however you specify it. So with each these terms, erstwhile you effort to survey decision-making successful the societal biologic sciences, you’ll person to instrumentality a word, specified arsenic “decision,” which we usage casually nevertheless we like, and past you’ll person to springiness it a small container that makes that explanation much concrete. It’s conscionable similar saying: “let X equal…,” right? At the apical of your leafage erstwhile you’re doing math, you tin accidental fto X adjacent the velocity of light. Now, from present on, whenever I constitute X, it means the velocity of light. And past for immoderate different person’s paper, fto X adjacent five, and past whenever they constitute X, it means five. 

So similarly, we say, “Let determination equal…” and past we specify it for the purposes. Typically, what determination analysts volition accidental defines a determination — the mode they bash their “let determination equal…” astatine the apical of their leafage — is they accidental that it is an irrevocable allocation of resources. Then it’s up to you to deliberation about, again, however you privation to specify what it means for the allocation to beryllium irrevocable, and what it means for the resources to beryllium allocated astatine all. 

Is this an enactment that a quality indispensable make? Is it an enactment that a strategy downstream of a quality mightiness make? And what are resources? Are resources conscionable money, oregon could they see time? Or opportunity? For example, what if I take to spell done this door? Well, successful this moment, successful this beingness close now, I didn’t take to spell done that door, and I can’t spell back. So successful that sense, perfectly each question that we marque is an irrevocable allocation of resources. 

And successful companies, if you’re Google, bash you bargain YouTube oregon not? I mean, that was a large determination backmost then. Do I prosecute this idiosyncratic oregon that person? If it’s a cardinal worker role, that tin person a immense interaction connected whether your institution succeeds oregon fails. Do I put successful AI? Do I oregon don’t I follow this exertion astatine this stage?

Right, and you tin take however to framework that to marque it definitionally irrevocable. If I prosecute Jon close present astatine this constituent successful time, past I’m possibly giving up doing thing else, specified arsenic eating my sandwich alternatively of going done each the paperwork of hiring Jon. So I could deliberation that’s irrevocable. If I prosecute Jon, I mightiness beryllium capable to occurrence Jon time and merchandise immoderate resources that I cared much astir than clip and existent opportunity. So past I could dainty that arsenic I’m capable to person a two-way doorway connected this decision. 

So really, it depends connected however you privation to framework it, and past the remainder volition somewhat travel successful the math. A large portion of however we deliberation astir decision-making successful science is to abstracted it into judgement and decision-making.

Judgment is abstracted from decision-making. Judgment comes successful erstwhile you undertake each the effort of deciding however to decide. What does it really mean for you to allocate your resources successful a mode without take-backsies? So it’s up to the decision-maker to deliberation astir that. What are we measuring? What’s important? How mightiness we really privation to attack this decision? 

Even saying thing like, “This determination should beryllium made by gut instinct alternatively than by effortful calculation,” is portion of that judgement process. And past the decision-making process that follows, that is conscionable riding the mathematical consequences of immoderate judgement setup you made.

So speaking of setup, springiness maine the emblematic setup. Why bash clients prosecute you? What kinds of positions are they successful wherever they’re like, “Okay, we request a determination idiosyncratic here”?

Well, typically, the large ones are those involving deployment of AI systems. How would you deliberation astir solving a occupation with AI? That’s a large decision. Should I adjacent enactment this AI strategy successful place? I’m perchance going to person to gut immoderate I’m already using. So if I’ve got immoderate handcrafted strategy immoderate bundle developers person already written for me, and I’m getting reasonably bully results from that, well, I’m not conscionable going to propulsion AI successful determination and anticipation for the best. Actually, successful immoderate situations you would bash that, due to the fact that you privation to say, “I’m an AI company.” And truthful you privation to default to putting the AI strategy successful unless you get talked retired of it. 

But rather often it’s effortful, it’s expensive, and we privation to marque definite that it’s going to beryllium bully capable and close for that company’s situation. So however bash we deliberation astir measuring that, and however bash we deliberation astir the realities of gathering it truthful it has each the features that we would necessitate successful bid to privation to proceed. It’s a immense decision, this AI decision.

How overmuch does a leader’s oregon a company’s values substance successful that assessment?

Incredibly. I deliberation that’s thing that radical truly miss erstwhile it comes to what looks similar information oregon math-y situations. Once we person that spot of math, it looks objective. It looks similar “you commencement here, you extremity up there,” and determination was lone 1 close answer. What we hide is that that small mathematics portion and that information portion and that codification portion signifier a bladed furniture of objectivity successful a big, abdominous subjectivity sandwich. 

That archetypal furniture is: What’s adjacent important capable to automate? What’s important capable to bash this successful the archetypal place? What would I privation to improve? In which absorption bash I privation to steer my business? What matters to me? What matters to my customers? How bash I privation to alteration the world? These questions person nary 1 close answer, and volition request to beryllium articulated intelligibly successful bid for the remainder to marque sense.

The companies thin to articulate those things done a ngo statement. Very often, astatine slightest successful my experience, those ngo statements aren’t astir elaborate capable to usher the granular and heavy bid of events that AI is going to pb america down, no? 

Absolutely, and this is simply a truly important constituent that blossoms into the full taxable of however to deliberation astir determination delegation. So the archetypal happening leaders request to recognize is that erstwhile they are astatine the precise apical of the nutrient concatenation successful their organizations, they don’t person the clip to beryllium progressive successful precise granular decisions. In fact, astir of the occupation is figuring retired however to delegate decision-making to everybody else, choosing whom to spot oregon what to spot if we’re going to commencement to delegate to automated systems, and past letting spell of that decision.

So you don’t privation to beryllium asking the CEO astir nitty-gritty topics around, let’s say, the cybersecurity pieces of the company’s shiny caller AI system. But what the institution needs to bash arsenic an enactment is marque definite that idiosyncratic successful the task is reasoning astir each the components that request to beryllium thought about, and that it’s each delegated to the close people. So portion of my relation past is asking a batch of questions astir what’s important, who tin bash this, however bash we enactment it each together, and however bash we marque definite that we’re not operating with immoderate unsighted spots oregon missing immoderate components.

How typically are clients acceptable to supply you with that information? Is that a speech they’re utilized to having?

Again, we’ve travel a agelong way, but for the longest time, arsenic a civilization moving with data, we’ve been fascinated by conscionable being capable to perchance bash a happening adjacent if we don’t cognize what it’s for. We thought, “Isn’t it chill that we tin determination this data? Isn’t it chill that we tin propulsion patterns retired of it? Isn’t it chill that we tin store oregon cod it astatine scale?” All without really asking ourselves, “Well, wherever are we going, and however are we going to usage it?” 

We are increasing retired of that painful, teething signifier wherever everyone was like, “This is fun, and let’s bash it for theory.” It’s benignant of similar saying, “Well, we’ve invented a wheel, and present we tin invent a amended wheel, and we tin present marque it into a tyre and it tin person rubber connected it, but possibly it’s made from c fiber.”

Now we are moving into, “Okay, this happening enables movement, antithetic investments successful this happening alteration antithetic speeds of movement, but wherever bash I privation to go? Because if I privation to spell 2 yards over, past I don’t really request the car, and I don’t request to beryllium fascinated by it for its ain sake.” 

Whereas if what I truly request to bash is beryllium successful the adjacent metropolis tomorrow, and I don’t presently person a car, well, past we’re besides not going to speech astir inventing it from scratch by hiring researchers. We’re not going to deliberation astir gathering it in-house. We’re going to ask, “Who tin get you thing that volition get you determination connected clip and connected spec?” These conversations are new, but this is wherever we’re going. We person to.

It sounds like, and close maine if I’m incorrect here, AI is going to assistance america a batch much with giving america facts and options and little with giving america values and goals.

I anticipation so. That is the hope, due to the fact that erstwhile you instrumentality values and goals from AI, what you’re doing is taking an mean from the internet, oregon possibly successful a strategy that has a small spot much logic moving connected apical of it to nonstop its output, past you mightiness beryllium taking those values and goals from the engineers who designed that system. So it’s similar saying, “If I’m going to usage AI arsenic my unsmooth draught each time, that unsmooth draught mightiness beryllium a small spot little maine and a small spot much the mean crockery of culture.” If everyone starts doing that, past it’s surely a benignant of blending oregon averaging of our insights. 

Perhaps you privation that, but I deliberation there’s inactive a batch of worth successful having radical who are adjacent to their occupation areas, who are adjacent to their businesses, who person idiosyncratic expertise, to think  a small spot earlier they begin, and to truly framework what the question is alternatively than instrumentality it from the AI system.

So Jon, however this would spell for you is, you mightiness inquire an AI system, “How bash I unrecorded the champion imaginable life?” And it’s going to springiness you an answer, and that reply is not going to acceptable you. That’s the thing. It’s going to acceptable the mean Joe. What is oregon who is the mean Joe, and however does that use to you? 

It’s going to spell to Instagram, and it’s going to look astatine who’s got the astir likes and followers, and past determine that those radical person the champion lives, and past instrumentality the attributes of those radical — however they look, however they talk, the level of acquisition they accidental they person — and say, well, here’s what you request to bash to beryllium similar these radical who, the information tells us, radical deliberation person the champion lives. Is that a mentation of what you mean?

Something similar that. More convoluted, due to the fact that thing that is worthy realizing is that an vantage machines person implicit america is representation and attention, right? What I mean by this is if I flash 50 digits onscreen close present and past inquire you to callback them, you’re going to person nary idea. Then I tin spell backmost to those 50 and say, “Yeah, the instrumentality remembered it for america this full time. It is intelligibly amended astatine representation than Jon is.” 

Then we flash these things, and I say, “Quick, what’s the sum of these digits?” Again, hard for you, but casual for a machine. So thing that fits successful our heads arsenic we sermon it is going to beryllium a shortcut of what’s really imaginable erstwhile you person representation and attraction astatine scale. In different words, we’ve described this Instagram process that fits successful our heads close now, but you should expect that immoderate is really going connected with these systems is conscionable excessively large for america to clasp successful there.

So sure, Instagram and immoderate different sources and astir apt adjacent immoderate websites astir however to unrecorded a bully beingness applied to us, but it’s each kinds of things each jumbled unneurotic into thing excessively analyzable for america to recognize what it is. But the important happening is it’s not tailored to america specifically, not without america putting successful rather a batch of effort to provender successful the accusation required for that tailoring, which I promote america to do. 

Certainly, knowing that proposal is cheaper than ever. I volition framework up immoderate is absorbing to maine and springiness it to the system. Of course, I’ll region the astir confidential details, but I’ve asked each kinds of things astir however I might, let’s say, amended looking astatine existent property fixed my peculiar concern and my peculiar tastes. I’ll get a precise antithetic reply than if I conscionable say, “Well, however bash I invest?” I’ve adjacent improved silly things, similar I discovered that I necktie my shoelaces excessively tight. I had nary idea, convey you, AI. I present person amended method for having feet that are little sore. 

Did you observe done AI that you necktie your shoelaces excessively tight?

Yeah, I went debugging. I wanted to effort to fig retired wherefore my feet were sore. To assistance maine diagnose this I gave the strategy a batch of accusation astir me, specified arsenic erstwhile my feet were sore, what I was doing astatine the time, what shoes I was wearing. We went done a small debugging process: “Okay, archetypal happening we’ll effort is utilizing a antithetic shoelace-tying method from the 1 that you person used, which was loop and past loosen a small bit.” I’m like, “Wow, present my feet don’t hurt. How awesome.” 

So immoderate it is that’s bugging you, you could spell and effort to debug it a small spot with AI, and conscionable spot what you get. Maybe it’s useful, possibly it isn’t. But if you simply springiness the strategy thing and inquire thing like, “How bash I go arsenic steadfast arsenic possible?” You’ll astir apt not get immoderate accusation astir what to bash with your shoelaces. You’re conscionable going to get thing from precise averaged-out, smoothed-out soup. 

In bid to get thing useful, you person to bring thing to the table. You person to cognize what’s important to you. You person to cognize what you’re trying to achieve. Sometimes, due to the fact that your feet wounded close now, it’s important to you close now, and you’re benignant of reacting the mode that I was. I astir apt wouldn’t inquire immoderate proactive questions astir my shoelaces, but sometimes what truly helps is stepping backmost and saying, “Well, what is determination successful my beingness close present that could beryllium better? And past wherefore not inquire for advice?” 

AI makes proposal cheaper than ever before. That’s the large revolution. It besides helps with each kinds of nuanced advice, similar pulling retired immoderate of your determination framing — “help maine framework my ideas, assistance maine inquire myself the questions that would beryllium important for getting done immoderate oregon different decision.”

Where are astir radical making the biggest mistakes, oregon wherever bash they person the biggest unsighted spots erstwhile it comes to decision-making? Is it asking the close questions? Is it deciding what they want? What would you accidental it is?

One is not getting successful interaction with their priorities. Again, erstwhile you’re not successful interaction with your priorities, anyone’s advice, adjacent from the champion person, could beryllium atrocious for you. And this is thing that besides applies to the AI sphere. If we aren’t successful interaction with what we request and want, and we conscionable inquire the crockery to springiness america backmost immoderate mean archetypal draught and past we travel it to a T, what are the chances it volition really acceptable america precise well?

Let maine enactment a circumstantial concern connected this, due to the fact that I’m the genitor of a soon to beryllium 17-year-old, second- semester inferior successful precocious schoolhouse who’s getting acceptable to use to colleges, and this is 1 of the archetypal large decisions that young radical make. It’s two-sided, which is truly fraught due to the fact that you’re deciding wherever to apply, and the schools are deciding who to fto in.

It seems similar that applies present too, due to the fact that immoderate radical are going to use to a schoolhouse due to the fact that their parents went there, oregon due to the fact that it’s an Ivy League. So done that framing, tin you speech astir the types of mistakes that radical marque from the position of a precocious schooler applying to college?

I’m going to support trying to necktie this backmost a small spot to what we tin larn astir our ain interactions with LLMs, due to the fact that I deliberation that’s adjuvant for radical successful this brave caller satellite of however we usage these AI tools. So again, we person 3 stages, approximately: you person to fig retired what’s worthy asking, what’s worthy doing, and past you request to get immoderate proposal oregon method help, immoderate execution spot — that mightiness beryllium you, it mightiness beryllium the LLM, oregon mightiness beryllium your dada giving you large advice. And past erstwhile you person the advice, you request to person a infinitesimal successful which you measure if it’s really bully for you. Do I travel this, and is it bully proposal oregon atrocious advice; and bash I instrumentality it and bash I execute it? It’s these 3 stages. 

So the archetypal one, the slightest comfy one, is asking yourself, “Well, however bash I really framework what I’m asking?” So to use it specifically to your kid, it would beryllium what is the intent of assemblage for me? Why americium I adjacent asking this question? What americium I imagining? What are immoderate things I mightiness get retired of this assemblage versus that college? What would marque each antithetic for me? What are my priorities? Why are these priorities my priorities? 

These are questions wherever if you are not successful tune with your answers, what volition hap is you volition person proposal from wherever — from the culture, from the internet, from your dada — and you are apt to extremity up doing what is bully for them alternatively than what’s bully for you, each from not asking yourself capable preliminary questions.

It’s similar the magician scenario. They provender you an reply subconsciously, and you extremity up spitting that backmost without adjacent realizing it’s not what you truly wanted.

Your dada mightiness say, arsenic my dada did, that economics is simply a truly absorbing and chill happening to study. This benignant of went into my caput erstwhile I was possibly 13 years old, and it kept knocking astir successful there. So that’s however I recovered myself successful economics classes and ended up majoring successful economics astatine the University of Chicago. 

Actually, it’s not ever existent that what your parents enactment successful determination makes its mode out, of course, due to the fact that some of my parents were physicists, and I precise rapidly discovered that I wanted thing to bash with physics due to the fact that of the changeless parental “you should bash amended successful physics, and you should instrumentality much physics classes.” And then, of course, aft I rebelled successful college, I ended up successful grad schoolhouse taking physics successful my neuroscience program. So determination you go, it comes astir afloat circle. 

But the constituent is that you person to cognize what you want, what’s important to you, and truly beryllium successful interaction with this truthful that you’re not pushed astir by different people’s proposal and adjacent what seems similar the champion proposal — and this is important — adjacent the champion proposal could beryllium atrocious for you. So erstwhile you deliberation idiosyncratic is competent and capable, and truthful I should perfectly instrumentality their advice, that’s a mistake. Because if what’s important to them is not what’s important to you, and you haven’t communicated intelligibly to them oregon they don’t person your champion interests astatine heart, past this intelligent proposal is going to pb you disconnected a cliff. I conscionable privation to accidental that with AI, it could beryllium a show system, but if you haven’t fixed it the discourse to assistance you, it’s not going to assistance you.

The AI constituent is wherever I wanted to go, and I deliberation you’ve talked astir this successful the past too. AI presents itself arsenic precise competent and precise definite that it’s close with precise small saltation that I’ve seen based connected the existent output. It’s not saying, “Eh, I’m not wholly sure, but I deliberation this erstwhile it’s astir to hallucinate,” versus, “Oh, here’s the reply erstwhile it’s perfectly right.” It’s definite astir 100 percent of the time.

So that’s a plan choice. Whenever you person existent probabilistic stages successful your AI output, you tin alternatively aboveground thing to bash with confidence, and this is achievable successful galore antithetic ways. For immoderate models, adjacent immoderate of the basal models, what happens determination is you get a probability first, and past that converts into enactment oregon output that the idiosyncratic sees for different situations. 

For example, successful the backend, you could tally that strategy aggregate times, and you could inquire it, “What is 2 positive two?” And past successful the backend you could tally this 100 times, and you observe that 99 retired of 100 times, the reply comes backmost with a 4 successful it. You could past amusement immoderate benignant of assurance astir this being astatine slightest what the taste crockery thinks the reply is, right?

Let’s ask, “What is the superior of Australia?” If the taste crockery says implicit and implicit that it’s Melbourne, which it isn’t, oregon that it’s Sydney, which it besides isn’t — for those for whom that’s a surprise, Canberra is the close answer. But if capable of the taste crockery says Sydney, and we’re lone sourcing from the taste soup, and we’re not kicking successful immoderate other logic to spell specifically to Wikipedia and lone gully from that, past you would get the incorrect reply with precocious confidence. But it would beryllium imaginable to people that confidence. 

In situations wherever the taste crockery isn’t truthful definite of something, past you would person a assortment of antithetic responses coming back, being averaged, and past you could say, “Well, the happening I’m showing you close present is lone showing up successful 20 percent of cases, oregon successful 10 percent of cases.” Or you could adjacent springiness a breakdown: “This is the modal answer, the astir communal answer, and past these are immoderate answers that besides amusement up.” Not to bash this is precise overmuch a user-experience plan determination positive a compute and hardware decision.

It’s besides a taste issue, isn’t it?

It is simply a taste issue.

It seems to maine that successful the US, and possibly this is existent of a batch of Western cultures, we worth confidence, and we worth certainty adjacent much sometimes than we worth correctness. 

There’s this civilization successful concern wherever we benignant of expect close down to the infinitesimal erstwhile a institution fails for the CEO to say, “I’m truly assured that we’re going to marque this work,” due to the fact that radical privation to travel idiosyncratic who’s confident, and past the adjacent time they say, “Ah, well, I failed, it didn’t enactment out.” We benignant of judge that and think, “Oh, well, they gave it their best, and they were truly confident.”

It’s the aforesaid successful sports, right? The team’s down 3 games to 1 successful a champion of 7 series, and the squad that’s lone got 1 win, they’re like, “Oh, we’re truly assured we tin win.” Well, really, the statistic accidental you’re astir apt not going to win, but we cognize that they person to beryllium assured if they’re going to person immoderate chance. So we judge that, and successful a mode we’ve created AI successful our ain representation successful that respect. 

Well, we’ve surely created AI successful our ain image. There’s a batch of user-experience plan that goes into that, but I don’t deliberation it’s an inevitable thing. I cognize that connected the 1 hand, determination is this conception of the fluency heuristic. So a idiosyncratic oregon strategy that appears much fluent, with little hesitation, little uncertainty, is perceived arsenic much trustworthy. This probe has been done; it’s aged probe successful psychology.

Now you spot that the fluency heuristic is perfectly hackable, due to the fact that if you hide that you’re dealing with a machine strategy that has immoderate advantages, similar memory, attention, and, well, fluency, you could conscionable precise rapidly rattle disconnected a clump of nonsense you don’t understand. And that lands connected the idiosyncratic oregon the listener arsenic competence, and truthful translates arsenic much trustworthy. So our fluency heuristic is perfectly hackable by instrumentality systems. It’s overmuch harder for maine to hack it arsenic a human. Though we bash person artists who negociate it precise well, it’s precise hard to talk fluently connected a taxable that you person nary thought astir and don’t cognize however immoderate of the words spell together. That lone works if that’s the unsighted starring the blind, wherever nary 1 other successful the country knows however immoderate of it works either. 

On the different hand, I’ll say, astatine slightest for me, I deliberation it has helped maine successful my vocation to signifier a estimation that, well, I accidental it similar it is, and truthful I’m not going to unreal I don’t cognize a happening erstwhile I don’t cognize it. You asked maine astir neuroscience, and I told you that it’s been a agelong clip since my postgraduate degree. Maybe we should set what I’m saying, right? I bash that. That is not for each markets. Let’s conscionable accidental galore would think, “She has nary thought what she’s talking about. Maybe we shouldn’t bash concern with her,” but for sure, there’s inactive worth successful my approach, and I’ve decidedly recovered it’s helped maine to go battle-bested and trustworthy. 

That said, erstwhile it comes to designing AI systems, that stuttering deficiency of assurance would not make a large idiosyncratic experience. But similarly, immoderate of the things that I talked astir present would beryllium costly compute-wise. What I spot a batch successful the AI manufacture is that we person concern radical reasoning that thing is not technologically imaginable due to the fact that it is not being fixed to users, and peculiarly not astatine scale, oregon adjacent offered to businesses. Quite often, it is precise overmuch technologically possible. It’s conscionable not profitable to connection that feature. There is nary bully concern case. There’s nary motion that users volition respond to it successful a mode that volition marque it worthy it. 

So erstwhile I’m talking astir moving thing 100 times and past outputting thing similar a assurance score, you would person immoderate decision-making astir whether it is 100, 10, oregon 1,000; and this depends connected a slew of factors, which, of course, we could get into if that’s the occupation you arsenic a concern are solving. But erstwhile you conscionable look astatine it connected the surface, I’m saying fundamentally 100 times much compute, right? Run this happening 100 times alternatively of once, and for what? Will the users respond to it? Will the concern attraction astir it? Yeah, often you’d beryllium amazed astatine what’s already possible. Agents similar [OpenAI’s] Operator, [Anthropic’s] Claude Computer Use, [Google’s] Project Mariner, each these things, they are underperforming, comparative to wherever they could beryllium performing, connected intent due to the fact that it is costly to tally them well. So it volition beryllium precise breathtaking erstwhile businesses and users are acceptable to wage much for these capabilities.

So backmost up for maine now, due to the fact that you near Google astir 2 years ago, a small little than that. You were determination for astir 10 years, and agelong earlier the OpenAI and ChatGPT question of AI enthusiasm had swept crossed the globe. But you were moving connected immoderate of this stuff. So I privation to recognize some the enactment astatine Google and what led you there. 

I deliberation you said that your dada archetypal mentioned economics to you erstwhile you were 13, and that sounds truly young, but I deliberation you started assemblage a mates of years later. So you were really connected your mode to those studies astatine the time. What made you determine to spell to assemblage that aboriginal and what was motivating you?

One of the things we don’t speech astir capable is that knowing what motivates idiosyncratic tells you much astir that idiosyncratic than beauteous overmuch thing other could. Because if you’re conscionable observing the outcomes, and you’re having to marque your ain inferences astir however they got there, what they did, wherefore they did it, peculiarly with survivorship bias occurring, it mightiness look similar they’re specified full heroes. Then you look astatine their existent determination process, and that whitethorn archer you thing precise different, oregon you whitethorn deliberation someone’s not precise palmy without realizing that they’re optimizing for a precise antithetic happening from you. This is each a precise agelong mode of saying that — I’m gladsome we’re friends, Jon, due to the fact that I’ll spell for it — but it’s ever conscionable specified a backstage question. But yeah, wherefore did I spell to assemblage truthful young? Honestly, it was due to the fact that I had skipped grades successful simple school.

The crushed I skipped grades successful simple schoolhouse was due to the fact that I came location — I was 9 years aged oregon truthful — and informed my parent that I wanted to bash this. I cannot retrieve why. For the beingness of me, I don’t know. I was doing thing connected a nine-year-old’s whim, and skipping grades wasn’t a done happening successful South Africa wherever I was increasing up. So my parents had to truly conflict with the schoolhouse and adjacent the section of acquisition to let it. So determination I was, getting to precocious schoolhouse astatine 12, and I really truly enjoyed being younger. Okay, you get bullied a small bit, but I enjoyed it. I enjoyed seeing that you could larn a lot, and I wasn’t intellectualizing it the mode I americium close now, but you could larn a batch from radical who were older than you.

They tin benignant of propulsion you, and I’m a immense believer successful conscionable the enactment of being surrounded by radical who volition propulsion you, which is possibly my biggest statement for wherefore assemblage inactive makes consciousness successful the AI era. Just spell beryllium successful a spot wherever everyone’s connected a travel of self-improvement. So I learned this and ended up making friends with 12th-graders erstwhile I was 13, and past astatine 14, they were each retired already and successful college. And I had spent astir of my clip with these older kids, and present I’m stuck, and I fundamentally privation my friends back. So that is wherefore I went truthful young. It was 100 percent conscionable a teen being driven by being a societal carnal and wanting to beryllium astir my adjacent group, which…

But beryllium just to yourself. It sounds arsenic if you conscionable wanted to spot however accelerated the car could go, right? That’s portion of what it was astatine nine. You realized that you were susceptible of bigger challenges than the ones you had been given. So you were benignant of like, “Well, let’s see.” And past you went and you saw that you were really capable to grip that, the intelligence part. People astir apt said, “Oh, but the societal portion would beryllium hard.” But “Hey, I got friends who are seniors. That part’s moving too. Well, let’s spot if I tin really thrust this astatine assemblage speed.” That was portion of it, right?

I americium truthful casual to manipulate with the words, “You can’t bash X.” So casual to manipulate. I’m like, “No, fto maine amusement you. I emotion a challenge. Let’s get this happening done.” So yeah, I deliberation you’re close successful your assessment.

So past you went connected to bash postgraduate work, aft the University of Chicago, to survey neuroscience, with immoderate economics successful determination too? 

So I really went to Duke for neuroeconomics. That was the field. You cognize however there’s macroeconomics and microeconomics? Well, this was similar nano-picoeconomics.This was astir however the encephalon implements decision-making. So, of course, the courses impact experimental microeconomics. That was portion of it, but this was from the science and neuroscience departments. So it’s technically a postgraduate grade successful science and neuroscience with a absorption connected the neuroscience of decision-making, which is called neuroeconomics. 

I besides went to grad schoolhouse twice, which is definitive impervious that I’m a atrocious decision-maker, successful lawsuit anyone was going to deliberation that I personally americium a bully one. I’ve conscionable got the technique, folks. I’ll counsel you. But I went to grad schoolhouse twice, and I’m conscionable kidding. It was really bully for maine to spell to grad schoolhouse twice, and my 2nd clip was for mathematical statistics. My undergraduate enactment was economics and statistics. So past I went for mathematics statistics, wherever I did a batch of what we called backmost past instrumentality learning, what we would telephone AI today. 

How galore PhDs were progressive there?

[Laughs] No PhDs were harmed successful the making of this person.

Okay, but studying some of those disciplines. What were you going to bash with that?

So coming backmost to college, wherever I was taking courses astir decision-making, contempt having been an economics and statistic major. I got a sensation for this. So I’ll archer you wherefore I was successful the stats major. The stats large happened due to the fact that astatine astir property 8 oregon nine, conscionable earlier this jumping of grades, I discovered the astir beauteous happening successful the world, which everybody knows is spreadsheets. That was for maine the astir gorgeous thing. Maybe it’s the librarian’s impulse to enactment bid into chaos. 

So I had this gemstone collection. Its full intent was to springiness maine different enactment for my spreadsheet. That was the full thing. I get an amethyst, I could beryllium like, Oh, it is purple, and however hard is it? And it’s translucent. And I inactive find, though I person nary concern doing it, that the enactment of information introduction with a bully solid of vino is conscionable specified a soothing happening to do.

So I had been playing with data. Once you commencement collecting it, you besides find that you commencement manipulating it. You commencement to person these urges like, “Oh, I wonderment if I could get the information of each my files connected my machine each into a spreadsheet. Well, fto maine fig retired however to bash that.” And past you larn a small spot of coding. So I conscionable got each these information skills for free, and I thought information was truly pretty. So I thought stats would beryllium my casual A. Little did I cognize that it’s really philosophy, and the doctrine bits are ever the bits that should footwear your butt oregon you’re missing the point. But of course, manipulating the information bits was super-duper easy. Statistics, I realized arsenic I began to soak successful the philosophy, is the subject of changing your caput nether uncertainty.

Economics is the subject of scarcity, and the allocation of scarce resources. And adjacent if wealth is not scarce, thing is ever scarce. People are mortal, clip is scarce. So asking the question, “How are you going to marque allocations, oregon what you mightiness telephone decisions?” got successful determination done economics. Questions similar “how to alteration your caput and what is your caput acceptable to do. What actions are connected the table? What would it instrumentality to speech you retired of it? 

I started asking these questions, and past however does this really enactment successful the quality animal, and however could it enactment better? These questions came successful done the science and neuroscience broadside of my studies. So I was studying decision-making from each perspective, and I was hoarding. So present arsenic well, did I cognize what vocation I was going to have? I was actively discouraged from doing this. When I was astatine the University of Chicago, adjacent astatine that wide arts place, my undergraduate advisor said, “I person nary thought what occupation you deliberation you’re going to get with each this stuff.”

I said, “That’s okay, I’m learning. I deliberation this is benignant of important.” I hadn’t articulated backmost past what I’ll accidental now, which is that information is pretty, but there’s nary “why” successful data. The wherefore comes from the decision-maker, right? The intent has to travel from people. It’s either your ain intent oregon the intent of the radical whom you represent, and that is what gives absorption to each the remainder of it. So [it’s] conscionable studying information wherever it feels similar there’s a close reply due to the fact that the prof acceptable the occupation up truthful that there’s a close answer. If they had acceptable it up differently, determination could person been antithetic answers. 

Realizing that the setup has infinite choices, that is what gives information its why, and its meaning. That is the determination piece. That’s the astir important happening I deliberation immoderate of america could walk our clip on. Though we each bash walk our clip connected it and bash attack it from antithetic lenses.

So past wherefore Google? Why did you committedness yourself you wouldn’t enactment for a institution for much than 10 years?

Well, we’re truly getting into each the things. So Google is simply a comic one, and present I’ll decidedly accidental immoderate things that I don’t deliberation I’ve said connected immoderate podcasts. But the existent communicative of that is that I was successful a mathematics stat PhD program, and what I didn’t cognize was that my advisor — this was astatine North Carolina State — had conscionable taken an connection astatine Berkeley, wherever helium could not bring immoderate of his students on with him. That was a beauteous atrocious happening for me, successful the mediate of  my PhD. 

Now, abstracted from this going connected that I had nary thought about, I instrumentality Halloween beauteous seriously. It’s my thing. At Kozyr, it’s a enactment holiday, truthful radical tin bask Halloween decently if they privation to. And I had travel connected Halloween greeting dressed arsenic a punch paper arsenic 1 does with due Fortran to people blessed Halloween arsenic 1 does, and a Googler was giving a talk, and I was sitting successful that audience, the lone idiosyncratic successful costume, due to the fact that everyone other is lame.

Let that spell connected the record. My erstwhile classmates should person been successful costume, but we tin inactive beryllium friends. And truthful astatine 9AM, I’m dressed similar this. The Googler woman talking to the caput of the section is like, “Who’s that grad pupil who was dressed arsenic a punch card?” The caput of the department, not having seen me, inactive said, “Oh, that’s astir apt Cassie. Last twelvemonth she was dressed arsenic a Sigma field,” conscionable from measurement theory. So I was being a immense nerd. The Googler thought “culture fit,” 100 percent, let’s get her exertion in.

And truthful the exertion was conscionable for a summertime internship, which seemed similar a harmless happening to do. Sure, let’s effort it. It’s an adventure. It’s Google. Then arsenic I was signing up for it, my advisor was like, “This is simply a precise bully happening for you. You shouldn’t adjacent hesitate. Don’t beryllium asking maine if I privation you present doing summertime research. Definitely spell to Google. You tin decorativeness your PhD there. Go to Google.” And the remainder is history. So a much, overmuch amended enactment than having to restart and refigure things with a caller adviser.

How did you extremity up becoming this translator betwixt the information radical and the decision- makers?

The relation that I ended up getting astatine Google, the ceremonial internship name, was decision-support intern. I thought to myself, “We’ll fig retired the support, and we’ll fig retired the intern.” But decision, this is what I’ve been grooming for my full life. The squad that I was successful was similar a SWAT squad for data-driven-decision making. It was very, precise adjacent to Google’s superior revenue. So this was a no-messing-around squad of statisticians that called itself determination support. It was hardcore statistic flavored with information science, and it besides had a precise hardcore engineering radical — it was a precise large group. I learned a batch there.

I applied to perchance enactment successful the aforesaid radical for a full-time relation with beardown prompting from my PhD adviser, and I thought I was going to articulation that group. A tangential happening happened, which is that I took a play successful New York City earlier going to Mountain View, which is wherever I picked retired my apartment. I thought I was going to articulation this group. I was really, truly excited to beryllium surrounded by heavy experts successful what I cared about. These experts were really moving much connected the information broadside of things due to the fact that what the decisions are and however we attack them are truthful regimented successful that portion of Google. But I took this travel to New York City, and I realized, and this was 1 of the biggest gut-punch decision-making moments for me. I realized I’m making a unspeakable mistake, that if I spell there, I volition conscionable not bask my beingness arsenic overmuch arsenic if I spell to New York City.

So determination was truthful overmuch instinct, determination was truthful much, “Oh, no, I should really truly reevaluate what I’m doing. Am I going to bask surviving successful Mountain View?” I was conscionable truthful acceptable connected getting the connection that I hadn’t done what I truly should person done, which was to measure my priorities properly. So the archetypal happening I did was I called the recruiter and I said, “Whoa, whoa, whoa, whoa. Can I get a relation successful New York City instead? It doesn’t substance which team. Is determination thing we tin find for maine to bash here?” So I joined the New York bureau instead. Very, precise antithetic projects, very, precise antithetic group. And determination I realized that not each of Google had this regimented attack to decision-making. There is truthful overmuch translation, adjacent astatine a spot similar Google, that’s indispensable for products that are little adjacent to the gross stream.

So past determination has to beryllium a batch much speech astir wherefore and however to bash assets allocation, and who’s adjacent successful complaint there, right? Things that erstwhile you’re moving billions astir astatine the click of a mouse, you thin to person those questions answered. But successful these different parts of Google, determination was truthful overmuch much colour successful however you could attack it, and specified a large chasm betwixt the radical tasked with that and immoderate of the information oregon engineering oregon information subject efforts we mightiness have. 

So to truly effort to capable that spread — to effort to enactment a span connected it, truthful that things could beryllium utile – I worked mode much than my ceremonial occupation said I should to effort to physique infrastructure. I built aboriginal statistical consulting, due to the fact that that wasn’t there. You couldn’t conscionable spell inquire a statistician who’d beryllium down with you and speech done what your task was going to be.

I convinced radical to connection their 20 percent time, stats radical by specialization, to connection their enactment connected projects that were not their ain project, to enactment immoderate operation to this, and made resources and courses for decision-makers for however to deliberation astir dealing with information folk. I truly tried to bring these 2 areas together, and yet it became my job. But for the longest time, it wasn’t. Sometimes I faced questions. What are you? Who are you? Why are you really doing what you’re doing? But conscionable seeing that things could beryllium made much effective, and kinder, for the experts who were going to enactment connected poorly specified problems unless you specified the problems good first, was motivating, truthful that’s wherefore I did it.

Trying to necktie this each together, it sounds similar that values and goals piece, and the doctrine constituent you talked astir successful schoolhouse arsenic being important, were coming backmost into play versus conscionable focusing connected the outer expectation, similar going to enactment for Google, of course, you’re going to spell to Mountain View. That’s wherever the powerfulness is. That’s wherever the information radical go, and you’re astute capable to beryllium with the information people. 

So if you’re going to tally the car arsenic accelerated arsenic possible, you’re going to spell implicit there, but you made a antithetic benignant of determination than possibly the nine-year-old Cassie made. You stepped backmost and said, Wait a minute, what’s going to beryllium champion for me? And however tin I enactment wrong that portion pulling successful immoderate of this different information?

Yeah, for sure. I deliberation that thing that we tin accidental to your 17-year-old is that it’s okay. It’s good if it’s hard erstwhile you’re young to instrumentality banal of what you really are. You’re not formed yet, and possibly it’s good to fto the upwind instrumentality you a small bit, peculiarly erstwhile you person a large dada who’s going to springiness you large advice. But it would beryllium bully if you tin yet mature into much of a wont of saying, “Well, I’m not the mean Joe, truthful what bash I really want?” And moving for what is thought of arsenic — I don’t privation to offend immoderate interior Googlers — but they did person a estimation for being the apical teams.

If you wanted to beryllium fig 1 and past fig 1 again and fig 1 immoderate much times, that would’ve been the mode to bash it. But again, possibly it’s worthy having thing other that you optimize for successful life. And I, arsenic it turns out, I’m a theatre kid, a lifelong theatre kid. I’m an implicit nerd of theater. I’m going to London for conscionable a fewer days successful 2 weeks, and I’m seeing each evening amusement and matinee. I’m conscionable going to hoard arsenic overmuch theatre arsenic I tin for the soul. And truthful surviving successful New York City was going to beryllium conscionable a amended fit, not lone for theatre but for truthful overmuch much that that metropolis has to offer. 

Having lived successful some Silicon Valley and the New York area, I committedness you that yes, the theatre is acold amended successful New York.

I mean, I went to each the plays successful Silicon Valley arsenic well, and I did my homework. I knew what I was getting into oregon retired of. But yeah, it takes signifier and accomplishment to cognize that immoderate of those questions are adjacent questions worthy asking. And I’ve developed that signifier and accomplishment from primitively knowing however to bash it to assistance others, having studied it formally, being publication astute astir it. These are the questions you ask. This is the bid you inquire them in. It’s thing other to crook that connected yourself and inquire yourself the hard questions, that publication smartness isn’t capable for that.

That’s bully proposal for each of us, whether we’re moving businesses oregon conscionable trying to fig retired life, we’ve each got decisions to make. Cassie Kozyrkov, laminitis and CEO of Kozyr, erstwhile main determination idiosyncratic astatine Google. Thanks for joining maine connected this occurrence of Decoder.

Thanks for having me, Jon.

_{Questions oregon comments astir this episode? Hit america up astatine [email protected]. We truly bash work each email!}