Presenter: Erik Brynjolfsson, Jerry Yang and Akiko Yamazaki Professor; Senior Fellow and Director, Digital Economy Lab, Stanford Institute for Human-Centered AI
Category
🤖
TechTranscript
00:00Good afternoon.
00:01I'm delighted to be here to share some of the research we've been doing at the Stanford
00:05Digital Economy Lab and also at my company, Work Helix, which is trying to take some of
00:10that research and put it into practice.
00:13There is a lot of excitement about AI.
00:14You mentioned maybe as big as the Industrial Revolution.
00:17I think that's true, but there's also a lot of hype out there.
00:20There's a lot of misdirection.
00:21There are so many examples of AI being applied to situations that just aren't going to work.
00:27And as a result, business value is not being created.
00:29So one of the things I want to do in the next 15 minutes or so is share with you some of
00:33the research we've been doing to provide a more data-driven approach to understanding
00:37where the real opportunities are, and I think they're enormous, and also what sorts of things
00:42to avoid.
00:43So let me just start with some technological benchmarks.
00:46Every year, my team and I put out a report called the AI Index.
00:50It's about a 520-page report filled with lots of facts and charts like this one.
00:56And as you can see, a lot of technical benchmarks are getting much, much better over time.
01:01Few things in the world improve as rapidly as AI, and the pace of change is improving.
01:06And I put there a horizontal line for what's happening, human-level performance on these
01:11different benchmarks.
01:12And as you can see, in many cases, the AI systems are matching or exceeding human-level
01:17performance.
01:18As an economist, that's a very important benchmark because when you cross that threshold, like
01:23when water crosses the boiling temperature, it's a phase change, and the economy fundamentally
01:28changes when you have a new way of doing something that does it better, as well or better than
01:33humans do.
01:34But it's not enough to have amazing technology.
01:37We're not translating it to business value.
01:39Mary Daly talked about some of the productivity gains we hope to see, but that's not going
01:43to happen unless we transform work.
01:46Simply buying technology almost never turns it into business value.
01:51One of the most common questions I get is, are we just going to see the end of work,
01:55end of jobs, as this technology replaces people?
01:57That's also not what's going to happen, at least not anytime soon.
02:01Jeff Hinton, one of the greatest minds in AI, pioneered deep learning techniques, and
02:06one of the reasons all those lines are growing so rapidly, a couple of slides earlier, is
02:10because of Jeff's work.
02:11He also has a side gig as an economist sometimes and predicts that we won't need radiologists
02:17anymore.
02:19If you look back in 2016, well, actually, we have more radiologists now, almost twice
02:25as many job postings.
02:27Now why is that?
02:28There's two main reasons for that.
02:30One is that, paradoxically, sometimes when you get better at something, when it becomes
02:34more productive and more efficient, instead of having fewer workers, you have more.
02:40Now why would that be the case?
02:41Well, think about jet engines.
02:43They made airline pilots vastly more productive, right?
02:46More passenger miles, lots of metrics went up.
02:49Do we have fewer pilots or more pilots than before jet engines were invented?
02:53Obviously, most of you flew in here, and jets are a big part of the reason for that, and
02:58with many other technologies, you have what's called Javan's paradox, that as the technology
03:02gets better, it actually increases the demand even faster.
03:06Not everywhere, but in some places, and you have to understand where.
03:09The second reason is that while AI can do many tasks, it can't do everything.
03:13You still need humans in the loop for almost every category of work.
03:18In fact, the right way to think about it is to break it down to what we call the task-based
03:22analysis, where the task is the fundamental unit of an organization.
03:27Think of it as like the DNA base pairs for organisms.
03:30Tasks are that fundamental unit of analysis for organizations.
03:34For over a decade, I've been analyzing companies using the task-based analysis, where you take
03:39a job, which is a bundle of many little tasks, and then you analyze each of those tasks individually.
03:45You see whether or not AI, Gen AI, robotics, or some other tool can help them.
03:50So lifting a box, Gen AI is not so helpful, but as we'll see later, robotics can help
03:54with that.
03:55Writing a memo, well, actually, that's one where Gen AI can help.
03:57And then there are other tasks where no technology currently can help a whole lot.
04:02Once you've done that analysis at the task-based level, you can aggregate them back up to the
04:07job, weighted by wages, or even to the entire organization.
04:12Now you're getting somewhere.
04:14Now you have a roadmap to understand where is this technology going to help, what specific
04:18ways, and instead of having anecdotes and stories from vendors and things that you read
04:24in places, you're going to have a data-driven approach to understanding where the opportunities
04:30lie.
04:31So we did this in a series of academic articles.
04:34I wrote one in science about eight years ago.
04:37My colleague Daniel Rock had one come out in science two weeks ago, where we used this
04:41approach to understand how the economy is going to change.
04:44Now, AI is not AGI, which means it can't do everything, but it can do certain things incredibly
04:50well.
04:51And let's take the case of radiology that Jeff Hinton brought up.
04:55In our initial analysis, we broke radiologists into 27 distinct tasks, and one of them, machine
05:02learning could do extremely well, interpreting images, reading the medical images, better
05:06than humans in many cases, and there's academic papers showing this.
05:09So that's great.
05:10But there are many other tasks that humans, you want to keep humans in the loop.
05:14You don't want to turn over the keys to an AI system to start sedating people or do some
05:19of the other things that are on this list.
05:21And this is exactly what we found in every occupation we looked at.
05:24We found some tasks where AI could help, other tasks where you really want to keep humans
05:28in the loop.
05:29Not once did we say AI run the whole table and just replace an entire occupation.
05:34So that is what you're going to be seeing in your organizations.
05:36You're going to see opportunities to use AI to help, but mostly as a complement, as
05:41a partner for humans to do it better.
05:45When Daniel Rock, used to be a student of mine, now he's a professor at Wharton, did
05:48this for Gen AI, he and his team found on the horizontal axis here is wages, vertical
05:54is the implication.
05:55You can see a sort of upward sloping.
05:57What that means is that actually, unlike some of the earlier technologies, Gen AI disproportionately
06:02affects some of the higher wage occupations.
06:05So a lot of managers, salespeople, doctors, lawyers, for the first time, they're going
06:10to really have their jobs affected, changed, transformed.
06:14And that actually is good news.
06:16It's where a lot of the productivity gains need to happen, and it also will help with
06:19income inequality.
06:20Let me give you a concrete example so you understand a little bit about how this plays
06:24out.
06:25We did a bunch of case studies of AI being rolled out in different situations.
06:29One of them was in a call center, a customer service.
06:33And this company, Cresta, did not try to replace call center operators.
06:38It would be most of it get kind of annoyed when you talk to a robot.
06:40Instead, they had humans talking to the customers.
06:43But the AI helped them by suggesting possible answers.
06:47And the human could use that answer or not use that answer.
06:50We had almost the perfect natural experiment to do causal estimates.
06:54Half the people got access to the technology, others did not.
06:57We looked at over 5 million conversations, 5,000 call center agents, and we were able
07:02to very cleanly and quickly see what a difference it made.
07:06The red line there is improvements in performance of the people who had access to the technology.
07:10And you can see within just four or five months, they went from about two resolutions per hour
07:15to over three per hour.
07:18The folks who did not get access to the technology didn't improve nearly as fast and never got
07:22to the same level.
07:23So this is a very clean causal estimate of the effects of large language models in a
07:27particular application.
07:29What's more, we tracked a dozen other KPIs.
07:32We looked at customer satisfaction, we looked at customer sentiment, basically the ratio
07:36of happy words to angry words in those transcripts.
07:39We looked at employee turnover.
07:41All these metrics also went up by comparable amounts.
07:44So stockholders were happier, the productivity was up.
07:48Customers were happier, more customer satisfaction, higher sentiment.
07:52And even employees, this was not an electronic sweatshop.
07:54Employee turnover went down, the employees seemed to be happier working with the system
07:58as well.
07:59So this was a situation where you could actually benefit all the groups.
08:01It was not a zero-sum implication of this.
08:04And interestingly, as I hinted earlier, the less skilled workers actually were the ones
08:08who benefited the most.
08:09And so this closed income inequality a little bit, and it was a big opportunity to do that
08:13better.
08:14Furthermore, we found that it did not de-skill the workers the way we were worried about.
08:20There were, as all systems, sometimes the system went down.
08:23And that's not great news for the customer, but it turned out to be another great natural
08:27experiment for us researchers.
08:29When the system went down, we continued to monitor the performance of those workers.
08:33And to our surprise, the workers who had been working with the system continued to outperform
08:38the ones who hadn't.
08:39They had learned.
08:40They had internalized some of those answers.
08:42They didn't do it as well as when they had access to it, but they had captured some of
08:46that knowledge.
08:47So it turns out to be a very good way to capture tacit knowledge and translate it over into
08:52people who hadn't otherwise gotten that.
08:54And this is the first time we really have a technology where you don't have to write
08:58down every single rule for what you want your workers to do.
09:01Instead, the LLM or the Gen-AI solution will capture some of that knowledge and transfer
09:06it to other workers.
09:09This opens up a whole new set of opportunities that previously didn't exist.
09:14Now I mentioned that you want to keep humans in the loop, and this graph shows one of the
09:17reasons why you want to do that.
09:19Machine learning, Gen-AI, is great when you have lots of data.
09:23But you don't always have lots of data.
09:25There are some tasks that come up over and over, those to the left of the chart there,
09:29like how do I change my password?
09:31How do I log on?
09:32You get those questions over and over, and you get to learn what's the best way to explain
09:35that answer.
09:36But there are other solutions or other questions at the far right of the tail that we only
09:40see one or two times or a small number of times in the data set.
09:43Well, Gen-AI doesn't have a chance to learn how to solve those, but we humans actually
09:48are pretty good at extemporizing, at dealing with exceptions.
09:54And so there's a natural division of labor.
09:56You have the machines do the types of projects and the types of questions where you have
10:01lots of data.
10:02They can learn better and better solutions.
10:04But exceptions are better done by humans.
10:07We're seeing this play out right now with self-driving cars where they work for 90,
10:1195, 99.9% of the tasks, but there's these exceptions that humans still need to intervene
10:17in.
10:18And eventually they'll get there, but in most of the projects we looked at, there's a natural
10:22division between humans doing the exceptions and machines doing the more frequently done
10:27things.
10:28Now, the example I gave was in call centers, but it was striking to us how that same pattern
10:34showed up in all these other applications, in coding, writing, management, diagnosis.
10:39In each case, we saw often double-digit gains in productivity, and coding is actually triple-digit
10:44gains in some cases, 100% or more productivity gains.
10:48We saw less skilled workers benefiting more, and we saw the performance happening relatively
10:53quickly.
10:54So while there's a lot of hype out there, there are definitely some situations where
10:57Gen AI in particular can very quickly lead to productivity gains, and eventually, as
11:01Mary was hinting, that will aggregate up to the whole economy, and we'll see better outcomes
11:06in the economy more broadly.
11:07Now, I'd like to give you guys a little bit of a roadmap so you can get to work right
11:12away and put some of these in place.
11:15Part of my mission is to shorten that time between technology and productivity, between
11:20AI and business value, and to do that, you have to change the way work is being done.
11:25And so Work Helix, the company I started, is very much geared toward shortening that
11:29time.
11:30And I should mention, I think over here, we've got our CEO, James Myland, who was nice enough
11:33to come join us here.
11:35So feel free to talk to me or him later.
11:37But basically, what we do is we do this task-based analysis.
11:39We put it into software.
11:40And this is something you can do.
11:41You can do it by hand, but it's a lot easier to do with organizational software, because
11:46we look at about 200,000 individual tasks.
11:49And then we roll them up.
11:50Every company has a different fingerprint.
11:53And so this particular company, you can see software engineers are the biggest opportunity.
11:57That's actually pretty common, but not always.
11:59Others are maybe call center, management, sales.
12:01And you see all five of those green dots there are places that are pretty ripe for applying
12:07the solutions.
12:08You can double-click and get in deeper about what are the tasks specifically that are involved
12:12in that, like monitoring system operation or providing advice on project costs, and
12:18look at exactly how much of an acceleration might be possible, how many hours per week
12:23people are doing those, what the dollars spent on it is, and where the biggest gains are
12:27likely to be.
12:28In this way, instead of having just anecdotes to try to decide where to prioritize, you
12:33get a roadmap of what's more important to focus on.
12:37And this was all done, I should say, these initial analyses we do with external data,
12:41so things like LinkedIn, Burning Glass, other data sources that we license.
12:45We already kind of know roughly how many software engineers there are in your company.
12:49As you can imagine, that stuff is posted on the open web.
12:52And so if we license it, we can see that.
12:56But we can also do a deep dive, which is an internal scan, which hooks up to the HRIS
13:00systems.
13:01And that allows you to get a more specific understanding of what's going on.
13:04And I encourage you to do these kinds of analyses on your own to understand what are
13:08the specific tasks that are being done in your organization and which of those are you
13:12likely to be able to have some sort of either a gen AI solution or predictive AI or robotics
13:18solution for.
13:20So what should you do tomorrow morning?
13:21Well, the first thing, you want to develop a plan.
13:24Instead of just picking and choosing a few cases based on the things that your fellow
13:30executives are pitching at you or vendors are pitching at you, do it in a systematic
13:34way.
13:35Use the task-based analysis to understand where the opportunities are.
13:38And then, secondly, you want to start tracking progress.
13:41You want to keep track of those different KPIs, like I did, customer satisfaction, customer
13:46sentiment, time handled, employee turnover.
13:51The set of KPIs is going to be different for every project.
13:54But if you do that, you're going to be able to turn these amazing technologies into productivity
13:58and avoid some of the pitfalls and some of the hype that you also otherwise might fall
14:03into.
14:04If you want to learn more about this, a lot of it's on my website at the Stanford Digital
14:08Economy Lab.
14:09We have all those research papers that I mentioned there that you can read and get an understanding
14:13of the task-based approach and, more generally, how AI is transforming work.
14:17I also have some on my personal website.
14:19And also, the WorkHealX website goes into more depth.
14:21So thanks a lot.
14:22I'm looking forward to talking to all of you later this afternoon.
14:25I'll be around until dinnertime and be happy to chat with you about how you're doing in
14:31your gen AI transformation and your journey.
14:33Thanks very much.