Susan Emerson, Senior Vice President, AI Product GTM, Salesforce Raj Seshadri, Chief Commercial Payments Officer, Mastercard Jeremy Wacksman, Chief Operating Officer, Zillow Group, Emma Hinchliffe, Fortune
Category
🤖
TechTranscript
00:00Hi, everybody. Thank you so much for joining us for lunch today.
00:04Hope you're getting to eat some food and chat.
00:07My name's Emma Hinchliffe. I'm a senior writer at Fortune.
00:10And yeah, thank you so much for being here.
00:12So we are here to discuss the question, do you really know your customer?
00:16With a great group of speakers. But before we get started, I want to make
00:20sure I thank our sponsor for today's session, Salesforce.
00:22So thank you so much. Some housekeeping, this is on the record.
00:26And obviously, we want it to be as interactive as possible.
00:29So any time you have a thought, question, just raise your hand.
00:33If you want to speak, you can just press the button with the face second from the
00:37or second or third from the right on your microphone.
00:39When you're done speaking, make sure to turn it off because only four are allowed
00:41to be on at once. But yeah, we'd love to hear from you at
00:43any point in the conversation. So please chime in.
00:47And please make sure to state your name and company before asking your question.
00:50So now I'd like to introduce our amazing discussion leaders.
00:53We have Susan Emerson, Senior Vice President, AI Product GTM at Salesforce.
00:57Raj Sasadri, Chief Commercial Payments Officer at MasterCard.
01:01And Jeremy Waxman, Chief Operating Officer at Zillow Group.
01:04So thank you all for being here. Yeah, maybe let's just start if each of you
01:08could share who is your customer. Susan, do you want to start?
01:12Who is our customer? Well, hopefully, it's all of you in the
01:18room. I mean, I'm sure many of you know Salesforce
01:23in terms of our set of capabilities around sales service, marketing, commerce,
01:28analytics, and data. There's just a lot of different ways to
01:32participate in our ecosystem. And, you know, what I would say just around
01:37this topic, like, sort of the founding roots of our company was putting your
01:42customer at the center of everything. And then we surround it with all the
01:46capabilities we have at the Salesforce. So we know that customer.
01:49We can engage with that customer. Our sales teams and marketing teams are
01:53all aligned. So who is our customer?
01:55It's the B2C customer. It's the B2B customer.
01:58It's organizations with more complex distribution chains that go B2B to C, and
02:03hopefully all of you in the room. Raj, what about you at MasterCard?
02:10So we are a B2B to C company. So the middle B is our customer, but their
02:15customers, consumers, which could be consumer small business or corporates, by
02:20the way, the NC, are indirectly our customers. But the middle B is our
02:24customer. So it's, you know, and we have, we're in the payments business.
02:27We're in a variety of services business. And across all of those, that middle B
02:31could be an issuing bank. It could be an acquiring bank.
02:35It could be, you know, one of the payment facilitators.
02:37It could be a merchant that sells you something, online or offline or both.
02:41It could be a, you know, it could be one of the travel providers.
02:45It could be, you know, a corporate of some kind, healthcare providers.
02:50There's a variety, governments. So that middle B, we serve a number of
02:53different customers, fintechs in that middle B, but we really focus on them.
02:58They are our customers. But of course, given what we do to help them run their
03:02businesses and engage better with their consumers, the NC also indirectly becomes
03:07our customer.
03:09Yeah, Jeremy, what about you?
03:11Yeah, so Zillow Group, our customer is the consumer. So hopefully all of you as
03:15buyers, renters, sellers, and homeowners. So we run a two-sided marketplace.
03:18We've been around for almost two decades now. I think most people in this room
03:22know and hopefully love and use our app to dream and shop and maybe look at your
03:26neighbor's home or look at your Zestiman. Maybe what you know less is that the last
03:30five or six years, we've built a set of services based on technology to go not
03:35just from dreaming and shopping and searching to actually buying, selling,
03:38and renting. And that's led us into mortgages, and that's led us into agent
03:42software. That's actually led us into payments in the rental space.
03:45So the consumer in our marketplace is sort of our North Star and our primary
03:49customer. But because of that transaction focus, similar to you all, we have an
03:52indirect, we have a B2C2B in many ways, where we're providing software,
03:56technology, and services to agents, to loan officers, to industry professionals
04:00to help the consumer get the job done.
04:03Interesting. So we're talking about, do you know your customer? Obviously,
04:07there's been a lot of development in all the ways you can do that over the past
04:10few years. So what's something, if anything, that you all have learned about
04:14your customer in the past year or two with new technology that maybe you
04:17didn't know five, ten years ago?
04:23So, you know, it's interesting. It's more of an evolution than anything
04:26dramatic, right? So when you think about it, we've been knowing our customer and
04:31knowing their consumers for...we've been in this business looking at data and AI
04:35and deploying it within payments or services for about two decades.
04:39So it is no...it's...what's true today is we know more about customers and
04:45consumers. It's more real-time, it has more nuances that you can measure,
04:51and it's more actionable increasingly. But I wouldn't say it's a dramatic,
04:55you know, it's all of a sudden we know something. It's been an evolution over a
04:58couple of decades for us.
05:00Any examples of what has been actionable, things you can now take action on that
05:04you couldn't before?
05:05Sure, there's a lot. You know, so we see something like 140 billion transactions
05:10in 210 countries every year. So we've been, you know, cleansing it, housing it,
05:14linking it, bringing third-party sources, creating...using AI, analytics,
05:19machine learning for decades. So things that are actionable immediately,
05:23you know, things are things like...we deploy this into our fraud model.
05:27So we deployed machine learning into our network many years ago.
05:30Over the last three years, we've stopped about 35 billion in fraudulent
05:35authorizations. It's things like loyalty. We provide services to help our partners
05:40bring the consumers into the loyalty programs to, you know, shop more often,
05:44to buy more. It's things like insights or macroeconomic insights that the MasterCard
05:48Economic Insight team puts out. Spending pulse, which measures not just card spend
05:54but spending the economy from a consumer across card, check, cash.
05:58So the health of the economy, the health of verticals, the health of geographies
06:02that we work with, you know, with customers as well as helps you forecast
06:06financials a lot better. But it also helps, you know, economics teams,
06:11central governments understand the economy better. So a variety of these things are,
06:16you know, more, like I said, deeper, more real-time, and more actionable now.
06:20Interesting. Yeah, I can sort of play off that with a similar theme.
06:24I've been at Salesforce for going on 15 years, and I can remember, well,
06:29we have a very strong DNA in terms of customer success and working with our
06:33customers, so they can really use all our platforms in anger. And a lot of people
06:38buy us for our business applications. Like you have a distribution team,
06:41you buy Sales Cloud, and that automates everything for sales.
06:44You have, you know, you service customers, you buy our Service Cloud,
06:48you do everything in that. But there's this very capable platform beneath everything.
06:53And back in the day, meaning, you know, a decade or so ago, our ability to understand
06:59customers' utility of our products, like, did you buy the licenses?
07:03Did you deploy them? Are you using them in the same way your cohort group is?
07:07You know, what are the, you know, are you getting full use of it?
07:11And if we see an opportunity to improve it, how do we deploy?
07:15Now, what's changed over the last bit of time is like what you were saying,
07:19the data, the data depth, the data velocity, the data specificity.
07:25So when we go and look at customers like deploying and receiving value,
07:30there's all sorts of insights we can have to help drive their journey in terms of
07:35what are things that we can train you about our software?
07:38What are areas of the platform you haven't fully exploited yet?
07:41So all of these things that help us lean into our customers so they can be better
07:46with our products and services.
07:48And we do that with data and AI products as well in terms of, you know,
07:53real-time signaling to be able to drive and accelerate adoption and value.
07:59Interesting. Yeah, Jeremy, what about you?
08:00What have you learned at Zillow in recent years?
08:02I mean, on the similar themes on the consumer side, I think we're much more attuned to the
08:08kind of rising consumer expectations.
08:10Like what we all think is novel in technology becomes table stakes and expected.
08:14And so personalization is a good example of that, right?
08:16The cookie-based personalization was novel to a lot of us.
08:20We used a website, you came back, and you got recommended homes.
08:23We've been doing that, and we've been using pre-gen AI AI technology for years to do that.
08:28Well, now gen AI comes along, all of a sudden we very quickly are going to want Zillow to be
08:33different for me than it is for Emily and like be very different.
08:36Remember everything you did and speak to you in a much more intimate way.
08:39We don't have that technology yet, but that's a consumer expectation change that's come.
08:44And our AI teams are using gen AI to try to figure out, well, how do we take personalization
08:48to the next level?
08:49So it's not just remembering what search result you got or remembering your last visit, but
08:53actually having a deeper understanding of you.
08:55Yeah.
08:56What is the consumer expectation right now?
08:58Like what are they expecting to see from you?
09:01I mean, I think we all, I mean, AI is a huge topic at this conference, and I think we're
09:06all waiting for the magic that I don't think is quite there yet for the consumer use cases.
09:11The example we give is, you know, it depends on the tool.
09:14So we use generative AI in visualization.
09:18Many of you probably don't know, we have 3D walkthroughs on a lot of Zillow listings and
09:21other competitors have those too.
09:22That's all generated by AI.
09:23You're just taking photos and panos and gen AI is stitching that together and making it
09:27feel like you can walk through the home.
09:28That's like magical consumer expectation, right?
09:30You should be able to do that.
09:31You expect to do that on your phone.
09:33Just a few years ago, you were happy to just get high-res photos, right?
09:35So that's a leap in consumer expectations that AI has done.
09:39The, like, remembering Emily and being deeply personal for you, that takes multimodal AI
09:44and that takes memory, right?
09:46Those are things that, like, we don't have yet.
09:49And Google this morning was talking about, well, maybe we'll get there.
09:53But today, we're just trying to deploy these models to be a little smarter about you, Emma,
09:57or about me, Jeremy.
09:58So, like, we're not quite there yet on the kind of memory side, but we definitely are
10:01there on some of the applications.
10:02Interesting.
10:03I think we had a couple of thoughts here.
10:04Please introduce yourself.
10:05Hi.
10:06My name is Nat.
10:07I work for GP.
10:09Just to clarify, how do you see customers versus users?
10:12At Zillow?
10:15All of you.
10:16Oh.
10:17I mean, for us, the user, the consumer, is our customer.
10:20And we're really intentional about that.
10:21Again, when you run a two-sided marketplace, you have to kind of pick who's your primary
10:24persona.
10:25Is your primary persona the C or the B?
10:28And for us, we say the North Star is the C. And then, of course, we work really hard to
10:31make sure the supply side has great tools and technology to serve that customer.
10:35Yeah.
10:36Jeremy, you have a question?
10:37Sorry.
10:38I was going to say, for us, it's both.
10:39Yeah.
10:40Because in the example I gave you around financial forecasting, the customer is the CFO and the
10:46CFO's team at a company.
10:47But there also could be, like, some of our loyalty businesses where we serve customers.
10:51It's really the end consumer that is the user.
10:54So it's both.
10:55I was just going to ask how either of you make that decision in your businesses.
11:02What factors do you consider to decide that, yes, the C is our customer compared to the
11:05B?
11:06I mean, for us, it was a founding principle.
11:09We were founded kind of on the vision and promise of turning on the lights for consumers.
11:14So it really guided the very first product we launched at the company was kind of a turn
11:19on the lights product.
11:20And that's driven a lot of our innovation since.
11:22But every marketplace has to serve both sides.
11:25So it can't be one at the expense of the other.
11:27The best marketplaces can build products that work for both sides.
11:30But for us, it was kind of from our founding.
11:33And for us, at the end of the day, whether it's our customer or the end consumer, the
11:37customer cares about their end consumers.
11:39So it really comes back to the same, you know, in order to help the customer build their
11:42businesses, drive their revenues, increase their profitability, increase their engagement
11:46with their consumers, you know, retain them for longer, acquire more of them, it ends
11:50up being helping them focus on their consumers.
11:53Interesting.
11:54Do we have another thought over here?
11:56Just to clarify, sorry.
11:57The reason I asked the question was because a lot of businesses, you know, when you say,
12:01do you know your customer, and most times customers are defined as people who pay you.
12:07And then the attention in an organization goes to people who pay you and not people
12:10who don't pay you, right?
12:13So that was my question behind my question saying, how do you think about these things?
12:16We think about it both.
12:17Like, you know, we have customers who are like in the classic sense of the sales campaign,
12:22like the economic buyer, but we really focus on the user and adoption as well.
12:26So even within all our core businesses, like on our B2B side, we focus on both.
12:31But then there's big portions of our portfolio that also go direct to consumers in some ways.
12:37So for example, we acquired a company called Tableau a number of years ago.
12:42And Tableau has been really aggressive, not just in the enterprise space, but also in
12:46the academic community and in individual users as well.
12:50So it sort of depends on our business line.
12:53But we, you know, our DNA from the ground up is accelerating everyone's value with our
12:58products.
12:59And so we do take care to understand what those segments and markets are and how we
13:03can activate it, whether it's training or best practices or creating opportunities for
13:09people to learn from their peers.
13:11There's a lot of different things that we do to accelerate it.
13:14Nice.
13:15Did we have someone else who wanted to share something?
13:18Okay, cool.
13:20Susan, when we were speaking before, you mentioned, you know, these particular high-value use
13:24cases that you're seeing.
13:25Could you tell us a little bit more about that?
13:26What are the highest value ones?
13:28Yeah, I guess specific to this conversation, because we have a big portfolio, and that
13:31could go, like, as long as the cows come home.
13:35But relative to this topic, like, we obviously we've been in the machine learning and the
13:43AI space for over a decade now.
13:46Like, we launched it formally under an Einstein brand back in 2016.
13:50And in this conversation, what we've been up to, you know, for that time, up until GenAI,
13:59is using various techniques with machine learning to help understand customer experiences.
14:05And we've had this really unique opportunity.
14:07Like, we're grounded in these product clouds, sales cloud, service cloud, manufacturing
14:14cloud, financial services cloud.
14:16And so these things are built out to know a data model, the process, all sorts of users
14:21and personas that get up every morning and spend their whole day in it.
14:25So we have this opportunity to really drive opinionated, value-added use cases straight
14:30out of the box.
14:31So in the case of knowing your customer, it would be things like, in marketing, what is
14:37a great send time and optimization that you don't have to build, you just turn on?
14:41What is a good customer engagement score?
14:43So you get the, you know, sort of the right amount of touches so you don't blow out your
14:46contact database.
14:48Or who is a high propensity in terms of signaling engagement that you should put on a different
14:53journey dynamically?
14:54These are just things that work.
14:57So that's sort of one opportunity that we exploit.
15:01And then the other is, like, most people, like, people come to Salesforce for all these
15:06out-of-the-box applications.
15:08But the platforms behind them are very, very composable and flexible.
15:12And so there's a number of things that people can do to build their own machine learning
15:17products or build their own churn models or build their own engagement models.
15:21Or to take the stacks that they've already invested in, because they love the predictions
15:25that are part of them, and bring them into our environment.
15:28So we don't really fight over how that AI gets created.
15:32The big thing is, how do we activate it in our suites of products?
15:35So technically, there's all sorts of pathways for exploiting know-your-customer in our suites.
15:42Yeah, interesting.
15:43You know, we've talked about knowing your customer, but what about, like, your customer's
15:48needs specifically?
15:49How do you go from knowing who they are to knowing what they need from you?
15:55I mean, I can start.
15:57We try and get creative in how we listen, especially on the consumer side.
16:00I think the business professionals we work with, they tell us a lot more directly.
16:06If you have a software user, I'm sure you all have a litany of experience at Salesforce
16:10with us as customers giving you feedback.
16:12But consumers don't always do that.
16:14And so you can watch, and you can pick up on data and patterns and model, and you can
16:18survey, but you have to kind of get a lot of indirect observation of, what are they
16:22looking for?
16:23And then we also do a lot of testing.
16:25So if you have an insight that's a research insight or a product insight, you can throw
16:28it in a 5% A-B test, and that's a really good way to learn your customer, too.
16:32Because maybe your team was right, and that takes off as an improvement.
16:36Maybe your team was wrong, and you learned why the hypothesis was wrong.
16:40Knowing your customer is as much about the qualitative and the kind of picturesque as
16:45it is about the data and the kind of trial and error of putting things in front of them.
16:51And I'd say the same thing.
16:53It's direct, it's indirect, some of it is qualitative, some of it is data-driven.
16:57The other thing I should mention is we also use, you mentioned piloting and testing.
17:02We have a platform called Test and Learn that takes the ambiguity out of it.
17:06So it's literally, you do massively parallel testing, A-B testing, and other types of testing.
17:11So you can double down very quickly on the experiments that are working and stop the
17:16experiments that are not.
17:17So as an example, we were working with a food delivery service, so they were testing four
17:22or five offers in the market.
17:25So the question is, what customer need was the biggest for a particular segment?
17:30And it is a segmented approach.
17:31You could actually look at the data and come up with a hypothesis and think about the four
17:35or five offers that might be valid for a particular segment, and then that could be a segment
17:40with many different dimensions, including geography, spending pattern, other preferences.
17:45But at the end of the day, this Test and Learn platform allows you to test them.
17:49And so it's sometimes counterintuitive which one, you get to know the need better by seeing
17:54what the consumer consumes most of.
17:58Yeah, and we've got a variety of things that we do, kind of maybe starting from the B2B
18:04side of things, everything from user research.
18:08We've got a lot of designers that think about the personas that sit in our software.
18:12They're always doing focus groups with companies.
18:15We invite customers into our design partnerships all the time to understand the types of things
18:21we should be building in.
18:22There's a huge sort of voice of customer set of activities we have.
18:27And then there's also like the data-driven side of things in terms of telemetry.
18:32I can't say the word cinnamon either.
18:34Those are like my two not superpowers.
18:38So all those types of activities.
18:41And then you're right in terms of you were talking earlier about data in terms of being
18:46able to see this and respond and pivot, sort of that sense and response cycle in terms
18:52of what people are reacting to.
18:54Yeah, interesting.
18:56Anyone want to chime in with anything or any examples from your own work?
19:00Got some more perspectives?
19:01No?
19:02Okay.
19:03Not yet.
19:04Well, please raise your hand at any time.
19:05Jeremy, you've talked about personalization.
19:08How is personalization changing and what kind of timeline do you see for some of these advancements
19:12we've talked about?
19:13Yeah.
19:14I mean, I think a year or two ago, I would have been more optimistic that we were going
19:18to get to this breakthrough level of at least kind of consumer at scale internet personalization
19:22faster with all the fawning we've all done over Gen AI and its capabilities.
19:27And I think the reality is that's going to take longer.
19:29And I think Google, this morning, I think helped.
19:32I was really interested to hear his thoughts on when does memory show up?
19:36When does multimodal show up?
19:37When does the next set of architecture show up to solve hallucinations?
19:40We need those things to really get to this movie level personalization from these services.
19:45One of the things that's unique about real estate, at least for our category, is you're
19:48not buying a book or plane tickets.
19:51You're spending six to nine months thinking about what you want to buy.
19:55Your preferences are changing, right?
19:58Where everyone starts the journey of what they buy or sell a house, they don't end.
20:01And personalization is really tough in that category, right?
20:04And so where we're actually spending a lot of our time is more on co-pilots and enabling
20:08the agent, the loan officer, the professionals that we all are going to end up working with.
20:11How can we make them better and help them be more personalized and deliver a more personalized
20:15service to you?
20:16That may look like, in technology sense, a Gen AI model that can help do call summarization
20:21or can help do predictive actions for an agent to use software to help recommend better homes
20:25to you or remember what you said to help be a better guide and advisor for you.
20:29That is personalization.
20:30That's not the personalization we might have all dreamt up in a tech forum, but to help
20:34you buy a house better, that's real-world personalization helping you actually get the
20:37job done until we maybe see more disruptive technology comes that could even automate
20:42that away.
20:43So I think for the stuff we all get excited about, it might be a while, but we're finding
20:48ways to use these technologies today to make these things smarter and better for us as
20:53consumers.
20:54Yeah.
20:56What I was going to add to that is you're spot on.
20:59Personalization is coming at a different pace to different parts of the economy.
21:02The place where we see...
21:03So a couple of years ago, we bought a company called Dynamic Yield that is in personalization.
21:07And the place that we see it taking off the fastest is in retail, in online retail in
21:14particular, but also online, offline, digital, physical retail, and with brands promoting
21:19themselves.
21:20So Dynamic Yield, for example, has this product called Shopping Muse, which uses Gen-AI, but
21:25it uses not just Gen-AI.
21:26I mean, at the end of the day, with all these techniques, you've got to blend them in the
21:29right way.
21:30And so it allows a consumer shopping on a site to be able to navigate very quickly to
21:35what they want with a Gen-AI module where you can actually converse with the site and
21:40pull things in.
21:42It's also got visual AI, so if something's tagged incorrectly, it'll tee it up correctly.
21:48It's really interesting.
21:49The Oracle course is our named test pilot.
21:51It's going really well.
21:53And there are several other marketplaces and brands we're working with.
21:58That's where we see it taking off.
21:59The ROI is very measurable.
22:01But I think something like the business you're in, it might take longer because of the complexity
22:05of the sale, right, and the length of the sale.
22:08Yeah.
22:09I mean, the co-pilots sound great for the consumer experience.
22:13That doesn't sound like future.
22:15That sounds like here for you.
22:16So that's terrific.
22:17I was going to maybe point out maybe one of the more unglamorous components of personalization,
22:22and that is actually really knowing who the customer is.
22:25We all have systems of records inside our companies that are inherently siloed.
22:30They were siloed because that's what you needed to build at the time to run that task.
22:35So a lot of the work that we do with organizations is to help unlock the trap value of all those
22:39different applications and then to bring it into focus.
22:43So if Susan with the Salesforce email address and Susan with the Yahoo address and Susan
22:49with the Gmail address and Susan coming in on this channel and Susan coming in through
22:53this service, we can get all those data points around Susan.
22:57So when we're delivering these experiences, whether it's visual, predictive, a recommended
23:02journey, a house, a credit card offer, although I know that's the other side of the business
23:07for you, we really know who Susan is because we've had this opportunity to harmonize all
23:12these notions of who the customer is.
23:17And Susan, what I'd add to that is you spot on.
23:19In the B2B context, I think we want to be our customers, so I know you do this.
23:23We do it with our customers too.
23:24In a business intelligence platform, for example, you can personalize to, it used to be we personalized
23:30to a job family or a type of user.
23:32You can now personalize to the individual user with their help, with their signaling
23:38to you what's most important to them.
23:39So that's where it's coming to life.
23:41Interesting.
23:42I think we had a question or comment over here.
23:44Yeah.
23:45And please introduce yourself.
23:46Yeah.
23:47I'm Helen Karnosopoulos.
23:48I'm a second startup founder, but I'm also a professor at the University of Toronto in
23:53computer science the last 11 years, and I'm putting together a co-pilot camp.
23:59And so for professionals and for my students and for just, I realize there's a lot of opportunities
24:04there and there was kind of a mention of co-pilots.
24:09If we're going to transform our industries, how do we create that education around those
24:15co-pilots and that usage?
24:17So using next best action, right?
24:19I did integrations with Salesforce.
24:21So it was like, how do you get the rest of the user to use these co-pilots that we're
24:28asking our users to use and how do we train on that?
24:31And that's kind of where I just would like to get some more insight on that.
24:36I have a question back.
24:37So to everyone, and specifically you, first it's an apology, like we're all using the
24:42word co-pilot in the space.
24:44And so sometimes you have to take a step back and define what that means, but I guess you
24:48mean an automated generative experience that is some of that.
24:52Yeah.
24:53Yeah.
24:54Yeah.
24:55So a couple of thoughts on that.
24:56And there's Salesforce opinionated thoughts, because that's my background at the moment.
25:01And then maybe a recommendation to some really good people in the education space who are
25:04doing great thinking about how to train our next wave of innovators in our workforce.
25:09So a co-pilot from the Salesforce perspective usually generally means foundationally, like
25:14how do we take utterance that people throw at an application, get the semantic understanding
25:20out of it so we understand the intent.
25:23And then we can create an AI orchestrated plan where we execute all the bits of that.
25:28So it's understanding, it's orchestration, and it's execution.
25:33It doesn't really have to get any more complicated than that.
25:36And then in terms of how to get people to use it, we have an unfair advantage, because
25:41we've got that system of a record and engagement in Salesforce where it's not optional, it's
25:47right on the screen.
25:48Or maybe it's not even not optional, it's proactive, it's not relying on someone to
25:53request it, it's having context to the interaction and suggesting it.
25:58So those kind of techniques of bringing the user experience to wherever people are working,
26:03like even if it's outside of Salesforce, I think those are great techniques.
26:06And then understanding where you want it proactive, user-directed, or anticipated and shoved in
26:13their face, like all these types of things.
26:15And then in terms of the academic stuff, like Ethan Mollick, I'm sure you've read some of
26:19his stuff.
26:20He's really good about talking about how we have to change education, and the work that
26:24he does at Wharton with his business school entrepreneurial class is sort of really next
26:28level for thinking about how to train our next wave of innovators and our four walls.
26:35I think I'd add maybe I can answer it as a tech employer and then as a software provider.
26:41So as a tech employer, we just encourage everyone to start using it.
26:44So we made the tools available, and our development teams have flocked and tried all the different
26:49tools, and we've centralized on ones that folks have used more.
26:52And so you're seeing it in software engineering, you're starting to see it in design.
26:55So just encouraging people to just get their hands dirty and try things, I think was one
27:00philosophy we had.
27:01And then on the software side and kind of our end customer side, we've focused on trying
27:05to constrain the problem.
27:07And so like trying to start small, and what we've actually found, you don't need all the
27:13bleeding edge technologies and models.
27:15When you're doing something specific for a loan officer's software for a set of customer
27:19conversations about a loan, you can actually have a pretty generic or open source model
27:22and just do a lot of tuning on it and build yourself a smaller, you constrain the box
27:27a little bit.
27:28And so breaking the problem down and starting small helped us get something in the market.
27:32And of course, what was in the market the very first time wasn't amazing, but it was
27:35good enough that we got great feedback from our users.
27:38To Susan's point, we also have the benefit of having dedicated users, we can drop it
27:41on their screen and get the feedback.
27:43So you do have to create a market where you can get the learning.
27:47But I do think trying to break the problem down can sometimes help get started.
27:50Interesting.
27:52Oh, yeah.
27:53Please introduce yourself, please.
27:54Hi.
27:55I'm Stacey Simpson.
27:56I'm the Chief Marketing Officer of Athena Health.
27:59We're a health technology company.
28:01So one of the things that we're really focused on is leveraging and standing up and leveraging
28:06our customer data platform, our CDP, to really make sure we're creating a much more seamless
28:12prospect to customer, so all the way through the customer, the onboarding, the long-term
28:17success of the customer, all the way through to advocacy.
28:20And we're finding that as you stand up your CDP, there's more and more use cases, right?
28:25So marketing is standing it up.
28:27Really, we started with looking at the prospect journey, but it has massive implications for
28:31the entirety, positive implications for the entirety of the customer journey as well.
28:37So I would love to hear maybe any thoughts you all have on the role that a CDP has played
28:44with you, for you, and maybe are there use cases or things that you're excited about
28:49with the confluence of all of these technologies coming together?
28:53I think the guidance that Jeremy had in terms of start small, like, really holds true in
28:57the case of a CDP, because that could be a boil-the-ocean thing, especially when you're
29:02talking about every element of prospecting to cash and advocacy, so definitely a page
29:09out of that book.
29:11We do the drink-our-own-champagne thing as well.
29:15We were customer zero for our own CDP, and our team, they started small, too.
29:22They started with the use case of know the customer so we can get them into the right
29:26journey that accelerates their knowledge and skill set.
29:31And for us, like, we had an ecosystem of over 60 different touch points that we had
29:37to rationalize, and it's everything from, you know, they log on to Salesforce, do they
29:42come to an event, do they go to a webinar, do they do an online trail, do they respond
29:47to a...
29:48And so, like, that was our first foray, is, like, let's find out who these folks are.
29:54Let's make sure we can use that to then journey them.
29:56And then it's really progressed, and that was, like, several years ago, but, you know,
30:00the guidance our team has, I wasn't part of that team, is they're, like, start reasonable,
30:06have a hypothesis of value, and then add on from there.
30:10Yeah.
30:11Do you guys have anything else to add on that one?
30:14No?
30:15That's great.
30:16Okay, cool.
30:17Anyone else?
30:18Well, when you were talking about personalization and how you haven't seen as much progress
30:21as you might have expected a year or two ago, can you break down some of the reasons
30:26for that?
30:27What are the reasons that things haven't progressed as fast, and what are some of the things you
30:31might have expected to see by now?
30:32Yeah, and, I mean, I think context by comments, you know, that I think it's the excitement
30:37we all have for what's capable, what we see in the movies, that's what we haven't seen
30:41yet.
30:42I think we've seen dramatic rises in personalization, and you've heard some of the experiences of
30:46where it's already working today.
30:48For me, it's the, we have a unique challenge in our business, maybe, in that as a consumer
30:54marketplace you only have so much data about the customer, because you're trying to get
30:58out of their way.
30:59And so you're trying to be smart and invent ways to learn more about them without forcing
31:02them to give you a lot of information.
31:04And so as the expectations of us as consumers rise, well, how can an app like Zillow be
31:09magical for you, but you're still barely telling us anything about you?
31:12That's the real trick.
31:13And that is where I think the power of generative AI technologies and more data and bringing
31:18first and third party data together into these models can get better.
31:21But it's why our recommended homes for you are still only as good as how much you're
31:25using the site today.
31:26So that's, like, I'm just more excited about the promise of what can come over the next
31:31couple years.
31:32Yeah.
31:33You know, I would agree.
31:35It's sort of the size of the problem and the depth of the data that sort of determines
31:39how near term it is versus medium versus long term and what you can turn on.
31:44So the example I was giving you in retail, it's a constraint problem.
31:47You're going to shop off an inventory that a retailer has.
31:51And the depth of the data, if it's an undisclosed customer, they may have turned on, you know,
31:55their location inside.
31:56So you might be able to see sort of what they've been looking for before.
31:59But if they're disclosed, if they're a loyalty customer, suddenly you have a lot more information
32:03on them.
32:04So the depth of the data also makes a difference.
32:05So it's a spectrum even in a constraint problem.
32:10But then when the problem set gets to be bigger, it's more complicated.
32:15And then those decisions about where you put the theoretical paywall in terms of not actually
32:21a paywall, but, like, that blend of the user experience and when you're willing to give
32:26more data.
32:27That's right.
32:28Jeremy, to your point about not asking the consumer to give you too much information,
32:33what do you guys sense about how the consumer or your customer is feeling right now when
32:38it comes to privacy and giving up their information?
32:41Do people want more personalization?
32:42Are they feeling reluctant?
32:43Are they feeling spied on?
32:44Like, where is the consumer mindset today?
32:46I mean, I'll start.
32:49I think privacy is always rooted in customer expectations and what we all have as expectations.
32:55And you can just think about apps and services.
32:57My favorite example is Uber.
32:59Like when Uber and Airbnb came along, the initial reaction was how creepy that is.
33:02And now we get into a stranger's car and we don't think twice about it.
33:05That's a change in customer expectations, which means you were willing to give up a
33:08lot more privacy.
33:11But I think in the era of a lot of data security issues and a lot of GNI issues, I think the
33:16expectation is going up that brands and companies are really good at respecting customers' privacy.
33:21And back to Susan's point, it's kind of where is the wall, right?
33:23So if you're just browsing and shopping on Zillow, you probably have an expectation that
33:27you haven't told us too much about who you are.
33:28You're not really a deep customer with us yet and we have to respect that.
33:31We have customers that are getting a mortgage from us.
33:33Like we originate mortgages.
33:35And that customer has gone through an authentication process and has seen that they can trust us
33:39with their data in a very different way than a shopper or a browser.
33:43And so it is about like which use case are you doing?
33:46But to me, it all comes back to like what's the expectation we have as customers?
33:51So this is something we've been thinking about, you know, for a long time, given that we've
33:55been in these businesses for a couple of decades, right?
33:57As data has evolved and technology has evolved and algorithms have evolved.
34:01So it's really, at the end of the day, it comes down to the consumer.
34:05It's a tradeoff between value and security and privacy.
34:10And that's really, at the core of it is trust.
34:14The most important thing is trust.
34:16And so in our approach to privacy and to security, we're very principled.
34:20We think about, and we came out with our privacy and security principles well before sort of
34:25it became the hot topic, you know, back in the 2018, 19 timeframes.
34:30And it really is, at the end of the day, it's very simple.
34:32You own your data.
34:34You decide who to share it with.
34:35You should benefit from it.
34:37And if you share it with us, our job is to keep it, make sure we understand your consent
34:41and to keep it private and to keep it secure.
34:44And to be, at the end of the day, you've got to be transparent about what the consumer
34:48has given you.
34:49And it has to be clear to the consumer how it's being used or what the constraints are
34:52that you're putting on it.
34:54But at the core of it, they've got to get some value back from it.
34:56That's really important because if there's no value back from it, then they won't share
35:00as much.
35:01So that's really at the heart of the question.
35:03It's trust.
35:05And any examples of that?
35:06What value are they getting back from it today in your business?
35:08So it depends on the product.
35:10We have such a wide array of products between our payments and services businesses.
35:14So it depends on the product.
35:15So for example, in the personalization example, we sometimes get customers, you know, as they
35:20use our Gen AI tools, Shopping Muse, et cetera, they tell us more.
35:24That's because they're clearly getting some value out of it.
35:26They like the answers that they're getting.
35:27It's helping them navigate quicker to the item that they want.
35:31And then the brands and the retailers like it because getting the customer more efficiently
35:36and quickly to what they want means you sell more, right?
35:39So at the end of the day, so that's a measure of value in that situation.
35:43But that value equation changes depending on which product, which use case you're looking
35:47at and who the end customer is.
35:49But everywhere, universally, the principle of you've got to think about the value and
35:52you've got to think about protection simultaneously.
35:54That's what creates the trust.
35:56Interesting.
35:57Anyone in the room have anything to add?
36:00Well, you probably all have experiences almost every day on some website where, like, you're
36:05investigating and researching something and at some point it's like, oh, and put your
36:09email and phone number to get the text and the email, and you're like, nope, out.
36:14It's like that sort of mindset, like, where's that nope, out, like, thing?
36:17I'm looking at solar panels for a roof and, like, everyone wants my address.
36:21That's a nope, out for me right now because I haven't progressed into that next stage.
36:26Yeah, interesting.
36:27Do you guys have personal examples of that?
36:29Where's your paywall or your line?
36:31I mean, we do, and I think that, yeah, Susan gives a good example.
36:35We have that same, you can use us pretty anonymously.
36:38You can start to give us information, and the value we give you back is that we can
36:42personalize the site and recommend things for you, and then at some point you actually
36:45might raise your hand and want help in a transaction, and once you do that, you might have a real
36:50relationship and you might talk to a human at our company or a professional that we've
36:53connected you with, and there's a harder line of information exchange there, but the
36:58consumer has opted into that, and then they expect a much higher level of service, right?
37:02Then they expect we remember that, and when they come back the next time, we don't, like,
37:06we get smarter based on that, so it is use case dependent, and for us, it's a little
37:11bit more like a funnel, and it's more like how far down the funnel are you, and then
37:14are we giving you enough value so that you want to keep going with us?
37:18Interesting.
37:19Marian, please.
37:20I'm interested to hear about the intergenerational divide in your customer base and also in your
37:26employee base, right?
37:29Because I, as a 20-year CMO, I see it, and I'm sort of curious to see what you guys are
37:34seeing and what you're doing about it.
37:40I was going to say it's there and it's not there, right?
37:44It's there that I think the myth of younger generations sharing more is definitely there,
37:54but they're also looking for value.
37:55I mean, they don't indiscriminately share, so it's there in that sense, and it's not
37:59there in that across all generations, depending on the use case and depending on the particular
38:05need a customer has, they may share, and that's true across generations, so it really depends
38:10on what is it for and what are they getting out of it.
38:15And it's a great question.
38:17I had to think about it for a second.
38:18I think maybe for us in housing, once you get into the transaction, it's such a big,
38:24scary transaction, everyone kind of behaves a little bit more the same, so maybe we see
38:28it more in how do different segments respond to marketing or to category introduction or
38:33to the fun parts of it, but once you get into, like, I need help and you push the help button
38:37and you actually want to buy and sell, I'm sure we see differences, but we have a lot
38:43of the same challenges for all those segments to actually, because we have challenges with
38:47affordability and budgeting and actually winning the house and actually finding a good agent
38:51and actually finding a good mortgage, like, those things are pretty universal, and I'm
38:55sure our approach to the customer segments differs, but some of that falls away once
39:00you get into the transaction.
39:01I think because it's such an infrequent purchase, everyone's kind of a first-time customer,
39:04even if you've been through it three or four times.
39:07It's a really interesting question.
39:08The other thing I'd add is it depends on the, you know, it's interesting you're talking
39:11about buying a house.
39:12In general, when it starts talking about credit and then your own money, it gets much more
39:18conservative versus buying a shirt off some, you know, website where it's, you know, I'm
39:24tend to a, I'm happy to share the relevant information more easily.
39:27And if I can just pivot for a second, because it was about knowing your customer, and obviously
39:31as a marketer, or you have a lot of people doing that job in your companies, the up-leveling
39:37of the skill, because the role of marketing has diametrically changed.
39:41So I'm sort of interested in how you're up-leveling your more senior people, what you're seeing
39:47in terms of the younger people, right, because there's that divide that we're also finding
39:51within our own organizations, not so much as a consumer of trying to buy a house, but
39:55in terms of being able to do their jobs, which is a thing these days now all of a sudden.
40:00I mean, I think back to the question on, like, co-pilots, we are trying, as employers, we're
40:05trying really hard to just make all the tools available and encourage use.
40:07And one of our big core principles is sort of fail fast.
40:11So one of our, we talk about it as move fast, think big at Zillow, and it goes back to Susan's
40:15comment on testing.
40:16Like, we encourage experimentation, have a hypothesis, and try something.
40:19And that applies as much to, like, a marketer creating a campaign as it does to just a marketer
40:24trying a Gen AI tool.
40:26And hey, you used to do it this way, try something off the shelf, and see how it goes.
40:29And so encouraging experimentation and encouraging learning through doing and failure, that's
40:36one of the things we've, I think we've always done, and I think it helps if you're reskilling,
40:40if you're learning new skills.
40:41I mean, marketing is always a creative field that's always changing.
40:45And this is just the next wave of how can marketers use technology to do their job.
40:49And I agree.
40:50It's learning, it's experimentation, for sure.
40:52And the other thing I'd add to that is co-creation.
40:55So you know, I mentioned financial forecasting.
40:57We initially did that, and eating your own dog food, we did that with our CFO, with our
41:00finance team, and then started doing it with customers.
41:04Completely different example, right?
41:06We have a consulting firm.
41:07So consulting firms, to get version one of an answer, you usually have to go hunting
41:11and pecking and looking at different documents and putting things together.
41:14So we have a pilot that we're working on with the consultants that uses AI to generate in
41:20a version one, which they then iterate and, you know, customize and do whatever they need
41:24to do for a particular customer, for a particular conversation.
41:28So it's co-creation with the relevant group of folks on what is it that's most helpful
41:34to them as that will help them do their jobs better, right, and faster.
41:39Yeah, and from my perspective at Salesforce, we're maybe an anomaly in that we're a tech
41:44company and we throw a lot of tech at our employees and we're all expected to use it,
41:49absorb it, sell it, and so forth.
41:51But we do, we are entertained in a lot of conversations of, you know, what does that
41:56future employee look like?
41:58What is their skill set?
41:59How is the skill set changing?
42:00Is it a role?
42:01Is it a task?
42:02Is it a new hire?
42:03Like, specifically around sort of the concept of interacting with large language models
42:08and who's a good prompt engineer, and maybe kind of a joint comment on that, like, in
42:13the field of marketing, so much of that interaction on the creative side, in terms of using these
42:18LLMs to create stuff, is very interactive, and the person with the hands-on keyboard
42:23will need skills with those products to really push them to the edge and fight with every
42:27word and that kind of thing.
42:29But then when you get into an employee base, let's say, of 10,000 people on a call center
42:34or 10,000 people in a distribution channel, you don't want the quality of the question
42:40to be, like, on the shoulders of those people.
42:43You want it proactive in the user experience.
42:46And so then, therefore, that skill set of prompt engineering is just part of the system
42:50baking.
42:52And so it really might depend in terms of what is the use case, but it's certainly
42:57– it's a good thing we have Coursera and all sorts of MIT and Stanford courses that
43:02we can all, like, up-level ourselves, because it's common.
43:05Yeah, that learning and development is important.
43:07We encourage everyone to go, you know, listen, learn, take a Coursera class, you know, learn
43:13from experts within the firm.
43:15And that expert isn't always somebody who's older or more senior.
43:18Sometimes it's somebody, as you said, who's just come out of university and has a new
43:21set of computer science skills that, you know, I may not have.
43:25So it's lifelong learning has to be part of the culture.
43:27Yeah.
43:28But I do think the commonality across our answers is, like, leadership setting the culture
43:31and the tone, which I think is indicative of your question, is, like, we all as leaders
43:34have to set the standard for what we expect, and then you also have to shine a light on
43:38early successes or early trials to set the tone for, like, this is – this should be
43:42the new normal.
43:43This should be celebrated.
43:44Yeah.
43:45It's okay to not get it right.
43:46I agree.
43:47I do think that's the – And the expectation to always be learning,
43:48right?
43:49Yeah.
43:50You know, those curious employees that are –
43:51Yep.
43:52Yep.
43:53Yep.
43:54Agreed.
43:55Yeah.
43:56Thank you for the great question, Miriam.
43:57Anyone else?
43:58Okay, cool.
43:59Well, Raj, there's this idea of when does the human need to be in the loop when we're
44:03talking about AI.
44:04So what's your perspective in your role on when decisions can be automated and when technology
44:10is augmenting those decisions?
44:12Yes.
44:13You know, it's interesting.
44:14We spend a lot of time thinking about this and have spent a lot of time thinking about
44:16this.
44:18Yeah.
44:19For, you know, for many, many years.
44:20And it's interesting.
44:21Yesterday, you know, the – Eric from Stanford was talking about you decompose a company
44:24into jobs and you decompose jobs into tasks.
44:27You remember that whole conversation?
44:28Yeah.
44:29That's really the approach that we take.
44:30So you take a particular type of job – I was talking about consulting earlier – you
44:34think about the tasks, and the answer's not the same for every single task.
44:39There's some tasks that are very repeatable where there's a lot of data where you can
44:43imagine automating it away and then taking it off the hands of the human, right?
44:47And the human's actually quite usually in those situations quite happy to let go of
44:50the stuff if they think about, you know, what they could use their time for.
44:54There are others where, you know, depending on the complexity of the task, how frequent
44:58it is, the type of judgment that's required, it is more of, you know, to borrow a term
45:03but with a very different definition, there's sort of this co-pilot concept where you want
45:08to prompt the human being with the input, like I was describing Gen 1 of a document
45:13for a consultant, right?
45:14Prompt the human being with the input to make their process more efficient and for them
45:18to free up their time – they still very much need it – where, you know, to free
45:22up their time to do the work that, you know, makes it what it is versus the work of assembling
45:28version one of it.
45:29And then there are other tasks that, you know, frankly at this point – so, you know, in
45:34wealth management, I spend a lot of time in wealth management – there's an emotional
45:37side to wealth.
45:38It's not just about the numbers.
45:39It's not just about the portfolios.
45:41It's about talking to the individual about their hopes and aspirations and what they
45:45want.
45:46That at this point, you need a human being to do and it's critically important and I
45:49can – it's your, you know, comment about generative AI and the journey in your – in
45:53real estate.
45:54I think it's going to be a long time before that gets automated or you can suggest things
46:00but at the end of the day, you need the human being there.
46:03So it really depends on the task and each job family often has a mixture of all of the
46:09above, the whole spectrum, right?
46:10Yeah.
46:11Yeah.
46:12And I mean, your last example of wealth advisor, we use that a lot as an analogy at Zillow
46:14for a real estate agent.
46:16It's not why can't tech replace what they do, it's how can tech actually enable the
46:20folks to do what they are good at doing instead of the busy, low-value work that they have
46:24to spend their time doing which I think equally applies to your wealth management concept.
46:27Yeah.
46:28And I would draw very specifically around that with employee-facing human-in-the-loop
46:33with our generative user experiences as a – for a whole bunch of reasons.
46:37That's mostly where people were comfortable.
46:39If you're in a regulated industry, you want that human-in-the-loop because it's part
46:43of your guardrails and it's also part of like not freaking out your employee bases
46:47in terms of like what's changing in your role in your job.
46:50So it was always human-in-the-loop, human at the helm and supporting them in these like
46:55repetitive but low-value tasks or unlocking the data that just wasn't available to them
47:02in an easy way so they just wouldn't use it.
47:04So like how can we make it simple, how can we make it proactive, how can we automate
47:09it?
47:10But we are starting to see like a transition now just like kind of a year and a half into
47:14Gen AI where all that sense and respond that a human might do, like take the example of
47:20the wealth manager, like there might be signal in terms of a high propensity to engage or
47:26buy or portfolio, like transition state in terms of asset classes or size and like why
47:33do you wait for the human set of eyes to look at that?
47:36Why don't we automate that and do some things on behalf of that wealth advisor in that example?
47:41So that sense and response signal that you're comfortable with still may be with human-in-the-loop
47:46but not always relying for that trigger state to be two sets of eyes on the keyboard or
47:51the mobile phone.
47:52And the human-in-the-loop, what I'd add to that, I agree with you, what I'd add to that
47:56is we also do it with the employees, right, so it's human-centric in how we do it.
48:01You don't do it to a particular group of employees, you do it with them because they often, if
48:05they understand the value that they'll get out of it, they embrace it and they run with
48:09it and they'll also tell you what's working and what's not and that helps you iterate
48:13faster and get to a better answer.
48:15Yeah, interesting.
48:16Jeremy, on a similar note, Zillow, of course, you run up against federal law around fair
48:22housing.
48:23Yeah.
48:24So as you've been deploying AI, what have you done and how can you be sure that AI does
48:29not engage in housing discrimination?
48:30Yeah, how much time you got?
48:31It's like a whole other lunch panel.
48:34Second next level.
48:35Yeah.
48:36Yeah, so many of you probably don't spend your time, as we do at Zillow, on all the
48:39challenges with fair housing, but fair housing has been rampant in the real estate industry
48:43kind of forever, which means that a lot of the data sets that come into AI may have that
48:47bias in them, right?
48:49And you don't have to go too far to think about an example of like, could you type a
48:53question into a chat TPT and get back an answer that an agent wouldn't give you because you
48:57can't say, you can't discriminate which house you buy or what neighborhood you look at or
49:01recommend homes based on protected class status, right?
49:04But you could make an AI, you can make a chatbot do that really easily, right?
49:07So we've had to do a couple of things.
49:11We rushed at all the plugins when they first came out.
49:13We're like, oh, let's build a plugin for chat TPT and for Google.
49:16And it took about five minutes for us to see fair housing violations, and so they all got
49:20pulled down.
49:21What were they?
49:22I mean, just asking questions about what's a good neighborhood for such, or where are
49:27the best places to live for this, or those things that aren't really legal to answer,
49:32that a good agent, a licensed professional would not answer, but that you could train
49:37a machine to answer.
49:38And so what we've done is we've actually built a fair housing classifier.
49:41So we've taken all the data in and we've built a classifier to train AI models to basically
49:46not answer those questions and not run afoul of fair housing guidelines, and we recently
49:50open sourced that.
49:51Now, it's early days, but the idea is we have to be able to train models in real estate
49:54to not do that.
49:57And it is both the data coming in, like making sure we're not using biased data in, and then
50:01also testing the output and saying, okay, we've taken the data and we think that's unbiased
50:05data.
50:06But can the model run afoul or not?
50:08And it's a tough problem.
50:09Yeah.
50:10So, sorry, go ahead.
50:11I was going to say, it's a problem.
50:14It's interesting hearing you talk about it, because we have a number of models, it's the
50:17same problem without the regulatory constraints that you're talking about in many cases.
50:20But it's a question of, you know, we spend a lot of time training our data scientists
50:24and our employees and thinking about, you know, what model they're trying to build,
50:28what's the purpose of it, what data are they using, what techniques are they using?
50:32And as they do that, what are the biases that they need to make sure they test for?
50:36And then we also have some automatic testing to keep an eye on it.
50:38What are the unintended consequences?
50:40And then even after you release the model into production, you've got to constantly
50:43monitor it, because it's in an evolving dynamic environment.
50:47So thinking about how you do the modeling is critically important.
50:51Yeah.
50:52And then all the guardrails that go with it, whether it's like, this is the question set
50:56you're allowed to engage in, and no more and no less, and, you know, when that question
51:00is asked, maybe that's the time to insert the human in the loop, or, and then documenting
51:08all these interactions, what was asked, what was responded to, was there toxicity, like,
51:12we sort of architect this all into our ecosystem with this thing we call the trust layer, which
51:17has a number of ways it inserts guardrails, whether it's, you know, looking for toxicity
51:23or, you know, masking data or not storing data on all these foundational models, and
51:30then all these other, you know, ecosystem things in terms of, these questions are okay,
51:35as soon as you get to this, it's human in the loop, that type of thing.
51:40Interesting.
51:41Jeremy, is there anything that, you know, when you talk about kind of trying these things
51:44out, and then after five minutes realizing that it was not quite going to work, are there
51:47things that, you know, if you didn't have these concerns of the dangers of AI engaging
51:51in housing discrimination that you would be doing, things you've had to slow down in your
51:55deployment of these technologies?
51:56No.
51:57I mean, I think we knew this was going to be a challenge when we went in, so it wasn't
52:00like we said, oops.
52:01It was more, how are we going to get to the answer to where we can train these models
52:05to do a better job?
52:06I think, but I do think maybe implicit in your question is sort of pace of AI, and how
52:11do you make sure you launch it responsibly, which Google talked about this morning, and
52:15I do think it is, as Susan said, it is like, what is your trust and safety guardrails?
52:20What are your barriers?
52:21So, we're very focused on our own industry, and we're very focused on how can we build
52:25models that are, help you with the real estate question, which is different than what a generic
52:29horizontal model might do.
52:30It takes our data.
52:31It takes third-party data.
52:32It takes a copilot with a professional to start to add value.
52:37That's not, we're just going to launch like an AskZillow thing on the website, and it's
52:41going to just magically do everything for you, right?
52:42So, it is about, like, it's more about meeting the customer expectations, what you asked
52:47about earlier, than it is about, like, well, just how fast can we go, to me.
52:52Yeah.
52:53Anyone have anything to add?
52:54Okay, cool.
52:55When you mentioned third-party data, for all of you, what about third-party data do you
52:59find useful?
53:00What's not useful?
53:01When do you turn to it, and when don't you?
53:03So, you know, we've been, thank you.
53:06I always forget to turn it on.
53:07Thank you, Susie.
53:08I've just stopped turning it on.
53:09Yeah.
53:10That might be the other answer.
53:11Just always on.
53:12Yeah.
53:13So, we've been using third-party sources for a long, long time, and it really, we use it
53:17only, it has to meet our privacy security criteria.
53:21It has to, we use it only when it enhances, adds nuance to a particular, you know, to
53:28a particular solution.
53:29So, for example, when we project consumer spending across all tender types, we can obviously
53:35easily extrapolate to all cards, but then we have some sources that help us extrapolate
53:39into cash and check, right?
53:41So, that's adding nuance into that data set, so we can then talk about consumer spending
53:46across all categories, across all geographies.
53:49So, we really look at where does it add something, where is it additive, and how is it additive,
53:54but then at the end of the day, it has to meet our security and privacy requirements.
53:59Not just in terms of the source, but also how the source is used, how it is created
54:03and constructed, and then how we use it.
54:05So, it's not just the data itself, it's the whole process by which it's assembled and
54:10used.
54:11Interesting.
54:12And any examples that come up?
54:14What can it capture that you can't with your own?
54:16So, the example I gave you is a great one, right?
54:19Consumer spending, you know, if you want to look at what consumers spent on retail marketplaces
54:25or in New York City, right?
54:28You can't do that without understanding all tender types.
54:30So, cash and check is not a business we're in, but we have sources that help us extrapolate
54:35very accurately to consumer spending.
54:38Interesting.
54:39Susan, what about you?
54:40Anything there?
54:41Well, I just had maybe two comments.
54:42You know, I see what our customers are able to do with our platforms, and in all industries
54:46there's data that can be acquired that helps understand, like, who that customer really
54:51is so they can drive engagement strategies into it, and that's a very broad topic across
54:56many different industries, whether you're buying, like, held-away assets for, like,
55:01a wealth or an asset management firm or, you know, usage of other products and services.
55:08We see, you know, in the tech field, having that third-party data, for example, of, like,
55:15what else does this big organization own, like, in terms – because you can, you know,
55:19acquire these data from data sources.
55:21It does help, like, if you're a B2B, you know, company selling, like, what does that
55:26mean in terms of white space?
55:28What does that mean in terms of next best product?
55:30So, there's always ways to, you know, accelerate and understand what that means that would
55:36then drive the engagement personalization and so forth, but I was speaking earlier about
55:41– thinking earlier about your guardrails and your experience with the Gen-AI in those
55:47early moments.
55:48There was a lot of customers back in the early day, you know, 14 months ago, where it was
55:53just a hard no, we turned the Internet off, right, like, you know, especially if you regulated
55:58in your healthcare and financial services, but what we've seen in just a very short
56:02period of time is, as people understand the, like, the toolkit they have for these guardrails,
56:08whether it's updating the policy in terms of we engage here, but we don't engage there,
56:12or we'll use foundational models as long as our data is safe.
56:16Nope, we have to fine-tune a model because that's who we are, like, nope, we can only
56:19use it for this use case, or no, it's only human in the loop, and we put speed bumps
56:23in the design so no one can fall asleep at the wheel, like, as people start to understand
56:27all these techniques, like, even folks that are in day one, we're like, nope, third
56:31rail are now, like, we know how we can proceed, and that's happened in a really short period
56:36of time.
56:37It's actually interesting.
56:38When it first came out, there were, you know, it's fascinating.
56:40There were the third rail folks, like you said, who said, turn it all off, shut it down,
56:45and then there were the folks who said, no, no, no, go, everyone go embrace everything.
56:49I think both those extremes have their own dangers.
56:52It's to, you know, finding the right balance, the ability to navigate, to understand it
56:55better, to engage, you know, people within the company who can help you think about it,
57:00I think is very important.
57:01Yeah.
57:02Yeah.
57:03Well, thank you so much to all of our speakers.
57:04That's all the time we have today.
57:06An extra big thank you to Salesforce for sponsoring this session, and a reminder that the afternoon's
57:10main stage programming begins at 2.45, so see you all there.
57:13Thanks so much.