WEBVTT - Monologue: Everyone Suddenly Cares About AI ROI

0:00:02.800 --> 0:00:07.000
<v Speaker 1>Zone Media. Hi, Mimo mo ed Zytron, and welcome to

0:00:07.040 --> 0:00:18.360
<v Speaker 1>this week's Better Offline Monologue. It's a triumphant week here

0:00:18.400 --> 0:00:21.279
<v Speaker 1>at the Zyron household. As I mentioned earlier in the week,

0:00:21.400 --> 0:00:25.040
<v Speaker 1>Uber COO Andrew McDonald said and I quote that its

0:00:25.079 --> 0:00:27.800
<v Speaker 1>AI costs were becoming harder to justify and that the

0:00:27.880 --> 0:00:31.160
<v Speaker 1>link was not there between spending money on AI tokens

0:00:31.160 --> 0:00:34.400
<v Speaker 1>and creating more useful features. In the days that have

0:00:34.479 --> 0:00:37.120
<v Speaker 1>follow the AI industry has been embroiled in a conversation

0:00:37.159 --> 0:00:40.280
<v Speaker 1>that mostly comes down to one thing. Hey, does anyone

0:00:40.360 --> 0:00:44.880
<v Speaker 1>have any way to measure the ROI of AI? Didn't

0:00:44.880 --> 0:00:47.320
<v Speaker 1>fucking think of that one digit fellas, Oh no, I

0:00:47.360 --> 0:00:49.000
<v Speaker 1>want to come up with that before I spent all

0:00:49.000 --> 0:00:50.680
<v Speaker 1>the fucking money. But I don't know. I don't run

0:00:50.720 --> 0:00:54.960
<v Speaker 1>a fucking business, drap. That's because up until recently, most

0:00:55.000 --> 0:00:57.480
<v Speaker 1>companies have been able to pay for a subscription with

0:00:57.560 --> 0:01:01.080
<v Speaker 1>subsidized token rates, meaning that forever dollar of their subscription

0:01:01.120 --> 0:01:03.520
<v Speaker 1>they could burn anywhere from three to thirteen dollars worth

0:01:03.560 --> 0:01:06.600
<v Speaker 1>of tokens. They just had rate limits. Whatever they did

0:01:06.680 --> 0:01:09.839
<v Speaker 1>just eight into those rate limits, and they didn't really

0:01:09.840 --> 0:01:12.280
<v Speaker 1>think about the token cost in fact, every one of

0:01:12.280 --> 0:01:16.520
<v Speaker 1>these companies deliberately obfuscates that information. You can find it

0:01:16.560 --> 0:01:19.720
<v Speaker 1>through CC usage, which you need to have command lined interface,

0:01:19.920 --> 0:01:24.039
<v Speaker 1>and there's also slash usage with anthropic Regardless, as a

0:01:24.040 --> 0:01:26.240
<v Speaker 1>result of all of this, nobody was really measuring the

0:01:26.280 --> 0:01:29.680
<v Speaker 1>actual positive outcomes of AI services, or even how much

0:01:29.680 --> 0:01:32.480
<v Speaker 1>a particular task would burning tokens, which means that with

0:01:32.560 --> 0:01:35.720
<v Speaker 1>the advent of token based billing for enterprises, everyone is

0:01:35.840 --> 0:01:38.480
<v Speaker 1>just guessing about how much their annual budget should be

0:01:38.520 --> 0:01:42.800
<v Speaker 1>based on virtually no useful information. Imagine you're doing something

0:01:42.880 --> 0:01:46.040
<v Speaker 1>for years, years and years, and you're claiming it it's

0:01:46.080 --> 0:01:48.160
<v Speaker 1>the future, and it's going to change everything. It's going

0:01:48.200 --> 0:01:50.480
<v Speaker 1>to change your business, and there's other businesses. It's the

0:01:50.520 --> 0:01:52.840
<v Speaker 1>future of your revenue and their revenue. Oh my god,

0:01:52.880 --> 0:01:57.960
<v Speaker 1>it's so amazing. But not once did your clever nbaar say,

0:01:58.800 --> 0:02:02.440
<v Speaker 1>why don't we make sure it actually is? No one

0:02:02.480 --> 0:02:05.360
<v Speaker 1>did that, not one of them, Not one of these

0:02:05.360 --> 0:02:07.320
<v Speaker 1>fuckers have been I've been checking a week, we're talking

0:02:07.360 --> 0:02:11.440
<v Speaker 1>to people a week. In truth, though, measuring ROI is

0:02:11.480 --> 0:02:14.040
<v Speaker 1>pretty difficult for a lot of knowledge work because it

0:02:14.080 --> 0:02:16.280
<v Speaker 1>requires you to actually know what a person does for

0:02:16.360 --> 0:02:18.400
<v Speaker 1>a living, and what doing a job is like, and

0:02:18.720 --> 0:02:21.080
<v Speaker 1>what the outcomes have said job are and what the

0:02:21.280 --> 0:02:24.200
<v Speaker 1>actual like good results at the end may be. Which

0:02:24.240 --> 0:02:27.000
<v Speaker 1>is a lot to ask for middle managers and CEOs

0:02:27.040 --> 0:02:29.560
<v Speaker 1>who mostly go to and from lunch and explain why

0:02:29.600 --> 0:02:31.799
<v Speaker 1>other people's work is actually theirs. And I want to say,

0:02:31.840 --> 0:02:34.600
<v Speaker 1>if you're a middle manager hearing this, more than likely

0:02:34.800 --> 0:02:37.919
<v Speaker 1>I'm completely correct about you. Most middle managers are fucking scum.

0:02:38.320 --> 0:02:41.080
<v Speaker 1>They exist only to steal work from people and nag

0:02:41.200 --> 0:02:43.280
<v Speaker 1>them and harass them. If you're not one of those,

0:02:43.320 --> 0:02:45.360
<v Speaker 1>great you shouldn't be offended by this. This is not

0:02:45.440 --> 0:02:47.799
<v Speaker 1>about you. But if you are, you're probably one of

0:02:47.840 --> 0:02:49.760
<v Speaker 1>the people who's going to email me, in which case, wei,

0:02:50.120 --> 0:02:52.320
<v Speaker 1>weare God, cry to someone else. I don't give a ship,

0:02:52.960 --> 0:02:55.600
<v Speaker 1>but the lack of measurement of both task costs and

0:02:55.720 --> 0:02:59.520
<v Speaker 1>ROI as executives already crying for mercy. An engineer at

0:02:59.520 --> 0:03:02.920
<v Speaker 1>a popular payment processor recently TOLDY that their organization has

0:03:03.240 --> 0:03:06.200
<v Speaker 1>in the last week burned over one and a half

0:03:06.280 --> 0:03:09.880
<v Speaker 1>million dollars in anthropic tokens, with one user, one person

0:03:10.280 --> 0:03:13.160
<v Speaker 1>burning over one hundred thousand dollars in that time period.

0:03:14.040 --> 0:03:16.480
<v Speaker 1>Over at Zilo engineers are both through ninety five percent

0:03:16.520 --> 0:03:18.920
<v Speaker 1>of their annual cursor budget in less than five months,

0:03:19.160 --> 0:03:21.880
<v Speaker 1>much like Uber's CTO revealed to Laura brand Over at

0:03:21.880 --> 0:03:25.320
<v Speaker 1>the Information they had burned its entire annual token budget

0:03:25.320 --> 0:03:29.040
<v Speaker 1>by the middle of April. Axios's Manny Mills reported that

0:03:29.080 --> 0:03:32.560
<v Speaker 1>a consultant advising a CFO at an unnamed company said

0:03:32.600 --> 0:03:37.040
<v Speaker 1>that an organization had spent five hundred million dollars in

0:03:37.120 --> 0:03:42.960
<v Speaker 1>the space of a month on anthropics tokens. Jesus fucking Christ,

0:03:44.040 --> 0:03:48.400
<v Speaker 1>Jesus Christ, Jesus Christ. I'm sorry, I know I'm ranting.

0:03:48.440 --> 0:03:50.000
<v Speaker 1>I guess that's what I do on these monologues. But

0:03:50.680 --> 0:03:53.600
<v Speaker 1>you spent half a billion dollars. You fuck nuts. Have

0:03:53.680 --> 0:03:55.760
<v Speaker 1>no idea what you did that, did you? None of

0:03:55.800 --> 0:03:57.920
<v Speaker 1>you'd don't. None of these companies do. They don't. They

0:03:57.920 --> 0:04:01.160
<v Speaker 1>don't have any idea why they're doing this. Executives only

0:04:01.200 --> 0:04:03.880
<v Speaker 1>do this when they're very sick. We got we've got

0:04:03.880 --> 0:04:07.160
<v Speaker 1>to give them some cap milk or maybe take them

0:04:07.160 --> 0:04:09.720
<v Speaker 1>to the vet to finally deal with the problem ourselves.

0:04:10.200 --> 0:04:12.360
<v Speaker 1>But the AI industry doesn't really have a response to

0:04:12.400 --> 0:04:14.440
<v Speaker 1>all of this, other than to say that somebody could

0:04:14.520 --> 0:04:18.039
<v Speaker 1>theoretically make a way to measure ROI in the future

0:04:18.080 --> 0:04:21.320
<v Speaker 1>based on some metrics that nobody can can explain. I've

0:04:21.320 --> 0:04:24.520
<v Speaker 1>read thousands of words now of random Twitter posts, mostly AI,

0:04:24.600 --> 0:04:27.160
<v Speaker 1>generated of course by people that don't realize that it's

0:04:27.200 --> 0:04:30.920
<v Speaker 1>obvious say that these huge posts, and they always like, yeah,

0:04:30.960 --> 0:04:34.039
<v Speaker 1>you know, the next thing is in AI. It's going

0:04:34.080 --> 0:04:39.159
<v Speaker 1>to be tokenomics measurement and extrapolation of ROI measurements from there.

0:04:39.240 --> 0:04:43.479
<v Speaker 1>They're just saying, yeah, at some point, maybe we need

0:04:43.520 --> 0:04:46.719
<v Speaker 1>to know why we're doing this, not just the because

0:04:46.760 --> 0:04:48.839
<v Speaker 1>we all love AI. It's like the end of it,

0:04:48.920 --> 0:04:51.040
<v Speaker 1>so the beginning of Death of Stalin, when they're all

0:04:51.040 --> 0:04:53.640
<v Speaker 1>standing around being like he looks great. Everyone's just like, yeah,

0:04:53.640 --> 0:04:55.360
<v Speaker 1>of course we all love AI. Of course we all

0:04:55.400 --> 0:04:57.359
<v Speaker 1>know this, and we all know that AI is the

0:04:57.360 --> 0:04:59.880
<v Speaker 1>most special and beautiful thing and it's changing my life,

0:05:00.320 --> 0:05:02.960
<v Speaker 1>of course my life as well. Yes, it's so amazing,

0:05:03.360 --> 0:05:06.880
<v Speaker 1>but like, do we have a way of measuring if

0:05:06.920 --> 0:05:09.760
<v Speaker 1>it's actually doing that? And no one does, not a

0:05:09.920 --> 0:05:13.440
<v Speaker 1>single one of them, does. You See the problem is

0:05:13.520 --> 0:05:16.520
<v Speaker 1>they don't want to face that everybody should have measured

0:05:16.600 --> 0:05:21.279
<v Speaker 1>ROI from the beginning and only avoided doing so because

0:05:21.279 --> 0:05:25.200
<v Speaker 1>subsidized subscriptions allowed everybody to ignore the problem right up

0:05:25.279 --> 0:05:28.760
<v Speaker 1>until it was too late. Anthropics supposed explosion of growth

0:05:28.760 --> 0:05:32.279
<v Speaker 1>appears to have come entirely from experimental revenue organizations that

0:05:32.360 --> 0:05:34.800
<v Speaker 1>have no way of measuring costs or return on investment,

0:05:35.000 --> 0:05:37.520
<v Speaker 1>run by business idiots who don't know what real work is,

0:05:37.760 --> 0:05:41.240
<v Speaker 1>directly incentivizing workers to use more AI right up until

0:05:41.240 --> 0:05:44.960
<v Speaker 1>their CFO says, hey, I just saw we spent tens

0:05:44.960 --> 0:05:50.159
<v Speaker 1>of millions of dollars on this. Is it producing anything

0:05:50.200 --> 0:05:53.920
<v Speaker 1>of value? Like? Can you can? You just can? One

0:05:54.000 --> 0:05:58.480
<v Speaker 1>of you? And what's insane? Is I would expect And

0:05:58.560 --> 0:06:01.120
<v Speaker 1>never in my wildest dreams did I think that it

0:06:01.160 --> 0:06:03.599
<v Speaker 1>would be everyone. I thought that there'd be a couple

0:06:03.640 --> 0:06:06.680
<v Speaker 1>of organizations who could be like, they have a messy metric,

0:06:06.760 --> 0:06:09.760
<v Speaker 1>they've hammered out, they've really they've bone smashed it like

0:06:09.800 --> 0:06:12.600
<v Speaker 1>clavicular but with metrics, or we're going to make this

0:06:12.640 --> 0:06:14.680
<v Speaker 1>look good and then we'll be able to say this

0:06:14.800 --> 0:06:17.120
<v Speaker 1>number went up. I thought they'd have a few. I

0:06:17.160 --> 0:06:19.920
<v Speaker 1>thought they'd have some sort of weird metric that they

0:06:19.960 --> 0:06:22.520
<v Speaker 1>came up with. They don't have to be talking to

0:06:22.600 --> 0:06:24.840
<v Speaker 1>the fuckers all week. I've been asking them, and no

0:06:24.960 --> 0:06:29.000
<v Speaker 1>one does. In fact, at one company they were literally saying, Yeah,

0:06:29.040 --> 0:06:31.640
<v Speaker 1>the way we measure someone's AI use and whether it's

0:06:31.640 --> 0:06:34.800
<v Speaker 1>good or not is how many pull requests they've done

0:06:34.920 --> 0:06:37.320
<v Speaker 1>via AI, and Paul Cress being someone's gonna email and

0:06:37.400 --> 0:06:39.720
<v Speaker 1>be mad about this, calmed down. The thing that you

0:06:39.800 --> 0:06:41.720
<v Speaker 1>do when you say, hey, here is the change I

0:06:41.800 --> 0:06:43.839
<v Speaker 1>want to make to this code or this thing. This

0:06:44.000 --> 0:06:46.919
<v Speaker 1>is the plan. Do you agree there? They're measuring the

0:06:46.960 --> 0:06:50.440
<v Speaker 1>percentage of those written by AI. That is a positive metric.

0:06:51.160 --> 0:06:54.600
<v Speaker 1>That's what they're measuring. I didn't think no one would

0:06:54.640 --> 0:06:58.920
<v Speaker 1>have any idea, though, I truly didn't. It's so funny,

0:06:59.160 --> 0:07:01.720
<v Speaker 1>it's so fuzzy, it's so fucking funny. It's the funniest thing.

0:07:01.880 --> 0:07:04.479
<v Speaker 1>I'm right. I've been right about this four years. I've

0:07:04.480 --> 0:07:08.120
<v Speaker 1>been saying four years that when organizations pay the actual

0:07:08.120 --> 0:07:11.000
<v Speaker 1>cost of AI, not subsidized subscriptions that allow you to

0:07:11.000 --> 0:07:13.200
<v Speaker 1>bone effectively as much as you want. They called the

0:07:13.280 --> 0:07:16.280
<v Speaker 1>start asking questions. All of these people who've been talking

0:07:16.280 --> 0:07:19.760
<v Speaker 1>about how amazing AI is are now going, well, you know,

0:07:19.960 --> 0:07:23.000
<v Speaker 1>we need to have very difficult or we need to

0:07:23.520 --> 0:07:26.400
<v Speaker 1>we need to measure it somehow, and the value of

0:07:26.440 --> 0:07:31.920
<v Speaker 1>AI has been entirely framed around unsustainable, unprofitable, impossible to measure,

0:07:31.960 --> 0:07:34.920
<v Speaker 1>tools that beguile, imbeciles, and those that don't want to

0:07:34.920 --> 0:07:39.200
<v Speaker 1>think about reality. To be clear, open ai also appears

0:07:39.240 --> 0:07:42.560
<v Speaker 1>to be moving most businesses towards token based billing by

0:07:42.600 --> 0:07:45.920
<v Speaker 1>giving them one thousand dollars in credits to use on codex.

0:07:45.960 --> 0:07:49.520
<v Speaker 1>To soften the blow, fucking Sam, No, nobody does it better.

0:07:50.160 --> 0:07:53.920
<v Speaker 1>Nobody does it better than Clammy Samultman. Clammy said glamy Sammy.

0:07:54.000 --> 0:07:57.280
<v Speaker 1>We love Clammy. One thousand dollars just for you, honey.

0:07:57.280 --> 0:07:59.800
<v Speaker 1>You can use it whatever you want, Please pay me.

0:08:00.200 --> 0:08:03.080
<v Speaker 1>And that's the thing. Open ai broken business as well.

0:08:03.120 --> 0:08:08.440
<v Speaker 1>But there's no real solution here. There's none, And I

0:08:08.440 --> 0:08:10.800
<v Speaker 1>think the next three months are going to be illuminating

0:08:10.880 --> 0:08:14.440
<v Speaker 1>and probably feature pullback across a ton of organizations as

0:08:14.440 --> 0:08:16.920
<v Speaker 1>they ask questions they should have been asking since fucking

0:08:17.000 --> 0:08:21.440
<v Speaker 1>twenty twenty three. It's insane, I would as I sound

0:08:21.480 --> 0:08:25.280
<v Speaker 1>completely insane myself, as I sound completely crazed, but people

0:08:25.320 --> 0:08:27.960
<v Speaker 1>will calling me crazy in twenty twenty four, being like, yeah,

0:08:28.000 --> 0:08:30.200
<v Speaker 1>you know, when businesses pay the real cost, They're not

0:08:30.240 --> 0:08:32.840
<v Speaker 1>gonna People were acting like I was a fucking lunatic.

0:08:33.040 --> 0:08:37.400
<v Speaker 1>Well who's crazy? Now? Would a crazy person laugh like this? Ah? Anyway,

0:08:38.800 --> 0:08:41.439
<v Speaker 1>there are no real solutions here though, folks. There are

0:08:41.440 --> 0:08:45.280
<v Speaker 1>no real solutions. While somebody theoretically could move their workloads

0:08:45.280 --> 0:08:47.560
<v Speaker 1>to deep Seek or a cheaper model, it's unlikely that

0:08:47.600 --> 0:08:50.000
<v Speaker 1>there are inference providers that can support that much traffic

0:08:50.120 --> 0:08:52.360
<v Speaker 1>or even provide that much stability. You have to remember,

0:08:52.400 --> 0:08:55.200
<v Speaker 1>a big organization is putting a lot more pressure on

0:08:55.240 --> 0:08:59.600
<v Speaker 1>deep Seek than say a casual code of someone who's

0:08:59.600 --> 0:09:02.160
<v Speaker 1>just running their own GPU, and you have to do

0:09:02.240 --> 0:09:04.400
<v Speaker 1>far more than just turn on GPUs. You actually have

0:09:04.480 --> 0:09:07.720
<v Speaker 1>to build for an organization that would be theoretically spending

0:09:07.760 --> 0:09:11.920
<v Speaker 1>millions a massive inference stack that cannot crap out. It's

0:09:11.920 --> 0:09:14.080
<v Speaker 1>one of the reasons people pay Anthropic and open AI,

0:09:14.200 --> 0:09:18.000
<v Speaker 1>even though Anthropic is barely stable. I also don't have

0:09:18.080 --> 0:09:20.480
<v Speaker 1>any substantive proof that deep Seek V four or other

0:09:20.520 --> 0:09:24.520
<v Speaker 1>models can replace an pa opus I say, opeth there

0:09:24.880 --> 0:09:27.920
<v Speaker 1>not editing that or Codex model, even if they are

0:09:27.960 --> 0:09:31.240
<v Speaker 1>dramatically cheaper. We have no proof of this, and despite

0:09:31.280 --> 0:09:34.920
<v Speaker 1>the conversation, no one's doing it. Not heard it, not

0:09:35.000 --> 0:09:38.000
<v Speaker 1>heard anyone say this. So in the end, if nobody

0:09:38.000 --> 0:09:41.960
<v Speaker 1>can measure the ROI of AI in general, why would

0:09:41.960 --> 0:09:46.840
<v Speaker 1>they spend anything now. AI boosters are currently either completely

0:09:46.880 --> 0:09:49.800
<v Speaker 1>ignoring this subject, which is what's happening with the obvious ones,

0:09:50.360 --> 0:09:52.120
<v Speaker 1>or they're going to answer by saying that we're in

0:09:52.120 --> 0:09:54.720
<v Speaker 1>the early days and that there are weekly breakthroughs that

0:09:54.760 --> 0:09:56.880
<v Speaker 1>will solve these problems. Be in a lot of fucking

0:09:56.960 --> 0:10:01.840
<v Speaker 1>weeks so far, been a lot of them, what like

0:10:01.920 --> 0:10:04.200
<v Speaker 1>a four hundred of them, four hundred and fifty coming

0:10:04.240 --> 0:10:06.960
<v Speaker 1>up with five hundred weeks. But this is a problem

0:10:07.000 --> 0:10:09.600
<v Speaker 1>that needed to be solved yesterday, and neither anthropic nor

0:10:09.640 --> 0:10:12.199
<v Speaker 1>open AI has an answer for bringing down costs, let

0:10:12.200 --> 0:10:15.839
<v Speaker 1>alone those of their customers. Even if even if they

0:10:15.920 --> 0:10:19.600
<v Speaker 1>were magically profitable overnight, even if they were, they're not.

0:10:19.880 --> 0:10:23.360
<v Speaker 1>They're not profitable in inference, their customers are spending too

0:10:23.440 --> 0:10:26.400
<v Speaker 1>much money. Their customers do not have cost control. That

0:10:26.640 --> 0:10:29.000
<v Speaker 1>makes it very hard to budget for how much you'll

0:10:29.040 --> 0:10:35.160
<v Speaker 1>spend basic business stuff, basic stuff. I'm not innovative in

0:10:35.240 --> 0:10:38.560
<v Speaker 1>my thinking here, you know. Is it's like that tweet

0:10:38.600 --> 0:10:41.080
<v Speaker 1>that was like, yeah, walking around a college campus looking

0:10:41.080 --> 0:10:44.560
<v Speaker 1>into a business school class, and it's like profit is

0:10:44.600 --> 0:10:47.880
<v Speaker 1>revenue minus costs. It really is that, except these people

0:10:47.880 --> 0:10:51.360
<v Speaker 1>have been ignoring that four years and making people feel

0:10:51.440 --> 0:10:55.440
<v Speaker 1>stupid for asking those questions. I also must be clear

0:10:55.440 --> 0:10:58.600
<v Speaker 1>that these companies cannot slow down. Anthropic and open AI

0:10:58.720 --> 0:11:01.080
<v Speaker 1>project to have over three hundred and fifty billion dollars

0:11:01.120 --> 0:11:03.680
<v Speaker 1>in combine down your revenue by twenty thirty, and they're

0:11:03.720 --> 0:11:06.160
<v Speaker 1>gonna need it, because between them they've made over one

0:11:06.200 --> 0:11:10.840
<v Speaker 1>point one trillion dollars in compute commitments. If their revenue

0:11:10.880 --> 0:11:13.680
<v Speaker 1>growth is entirely based on dimwitted executives allowing a no

0:11:13.880 --> 0:11:17.840
<v Speaker 1>it loads refused compute dump for no apparent return on investment,

0:11:18.080 --> 0:11:21.240
<v Speaker 1>that is not a stable, sustainable, or even a growth business.

0:11:21.440 --> 0:11:24.360
<v Speaker 1>It's a grift targeting a society wide executive ignorance of

0:11:24.400 --> 0:11:27.720
<v Speaker 1>production itself. I look forward to telling you more about

0:11:27.720 --> 0:11:30.240
<v Speaker 1>it next week. I've had a lot of fun this week,

0:11:30.320 --> 0:11:32.840
<v Speaker 1>got Paul Kedrowski coming back on the show on Tuesday,

0:11:33.480 --> 0:11:36.880
<v Speaker 1>actually Wednesdays, sorry, twelve am met Wednesday. I should really

0:11:36.920 --> 0:11:39.160
<v Speaker 1>know that by now it's been years. And then yeah,

0:11:39.200 --> 0:11:41.080
<v Speaker 1>I'll have another monologue for you. It's been a lot

0:11:41.120 --> 0:11:43.320
<v Speaker 1>of fun. I hope you're enjoying it. ZiT run Out