Speaker 1: Zone Media. Hi, Mimo mo ed Zytron, and welcome to
Speaker 1: this week's Better Offline Monologue. It's a triumphant week here
Speaker 1: at the Zyron household. As I mentioned earlier in the week,
Speaker 1: Uber COO Andrew McDonald said and I quote that its
Speaker 1: AI costs were becoming harder to justify and that the
Speaker 1: link was not there between spending money on AI tokens
Speaker 1: and creating more useful features. In the days that have
Speaker 1: follow the AI industry has been embroiled in a conversation
Speaker 1: that mostly comes down to one thing. Hey, does anyone
Speaker 1: have any way to measure the ROI of AI? Didn't
Speaker 1: fucking think of that one digit fellas, Oh no, I
Speaker 1: want to come up with that before I spent all
Speaker 1: the fucking money. But I don't know. I don't run
Speaker 1: a fucking business, drap. That's because up until recently, most
Speaker 1: companies have been able to pay for a subscription with
Speaker 1: subsidized token rates, meaning that forever dollar of their subscription
Speaker 1: they could burn anywhere from three to thirteen dollars worth
Speaker 1: of tokens. They just had rate limits. Whatever they did
Speaker 1: just eight into those rate limits, and they didn't really
Speaker 1: think about the token cost in fact, every one of
Speaker 1: these companies deliberately obfuscates that information. You can find it
Speaker 1: through CC usage, which you need to have command lined interface,
Speaker 1: and there's also slash usage with anthropic Regardless, as a
Speaker 1: result of all of this, nobody was really measuring the
Speaker 1: actual positive outcomes of AI services, or even how much
Speaker 1: a particular task would burning tokens, which means that with
Speaker 1: the advent of token based billing for enterprises, everyone is
Speaker 1: just guessing about how much their annual budget should be
Speaker 1: based on virtually no useful information. Imagine you're doing something
Speaker 1: for years, years and years, and you're claiming it it's
Speaker 1: the future, and it's going to change everything. It's going
Speaker 1: to change your business, and there's other businesses. It's the
Speaker 1: future of your revenue and their revenue. Oh my god,
Speaker 1: it's so amazing. But not once did your clever nbaar say,
Speaker 1: why don't we make sure it actually is? No one
Speaker 1: did that, not one of them, Not one of these
Speaker 1: fuckers have been I've been checking a week, we're talking
Speaker 1: to people a week. In truth, though, measuring ROI is
Speaker 1: pretty difficult for a lot of knowledge work because it
Speaker 1: requires you to actually know what a person does for
Speaker 1: a living, and what doing a job is like, and
Speaker 1: what the outcomes have said job are and what the
Speaker 1: actual like good results at the end may be. Which
Speaker 1: is a lot to ask for middle managers and CEOs
Speaker 1: who mostly go to and from lunch and explain why
Speaker 1: other people's work is actually theirs. And I want to say,
Speaker 1: if you're a middle manager hearing this, more than likely
Speaker 1: I'm completely correct about you. Most middle managers are fucking scum.
Speaker 1: They exist only to steal work from people and nag
Speaker 1: them and harass them. If you're not one of those,
Speaker 1: great you shouldn't be offended by this. This is not
Speaker 1: about you. But if you are, you're probably one of
Speaker 1: the people who's going to email me, in which case, wei,
Speaker 1: weare God, cry to someone else. I don't give a ship,
Speaker 1: but the lack of measurement of both task costs and
Speaker 1: ROI as executives already crying for mercy. An engineer at
Speaker 1: a popular payment processor recently TOLDY that their organization has
Speaker 1: in the last week burned over one and a half
Speaker 1: million dollars in anthropic tokens, with one user, one person
Speaker 1: burning over one hundred thousand dollars in that time period.
Speaker 1: Over at Zilo engineers are both through ninety five percent
Speaker 1: of their annual cursor budget in less than five months,
Speaker 1: much like Uber's CTO revealed to Laura brand Over at
Speaker 1: the Information they had burned its entire annual token budget
Speaker 1: by the middle of April. Axios's Manny Mills reported that
Speaker 1: a consultant advising a CFO at an unnamed company said
Speaker 1: that an organization had spent five hundred million dollars in
Speaker 1: the space of a month on anthropics tokens. Jesus fucking Christ,
Speaker 1: Jesus Christ, Jesus Christ. I'm sorry, I know I'm ranting.
Speaker 1: I guess that's what I do on these monologues. But
Speaker 1: you spent half a billion dollars. You fuck nuts. Have
Speaker 1: no idea what you did that, did you? None of
Speaker 1: you'd don't. None of these companies do. They don't. They
Speaker 1: don't have any idea why they're doing this. Executives only
Speaker 1: do this when they're very sick. We got we've got
Speaker 1: to give them some cap milk or maybe take them
Speaker 1: to the vet to finally deal with the problem ourselves.
Speaker 1: But the AI industry doesn't really have a response to
Speaker 1: all of this, other than to say that somebody could
Speaker 1: theoretically make a way to measure ROI in the future
Speaker 1: based on some metrics that nobody can can explain. I've
Speaker 1: read thousands of words now of random Twitter posts, mostly AI,
Speaker 1: generated of course by people that don't realize that it's
Speaker 1: obvious say that these huge posts, and they always like, yeah,
Speaker 1: you know, the next thing is in AI. It's going
Speaker 1: to be tokenomics measurement and extrapolation of ROI measurements from there.
Speaker 1: They're just saying, yeah, at some point, maybe we need
Speaker 1: to know why we're doing this, not just the because
Speaker 1: we all love AI. It's like the end of it,
Speaker 1: so the beginning of Death of Stalin, when they're all
Speaker 1: standing around being like he looks great. Everyone's just like, yeah,
Speaker 1: of course we all love AI. Of course we all
Speaker 1: know this, and we all know that AI is the
Speaker 1: most special and beautiful thing and it's changing my life,
Speaker 1: of course my life as well. Yes, it's so amazing,
Speaker 1: but like, do we have a way of measuring if
Speaker 1: it's actually doing that? And no one does, not a
Speaker 1: single one of them, does. You See the problem is
Speaker 1: they don't want to face that everybody should have measured
Speaker 1: ROI from the beginning and only avoided doing so because
Speaker 1: subsidized subscriptions allowed everybody to ignore the problem right up
Speaker 1: until it was too late. Anthropics supposed explosion of growth
Speaker 1: appears to have come entirely from experimental revenue organizations that
Speaker 1: have no way of measuring costs or return on investment,
Speaker 1: run by business idiots who don't know what real work is,
Speaker 1: directly incentivizing workers to use more AI right up until
Speaker 1: their CFO says, hey, I just saw we spent tens
Speaker 1: of millions of dollars on this. Is it producing anything
Speaker 1: of value? Like? Can you can? You just can? One
Speaker 1: of you? And what's insane? Is I would expect And
Speaker 1: never in my wildest dreams did I think that it
Speaker 1: would be everyone. I thought that there'd be a couple
Speaker 1: of organizations who could be like, they have a messy metric,
Speaker 1: they've hammered out, they've really they've bone smashed it like
Speaker 1: clavicular but with metrics, or we're going to make this
Speaker 1: look good and then we'll be able to say this
Speaker 1: number went up. I thought they'd have a few. I
Speaker 1: thought they'd have some sort of weird metric that they
Speaker 1: came up with. They don't have to be talking to
Speaker 1: the fuckers all week. I've been asking them, and no
Speaker 1: one does. In fact, at one company they were literally saying, Yeah,
Speaker 1: the way we measure someone's AI use and whether it's
Speaker 1: good or not is how many pull requests they've done
Speaker 1: via AI, and Paul Cress being someone's gonna email and
Speaker 1: be mad about this, calmed down. The thing that you
Speaker 1: do when you say, hey, here is the change I
Speaker 1: want to make to this code or this thing. This
Speaker 1: is the plan. Do you agree there? They're measuring the
Speaker 1: percentage of those written by AI. That is a positive metric.
Speaker 1: That's what they're measuring. I didn't think no one would
Speaker 1: have any idea, though, I truly didn't. It's so funny,
Speaker 1: it's so fuzzy, it's so fucking funny. It's the funniest thing.
Speaker 1: I'm right. I've been right about this four years. I've
Speaker 1: been saying four years that when organizations pay the actual
Speaker 1: cost of AI, not subsidized subscriptions that allow you to
Speaker 1: bone effectively as much as you want. They called the
Speaker 1: start asking questions. All of these people who've been talking
Speaker 1: about how amazing AI is are now going, well, you know,
Speaker 1: we need to have very difficult or we need to
Speaker 1: we need to measure it somehow, and the value of
Speaker 1: AI has been entirely framed around unsustainable, unprofitable, impossible to measure,
Speaker 1: tools that beguile, imbeciles, and those that don't want to
Speaker 1: think about reality. To be clear, open ai also appears
Speaker 1: to be moving most businesses towards token based billing by
Speaker 1: giving them one thousand dollars in credits to use on codex.
Speaker 1: To soften the blow, fucking Sam, No, nobody does it better.
Speaker 1: Nobody does it better than Clammy Samultman. Clammy said glamy Sammy.
Speaker 1: We love Clammy. One thousand dollars just for you, honey.
Speaker 1: You can use it whatever you want, Please pay me.
Speaker 1: And that's the thing. Open ai broken business as well.
Speaker 1: But there's no real solution here. There's none, And I
Speaker 1: think the next three months are going to be illuminating
Speaker 1: and probably feature pullback across a ton of organizations as
Speaker 1: they ask questions they should have been asking since fucking
Speaker 1: twenty twenty three. It's insane, I would as I sound
Speaker 1: completely insane myself, as I sound completely crazed, but people
Speaker 1: will calling me crazy in twenty twenty four, being like, yeah,
Speaker 1: you know, when businesses pay the real cost, They're not
Speaker 1: gonna People were acting like I was a fucking lunatic.
Speaker 1: Well who's crazy? Now? Would a crazy person laugh like this? Ah? Anyway,
Speaker 1: there are no real solutions here though, folks. There are
Speaker 1: no real solutions. While somebody theoretically could move their workloads
Speaker 1: to deep Seek or a cheaper model, it's unlikely that
Speaker 1: there are inference providers that can support that much traffic
Speaker 1: or even provide that much stability. You have to remember,
Speaker 1: a big organization is putting a lot more pressure on
Speaker 1: deep Seek than say a casual code of someone who's
Speaker 1: just running their own GPU, and you have to do
Speaker 1: far more than just turn on GPUs. You actually have
Speaker 1: to build for an organization that would be theoretically spending
Speaker 1: millions a massive inference stack that cannot crap out. It's
Speaker 1: one of the reasons people pay Anthropic and open AI,
Speaker 1: even though Anthropic is barely stable. I also don't have
Speaker 1: any substantive proof that deep Seek V four or other
Speaker 1: models can replace an pa opus I say, opeth there
Speaker 1: not editing that or Codex model, even if they are
Speaker 1: dramatically cheaper. We have no proof of this, and despite
Speaker 1: the conversation, no one's doing it. Not heard it, not
Speaker 1: heard anyone say this. So in the end, if nobody
Speaker 1: can measure the ROI of AI in general, why would
Speaker 1: they spend anything now. AI boosters are currently either completely
Speaker 1: ignoring this subject, which is what's happening with the obvious ones,
Speaker 1: or they're going to answer by saying that we're in
Speaker 1: the early days and that there are weekly breakthroughs that
Speaker 1: will solve these problems. Be in a lot of fucking
Speaker 1: weeks so far, been a lot of them, what like
Speaker 1: a four hundred of them, four hundred and fifty coming
Speaker 1: up with five hundred weeks. But this is a problem
Speaker 1: that needed to be solved yesterday, and neither anthropic nor
Speaker 1: open AI has an answer for bringing down costs, let
Speaker 1: alone those of their customers. Even if even if they
Speaker 1: were magically profitable overnight, even if they were, they're not.
Speaker 1: They're not profitable in inference, their customers are spending too
Speaker 1: much money. Their customers do not have cost control. That
Speaker 1: makes it very hard to budget for how much you'll
Speaker 1: spend basic business stuff, basic stuff. I'm not innovative in
Speaker 1: my thinking here, you know. Is it's like that tweet
Speaker 1: that was like, yeah, walking around a college campus looking
Speaker 1: into a business school class, and it's like profit is
Speaker 1: revenue minus costs. It really is that, except these people
Speaker 1: have been ignoring that four years and making people feel
Speaker 1: stupid for asking those questions. I also must be clear
Speaker 1: that these companies cannot slow down. Anthropic and open AI
Speaker 1: project to have over three hundred and fifty billion dollars
Speaker 1: in combine down your revenue by twenty thirty, and they're
Speaker 1: gonna need it, because between them they've made over one
Speaker 1: point one trillion dollars in compute commitments. If their revenue
Speaker 1: growth is entirely based on dimwitted executives allowing a no
Speaker 1: it loads refused compute dump for no apparent return on investment,
Speaker 1: that is not a stable, sustainable, or even a growth business.
Speaker 1: It's a grift targeting a society wide executive ignorance of
Speaker 1: production itself. I look forward to telling you more about
Speaker 1: it next week. I've had a lot of fun this week,
Speaker 1: got Paul Kedrowski coming back on the show on Tuesday,
Speaker 1: actually Wednesdays, sorry, twelve am met Wednesday. I should really
Speaker 1: know that by now it's been years. And then yeah,
Speaker 1: I'll have another monologue for you. It's been a lot
Speaker 1: of fun. I hope you're enjoying it. ZiT run Out