Speaker 1: A media Hello and welcome to Better Offline. I'm your
Speaker 1: host ed Zitron. What a lot of you have been
Speaker 1: getting in touch? Yes, you're getting your Deep Seek episode.
Speaker 1: In fact, this is the first of a two parter.
Speaker 1: This will come out on Friday, which is when you're
Speaker 1: listening to this, and then it'll follow up on Monday.
Speaker 1: I apologize. I spent a lot of Monday writing this
Speaker 1: and also learning about a lot of this stuff in
Speaker 1: an attempt to distill it as best I could. This
Speaker 1: situation is extremely weird, and it's developing, and I think
Speaker 1: even when I put out this episode there will be
Speaker 1: new parts of it that I have yet to really
Speaker 1: get to. I will do my absolute best to explain
Speaker 1: in these episodes both what is happening with Deep Seek,
Speaker 1: what it means, what they've built, and what it's going
Speaker 1: to do in the future. But let's again, so, as
Speaker 1: general came to a close, the entire generative AI industry
Speaker 1: found itself in a kind of chaos. In sure, the
Speaker 1: recent AI bubble and in particular the hundreds of billions
Speaker 1: of dollars being spent on it, hinged on this big
Speaker 1: idea that we need bigger models, which are both trained
Speaker 1: and run on bigger and even larger GPUs, almost entirely
Speaker 1: sold by Nvidia, and in turn they're based in bigger
Speaker 1: and bigger data centers owned by companies like Microsoft, Oracle, Amazon,
Speaker 1: and Google. Now, there was also this expectation that this
Speaker 1: would always be the case. Hubris within this industry is
Speaker 1: kind of part of the whole deal, and generative AI
Speaker 1: was always meant to be this way, at least for
Speaker 1: the American developers. It was always meant to be energy
Speaker 1: and compute hungary. Throwing entire Zoo's worth of animals and
Speaker 1: boiling lakes was necessary to do this. There was never
Speaker 1: any other way to do it, and I thought, at
Speaker 1: least I've thought for a while that this was because
Speaker 1: they just they tried to make them more efficient, but
Speaker 1: they couldn't. There was just something about transformer based architecture,
Speaker 1: like the stuff that underpins Chat GPT, so the GPT
Speaker 1: model under Chat GPT either. It wasn't the case, though.
Speaker 1: A Chinese artificial intelligence company that few people had really
Speaker 1: heard of, called deep Seak came along a few weeks
Speaker 1: ago with multiple models that aren't merely competitive with open aiyes,
Speaker 1: but actually undercut them in several meaningful ways. Deep Seak's
Speaker 1: models are both open source, which means that their source
Speaker 1: code and research is public, and they're significantly more efficient
Speaker 1: as well as much as thirty times cheaper to run.
Speaker 1: In the case of their reasoning model are one which
Speaker 1: is competitive with open Aizo one and fifteen or more
Speaker 1: times more efficient than GPT four. Oh, it's actually kind
Speaker 1: of crazy when you think about it, and as you're
Speaker 1: going to hear, this whole thing has jokeified me all
Speaker 1: over again. And what's crazy is that some of them
Speaker 1: can be distilled, which I'll get too later and run
Speaker 1: on local devices like a laptop. It's kind of crazy,
Speaker 1: and as a result, the markets have kind of panicked
Speaker 1: because the entire narrative of the AI bubble has been
Speaker 1: that these models have to be expensive because they are
Speaker 1: the future, and that's why hyperscalers had to burn two
Speaker 1: hundred billion dollars in capital expenditures for infrastructure to support
Speaker 1: this wonderful boom, and specifically the ideas of open AI
Speaker 1: and anthropic the idea that there was another way to
Speaker 1: do this, that in fact, we didn't need to spend
Speaker 1: all this money, and that maybe we could find a
Speaker 1: more efficient way of doing it. Well, that would require
Speaker 1: them to have another idea rather than throw as much
Speaker 1: money at the problem as possible. Yeah, they just didn't
Speaker 1: consider it, it turns out. And now as long as
Speaker 1: come this outsider that's upended the whole conventional understanding and
Speaker 1: perhaps even dethroned a member of America's tech royalty, Sam Altman,
Speaker 1: a man who has crafted, if not a cult of personality,
Speaker 1: some sort of public image of an unassailable visionary that
Speaker 1: will lead the vanguard in the biggest technological change since
Speaker 1: the Internet. Yeah, he's wrong. He never was doing that.
Speaker 1: I've been saying it for a while. He's never been
Speaker 1: doing this. But Deep Seek isn't just an outsider now.
Speaker 1: They are a company that's emerged as a side project
Speaker 1: from a tiny, tiny Chinese hedge fund, at least by
Speaker 1: the stands of hedge funds, like five point five billion
Speaker 1: dollars on the assets under management, and their founding team
Speaker 1: has nowhere near the level of fame and celebrity or
Speaker 1: even the accolades of Sam Moltman. It's distinctly humiliating for
Speaker 1: everyone involved that is in Deep Seek. And on top
Speaker 1: of all of that, Deep Seek's biggest ugliest insult is
Speaker 1: that its model, deep seek are one, is competitive, like
Speaker 1: I said, with open AI's incredibly expensive oh one reasoning model,
Speaker 1: yet significantly and I mean ninety six percent cheaper to run.
Speaker 1: And it can even be run locally. Like I said
Speaker 1: speaking to a few developers, I know one was able
Speaker 1: to run deep Seek's R one model and their twenty
Speaker 1: twenty one MacBook Pro with an M one chip that
Speaker 1: is a four year old computer, not a thirty thousand
Speaker 1: GPU inside. It's kind of crazy. Worse still, Deep seeks
Speaker 1: models are made freely available to use, with the source
Speaker 1: code published under the MIT tech line, along with the
Speaker 1: research on how they were made, although not the training data,
Speaker 1: which makes some people say it's not really open source.
Speaker 1: But for the sake of argument, I'm just going to
Speaker 1: say open source. And this means by the way that
Speaker 1: deep seeks models can be adapted and used for commercial
Speaker 1: use without the need for royalties or fees. Anyone can
Speaker 1: take this and build their own. It's kind of crazy.
Speaker 1: By contrast, open ai is anything but open and its
Speaker 1: last LM to be released under the MIT license was
Speaker 1: twenty nineteen. Is GBT two No no wait wait, ship
Speaker 1: let me correct that deep Seek's biggest ugliest secret is
Speaker 1: actually that it's obviously taking aim at every element of
Speaker 1: open aiy's portfolio. As the company was already dominating headlines,
Speaker 1: this week it quietly dropped its Janus Pro seven B
Speaker 1: image generation and analysis model, which the company says outperforms
Speaker 1: both stable diffusion and open AI's Daly three. And those
Speaker 1: are by the way image generation thinks. So you type
Speaker 1: in something you like Garfield with boobs, and then outcomes
Speaker 1: of Garfield with juicy cans, and that's probably the first
Speaker 1: time you hear that on the podcast, but probably not
Speaker 1: the last. And as with its other code, deep Seek
Speaker 1: has made this freely available to both commercial and personal
Speaker 1: users alike, whereas open ai is largely paywall darly three.
Speaker 1: This is really, it's a truly crazy situation. And it's
Speaker 1: also this cynical, vulgar version of David and Goliath, where
Speaker 1: a tech startup back by a shadowy Chinese hedge fund
Speaker 1: with eight billion dollars under management is somehow the plucky
Speaker 1: upstart against the lumbering loss eo fish one hundred and
Speaker 1: fifty billion dollars startup back by multiple public tech companies
Speaker 1: with a market capitalization of other three trillion dollars I realized,
Speaker 1: by the way I said earlier, five point five billion
Speaker 1: dollars under management. This is why you check your notes
Speaker 1: in advance. But I'm not cutting it. This is fresh.
Speaker 1: I am inside a closet in New York. The content
Speaker 1: must flow anyway. Deep Seek's V three model, which is
Speaker 1: comparable and competitive with both open AI's GPT four roh
Speaker 1: and anthropics Claude Sonnet three point five models, which by
Speaker 1: the way, has some reasoning features. As I said, it's
Speaker 1: fifty three times cheaper to run the R one when
Speaker 1: using the company's own cloud services, and as mentioned earlier,
Speaker 1: said model is effectively free for anyone to use locally
Speaker 1: or on their own cloud instances, and could be taken
Speaker 1: by any commercial enterprise and turned into a product of
Speaker 1: their own should they desire to say, compete with open Ai,
Speaker 1: the loudest and most annoying startup of all time. In essence, Deepseek,
Speaker 1: and I'll get into its background and the concerns people
Speaker 1: might have about its Chinese origins released two models that
Speaker 1: perform competitively and even beat models from both open Air
Speaker 1: and Anthropic, undercut them in price, and then made them
Speaker 1: open undermining not just the economics of the biggest generative
Speaker 1: AI companies, but laying bare exactly how they work. The
Speaker 1: magic's gone. There's no more voodoo inside Samultman's soul. It's
Speaker 1: all out there. And the last point is extremely important
Speaker 1: when it comes to open EI's reasoning model, which specifically
Speaker 1: hid its chain of thought for fear of these unsafe
Speaker 1: thoughts that might manipulate the customer. And then they add
Speaker 1: slightly under their breath that the actual reasons they did
Speaker 1: it was a competitive advantage. Now to explain what that means.
Speaker 1: When you make a request with open Aiy's oh one model,
Speaker 1: say give me all the states with the letter are
Speaker 1: in them, it actually shows you like the thinking. And
Speaker 1: by the way, these things don't fucking think. They're they're
Speaker 1: computer bullshit, like they don't think at all. But I'm
Speaker 1: going to use it just for this so you see it.
Speaker 1: Say okay, here are all the American states, which ones
Speaker 1: have that letter? I'm checking all of those. It's effectively
Speaker 1: having a large language model check a large language model. Now,
Speaker 1: the thing is the steps they were showing you were
Speaker 1: all cleaned up. They would look nice, they would be
Speaker 1: formatted nicely. Deep Seak's chain of thought is completely laid bare,
Speaker 1: which is very interesting because it really takes the wind
Speaker 1: out of open Aiy's sales. And on top of that,
Speaker 1: it allows you to see actually how these things think
Speaker 1: through things, again not really thinking, but still you can
Speaker 1: see things about how large language models work that these
Speaker 1: companies didn't want you to have. On top of this,
Speaker 1: open aiy one model has something even shittier to it,
Speaker 1: which is these chain of thought things all cost money.
Speaker 1: When you see it generate these thoughts, it's actually generating
Speaker 1: more thoughts than you see because they're hiding the chain
Speaker 1: of thought. So open ai is just charging you an
Speaker 1: indeterminate amount of money, an insane amount of money, as
Speaker 1: I'll get too later. But nevertheless, you don't know what
Speaker 1: you're being charged for. You don't even know what's really
Speaker 1: going on under the hood. Or you could use deep
Speaker 1: seek and let's be completely clear, by the way, open
Speaker 1: AI's literal only competitive advantage against Meta and Anthropic was
Speaker 1: its reasoning models OH one and O three and O three,
Speaker 1: by the way, is currently in a research preview and
Speaker 1: is mostly just more of the same. Although I mentioned
Speaker 1: earlier in the show that anthropics. Claudes Sonnet three point
Speaker 1: five has some reasoning features. They're comparatively more rudimentary than
Speaker 1: those in O one and O three, and i'd argue
Speaker 1: are one, which is deep Seek's model. In an AI context,
Speaker 1: reasoning works by breaking down a prompt into a series
Speaker 1: of different steps with considerations of different approaches. Like I
Speaker 1: said earlier, effectively a large language model checking its own
Speaker 1: homework with no thinking involved, because like I said, they
Speaker 1: do not think or no things an open Ai rushed
Speaker 1: to launch its O one reasoning model last year because,
Speaker 1: and I quote fortune from last October, Sam Mormon was
Speaker 1: eager to prove to potential investors that in the company's
Speaker 1: latest funding around, the open ai remains at the forefront
Speaker 1: of AI development, and as I've noted in my newsletter
Speaker 1: at the time, it was not particularly reliable, failing to
Speaker 1: accurately count the number of times the letter R appeared
Speaker 1: in the word strawberry, which was the code name for
Speaker 1: one very funny stuff. At this point, it's fairly obvious
Speaker 1: that open ai wasn't anywhere near the forefront of AI development,
Speaker 1: and now that its competitive advantage is effectively gone, there
Speaker 1: are genuine doubts about what comes next for the company.
Speaker 1: As I'll go into there are many questionable parts of
Speaker 1: Deepseek's story. It's funding, what GPUs it has, and how
Speaker 1: much it actually spent training these models. But what we
Speaker 1: definitively understand to be true is badly for open Ai,
Speaker 1: and I would argue every other large US tech firm
Speaker 1: that's jumped onto the generative AI bandwagon in the past
Speaker 1: few years. Deep seeks models actually exist. They work, at
Speaker 1: least by the standards of hallucination PRONELLA lams that don't,
Speaker 1: at the risk of repeating myself know anything. They've been
Speaker 1: independently verified to be competitive in performance, and their magnitudes
Speaker 1: cheaper in price than those from both hyperscalers, Google's Gemini, Mets, Lama,
Speaker 1: Amazon Que and so on and so forth, and from
Speaker 1: those released by open Ai and Anthropic. Deep seeks models
Speaker 1: don't require massive new data centers. They run on GPUs
Speaker 1: currently used to run services like chat, GPT, and even
Speaker 1: work on more austere hardware, Nor do they require an
Speaker 1: endless supply of bigger, faster Nvidio GPUs every single year
Speaker 1: to progress. The entire AI bubble was inflated based on
Speaker 1: the premise that these models were simply impossible to build
Speaker 1: without burning massive amounts of cash, straining the power grid,
Speaker 1: and blowing past emission skulls, and that these costs were
Speaker 1: both necessary and really good because they'd lead to creating
Speaker 1: powerful AI, something that's yet to happen. And it's kind
Speaker 1: of obvious at this point that that wasn't true. Now
Speaker 1: the markets are sitting around there asking a very reasonable question, Shit,
Speaker 1: did we just waste two hundred billion dollars? Anyway, let's
Speaker 1: get into the nitty grit. What is deep Seek? First
Speaker 1: of all, if you want to super deep dive into
Speaker 1: what it is, I can't recommend venture beats right up enough.
Speaker 1: I'll link to it in the show notes as they
Speaker 1: usually do. It's really good and it goes into a
Speaker 1: lot more detail than I woar. But here's the too
Speaker 1: long didn't read for you. Deep Seek is a spin
Speaker 1: off from a Chinese hedge fund called high Flyer Quant.
Speaker 1: It's a relatively small and young company, and from its
Speaker 1: inception it went big on algorithmic and AI driven trading.
Speaker 1: Later it started building its own standalone chat bots, including
Speaker 1: a chat GPT equivalent for the Chinese market. This is
Speaker 1: what we need, right Now, I'm sure some of you
Speaker 1: will say, oh, well, who knows if that's really true. Sure,
Speaker 1: I think that that's fair. I also think that there
Speaker 1: are parts of Sam Mortman's legend that we should question
Speaker 1: as well. I think the circumstances under which Sam Mortman
Speaker 1: got made head of y Combinator are extremely questionable. I'm
Speaker 1: saying you can question deep Seek, and indeed you should.
Speaker 1: We should be more critical of these powerful companies, but
Speaker 1: don't do it halfway. If we're going to be worried,
Speaker 1: let's be worried about everyone. Now. Deepseak did a few
Speaker 1: things differently, like open sourcing its models, although it likely
Speaker 1: built upon take from other companies like Metaslama and the
Speaker 1: mL library PyTorch to train its models. It's secured over
Speaker 1: ten thousand Nvidia GPUs right before the US imposed export restrictions,
Speaker 1: which sounds like a lot, but it's a fraction of
Speaker 1: what the big AI labs like Google, Open Air, and
Speaker 1: Anthropic have to play with. I think I've heard estimates
Speaker 1: of like one hundred thousand to three hundred thousand each,
Speaker 1: if not more. Now you've likely seen or heard that
Speaker 1: deep Seak trained its latest model for five point six
Speaker 1: million dollars a poster to the insane amounts that I'll
Speaker 1: get to later, and I want to be clear that
Speaker 1: any and all mentions of this number are estimates. In fact,
Speaker 1: the provenance of the five point five to eight million
Speaker 1: dollar number appears to be a citation of a post
Speaker 1: made by an nvidio engineer in an article from the
Speaker 1: South China Morning Post, which links to another article from
Speaker 1: the South China Morning Post which simply states that deep
Speaker 1: Seat V three comes with six hundred and seventy one
Speaker 1: billion parameters and was trained in around two months at
Speaker 1: the cost of five point five eight million dollars with
Speaker 1: no additional citations of any kind. So you should take
Speaker 1: it with a pinch of salt. But it's not totally ludicrous. Well,
Speaker 1: there are some that have estimated the cost. Deep Seeks
Speaker 1: V three models allegedly trained using two thousand and forty
Speaker 1: eight n video h eight hundred GPUs according to its paper,
Speaker 1: and Ben Thompson of Strategory has made this clear that
Speaker 1: the five point five million dollar number only covers the
Speaker 1: literal training cost of the official training run, and this
Speaker 1: is made fairly clear in the paper by the way
Speaker 1: of V three, and that's the one that's competitive with
Speaker 1: Opening Eyes GPT four O model, meaning that any costs
Speaker 1: related to prior research or experiments on how to build
Speaker 1: the mooddle were left out. Now big big shower to
Speaker 1: Minimaxer the guy on Blue Sky and Twitter, he's great.
Speaker 1: He is wonderful, and also added that this is fairly
Speaker 1: standard for the industry. Again, you choose how you feel
Speaker 1: about this, but I want to give you the information.
Speaker 1: And while it's safe to say that deep Seak's models
Speaker 1: are cheaper to train, the actual costs, especially as deep
Speaker 1: Seak doesn't share its training data, which some might argue
Speaker 1: means its models are not really open source. As I said,
Speaker 1: the numbers get a little harder to guess at. Thompson
Speaker 1: notes that Deep Seek had to craft a bunch of
Speaker 1: elegant workarounds to make the model perform, including writing code
Speaker 1: that ultimately changed how GPUs actually communicated with each other.
Speaker 1: This functionality isn't otherwise possible using Nvidia's developer tools. They
Speaker 1: really had to get in there. It's kind of cool.
Speaker 1: Deep seaks models V three and R one are more
Speaker 1: efficient and as a result, cheaper to run, and can
Speaker 1: be accessed via its API at prices that are astronomically
Speaker 1: cheaper than open eyes, Deep seat Chat running deep six
Speaker 1: GPT four oh competitive V three model cost zero points
Speaker 1: zero seven cents per one million input tokens as in
Speaker 1: commands given to the model, and one dollar one ten
Speaker 1: per one million output tokens as in the resulting output
Speaker 1: from the model. I know that these numbers kind of
Speaker 1: like just sound like numbers like you, Maybe you don't
Speaker 1: have context, so let me give you some. This is
Speaker 1: a dramatic price drop from the two dollars fifty cents
Speaker 1: per one million input tokens and ten dollars per one
Speaker 1: million output tokens the open Ai charges for GPT four. Oh,
Speaker 1: this isn't just undercutting, this is this is a bunker buster. If. Now,
Speaker 1: there is a side that I'll kind of get into
Speaker 1: a little bit later, in that you are using models
Speaker 1: hosted in the country that you don't know, probably China.
Speaker 1: There are data concerns. But again, you can put this
Speaker 1: on your own server. You could put this in Google Cloud.
Speaker 1: Both Microsoft and Google are apparently thinking about it now.
Speaker 1: The Information reported that Google had added it to Google Cloud.
Speaker 1: No they did not. They didn't do that. They allowed
Speaker 1: you to connect hugging face. This is a whole bunch
Speaker 1: of technical stuff that if you understand, you'll be like, yeah, Ed,
Speaker 1: I know. Long story short, the hyperscalers are already bringing
Speaker 1: deep Seek out, and I'll get to why that's bad
Speaker 1: later in detail. But it's also very funny. Now here's
Speaker 1: something else that's funny. Deep seek reasoner. It's reasoning model
Speaker 1: costs that fifty five cents per one million input tokens
Speaker 1: and two dollars and nineteen cents per one million output tokens.
Speaker 1: Now that sounds expensive. Maybe it is. Whatever, that's goddamn
Speaker 1: nothing compared to the fifteen dollars per one million input
Speaker 1: tokens and sixty dollars per one million output tokens of
Speaker 1: open ai WOF. If I'm Sam Orman, I'm shitting myself.
Speaker 1: But there's an obvious bar here. We do not know
Speaker 1: where deep seek is hosting its models, who has access
Speaker 1: to that data, or where that data is coming from
Speaker 1: or going to. We don't know who funds deep Seek
Speaker 1: other than it's connected to High Flyer, the hedge fund
Speaker 1: that I mentioned earlier that it's split from. In twenty
Speaker 1: twenty three, there are concerns that deep seak could be
Speaker 1: stayed funded, and that deep Seek's low prices are a
Speaker 1: kind of geopolitical weapon breaking the back of the generative
Speaker 1: AI industry in America. I'm not really sure whether that's
Speaker 1: the case or not. It's certainly true that China has
Speaker 1: long treated AI as a strategic part of its national
Speaker 1: industrial policy and is reported to help companies and sectors
Speaker 1: where it wants to catch up with the Western world.
Speaker 1: The Made in China twenty twenty five initiatives SAW are
Speaker 1: reported hundreds of billions of dollars provided to Chinese firms
Speaker 1: working in industries like chip making, aviation, and yeah AI.
Speaker 1: The extent of that support isn't exactly transparent, surprise, surprise,
Speaker 1: and so it's not entirely out of the realm of
Speaker 1: possibility that deep Seek is also the recipient of state aid.
Speaker 1: The good news is that we're going to find out
Speaker 1: fairly quickly. American AI infrastructure company Grok is already bringing
Speaker 1: deep Seek's model online, meaning that we'll get at least
Speaker 1: a very some sort of confirmation of whether these prices
Speaker 1: are realistic or whether they're heavily subsidized by whoever it
Speaker 1: is that backs deep Seek. It's also true that deep
Speaker 1: seek is owned in part by a hedge fund, which
Speaker 1: likely isn't short of cash to pump into them. But
Speaker 1: as in the side, given the open AI is the
Speaker 1: benefactor of billions of dollars of cloud compute credits and
Speaker 1: gets reduced pricing for Microsoft's zero cloud services to run
Speaker 1: its actual models, it's a bit tough for them to
Speaker 1: complain about Arrival being subsidized by a larger entity with
Speaker 1: the ability to absorb the costs of doing business should
Speaker 1: that be the case. Same goes for anthropic by the way,
Speaker 1: and yes, I know Microsoft isn't a state, but with
Speaker 1: a market cap of three point two trillion dollars in
Speaker 1: quarterly revenues, larger than the combined GDPs of some EU
Speaker 1: and NATO nations, it's kind of the next best thing.
Speaker 1: But I digress. Whatever concerns there may be about malign
Speaker 1: Chinese influence of bordering on irrelevant outside of the low prices,
Speaker 1: of course, offered by deepseek itself, and even that is
Speaker 1: speculative at this point. Once these models are hosted elsewhere,
Speaker 1: and once deep Seek's methods, which I'll get to in
Speaker 1: a little bit, are recreated, and by the way, that's
Speaker 1: not really going to take very long. I believe we're
Speaker 1: going to see that these prices are indicative of how
Speaker 1: cheap these models are to run. So you might be wondering,
Speaker 1: how the hell is this so much cheaper? And that's
Speaker 1: a bloody good question. And because I'm me, I have
Speaker 1: a hypothesis. I do not believe that the companies making
Speaker 1: these foundation models, such as Open Air and Anthropic, have
Speaker 1: actually been incentivized to do more with less. And because
Speaker 1: they're chummy little relationships with hyperscalers like Amazon, Google and
Speaker 1: Microsoft were focused almost entirely on making the biggest, most
Speaker 1: hugest models possible, using the biggest, even hugerris chips. And
Speaker 1: because the absence of profitability didn't stop them from raising
Speaker 1: more money. Well, they've never had to be fucking efficient,
Speaker 1: have they. They've never had to try. Maybe they should
Speaker 1: buy less avocado fucking toast. Anyway, let me put it
Speaker 1: in simpler terms. Imagine living on fifteen hundred dollars a month,
Speaker 1: and then imagine how you'd live on one hundred and
Speaker 1: fifty thousand dollars a month, and that you have to,
Speaker 1: like Brewster's millions, spend as much much of it as
Speaker 1: you can to complete a mission, a very simple mission. Live.
Speaker 1: In the former example, you concern survival, you have a
Speaker 1: limited amount of money and must make it go as
Speaker 1: far as possible, with real sacrifices to be made with
Speaker 1: every dollar you spent. If you want to have fun,
Speaker 1: you're going to have to eat less. Potentially all the
Speaker 1: food you eat will have to be cheaper. You have
Speaker 1: to live on a budget. You have to make decisions,
Speaker 1: and indeed you might learn to cook at home. You
Speaker 1: might walk more, you might do things that will help
Speaker 1: you not spend all your money. In the latter example,
Speaker 1: where you have one hundred and fifty thousand dollars a
Speaker 1: month that you must spend, your incentivize the splurge to
Speaker 1: lean into excess to pursue this vague idea of living
Speaker 1: your life, your actions are dictated not by any existential threats,
Speaker 1: or indeed any kind of future planning, but by whatever
Speaker 1: you perceive to be an opportunity to live. Open AI
Speaker 1: and anthropic are emblematic of what happens when survival takes
Speaker 1: a back seat to living. They have been incentivized by
Speaker 1: frothy venture capital and public markets desperate for the next
Speaker 1: big thing thing, the next big growth to build bigger
Speaker 1: models and sell even bigger dreams. Like Dario Amaday of
Speaker 1: Anthropics saying that your AI and I quote could surpass
Speaker 1: almost all human beings at almost everything shortly after twenty
Speaker 1: twenty seven, I just want to take a fucking second. Journalist,
Speaker 1: if you're listening to this, stop fucking quoting this bullshit.
Speaker 1: Stop it. You're doing nothing. You are failing at your
Speaker 1: goddamn job every single time you quote this bullshit, this nonsense.
Speaker 1: Shortly after twenty twenty seven. What the fuck does that mean?
Speaker 1: Twenty twenty eight, twenty twenty nine, twenty thirty, what does
Speaker 1: surpassing humans and almost everything even mean? This shit doesn't work.
Speaker 1: This shit is not good. Oh my god. Anyway, back
Speaker 1: to the podcast, the Calm Damn. Both Open AI and
Speaker 1: Anthropic have effectively lived their existence with the infinite money
Speaker 1: cheap from the SIMS. And I know some of you
Speaker 1: might say, by the way, it's not an infant money,
Speaker 1: you just add you go into the console. You get
Speaker 1: my point. And both companies have been bleeding billions of
Speaker 1: dollars a year after revenue, and that's, by the way,
Speaker 1: making billions of dollars and then still losing billions is insane,
Speaker 1: and they still operated as if money would never run
Speaker 1: out because it and it wouldn't. If they were actually
Speaker 1: worried about that happening, they would have certainly tried to
Speaker 1: do what Deep seek has done, except they didn't have
Speaker 1: to because both of them had the endless cash and
Speaker 1: access to GPUs from either Microsoft, Amazon or Google. And
Speaker 1: the stargate thing is just I will mention it later,
Speaker 1: just long story short. They're not going to put five
Speaker 1: hundred billion dollars into the it was up to five
Speaker 1: hundred bill I'm so tired of this shit. Open iron
Speaker 1: anthropic have never been made to sweat, unlike me in
Speaker 1: this closet where I'm recording this. And they've received endless
Speaker 1: amount of free marketing from a tech and business media
Speaker 1: happy to print whatever vapid bullshit they spout, and it's
Speaker 1: just very frustrating. They've raised money at will with ananthropic,
Speaker 1: by the way, is currently raising another two billion dollars,
Speaker 1: valuing the company at sixty billion dollars. And this was
Speaker 1: I think happening while deep Zeek was going on, which
Speaker 1: is really funny. And they've done all of this off
Speaker 1: of a narrative of them. We need more money than
Speaker 1: any company is ever needed ever because the things we're
Speaker 1: doing have to cost this much. There is no other way.
Speaker 1: You must give us more money. My name is Sam Altman.
Speaker 1: I need more money than has ever been made from
Speaker 1: my huge, beautiful company that sucks and needs money to
Speaker 1: train it. Help me, please, My big, beautiful sick company
Speaker 1: is dying, but the best and most important company of
Speaker 1: all time. It's also normal. Now. Do I think that
Speaker 1: they were aware that there were methods to make their
Speaker 1: models more efficient? Sure, open AI tried and failed in
Speaker 1: twenty twenty three to deliver a more efficient model to
Speaker 1: Microsoft called Arakis. I'm sure there are teams that both
Speaker 1: Anthropic and OPENII that are specifically dedicated to making things
Speaker 1: kind of more efficient. But they didn't have to do it,
Speaker 1: and so they didn't. And as I've written before in
Speaker 1: my newsletter and argued on this very podcast, open AI
Speaker 1: simply burns money and have been allowed to burn money,
Speaker 1: and up until recently likely would have been allowed to
Speaker 1: burn even more money because everybody, all of the American
Speaker 1: model developers, appeared to agree that the only way to
Speaker 1: develop large language models was to make them as big
Speaker 1: as humanly possible and work out troublesome stuff like making
Speaker 1: them profitable or turning them into a useful thing later,
Speaker 1: which is I presume when AGI happens, a thing that
Speaker 1: they're still in the process of defining, let alone doing.
Speaker 1: Deep Seek, on the other hand, had to work out
Speaker 1: a way to make its own large language models within
Speaker 1: the constraints of the hamstrung end video chips that can
Speaker 1: be legally sold to China. While there's a whole cottaged
Speaker 1: industry of selling chips in Chines using resellers and other
Speaker 1: parties to get restricted silicon into the country, the entire
Speaker 1: way in which deep Seek went about developing its models
Speaker 1: suggests that it was working around very specific memory bandwidth constraints,
Speaker 1: meaning that the amount of data that could be fed
Speaker 1: into it and out of it and into the chips.
Speaker 1: In essence, doing more with less wasn't something it shows,
Speaker 1: but it's something they had to do. I've touched already
Speaker 1: on the technical how of these models in greater depth,
Speaker 1: and you can really read in that in my news
Speaker 1: there and you can go to whez your hed not
Speaker 1: at it's at the end of the episode. But I'll
Speaker 1: also have show notes to what cales like Ben Thompson's
Speaker 1: some strategory because there are lots of things to read here.
Speaker 1: I know there are some really technical listeners, and I'm
Speaker 1: sure you're gonna flame me in my emails. Please go
Speaker 1: and read it. I'm not wrong. I've checked with a
Speaker 1: lot of people too, and by the way, all of
Speaker 1: this austerity stuff seems to have worked. There's also the
Speaker 1: training data situation and another mayor culper. I've previously discussed
Speaker 1: the concept of model collapse and how feeding synthetic data,
Speaker 1: which is training data created by a generative model, into
Speaker 1: another model, could end up teaching it bad habits, which
Speaker 1: in turn would destroy the model. But it seems that
Speaker 1: deep Seekers succeeded in training its models using generative data
Speaker 1: specifically though, and I'm quoting geekwise John Turou like mathematics
Speaker 1: where correctness is unambiguous, and using and i quote again,
Speaker 1: highly efficient reward functions that could identify with which new
Speaker 1: training examples would actually improve the model, avoiding wasted compute
Speaker 1: on redundant data, and it seems to have worked. Though
Speaker 1: model collapse may still be a possibility. This approach extremely
Speaker 1: precise use of synthetic data is in line with some
Speaker 1: of the defenses against model collapse I've heard from LLLM
Speaker 1: developers i've talked to. This is also a situation where
Speaker 1: we don't know the exact training data, and it doesn't
Speaker 1: negate any of the previous points I've made about model collapse.
Speaker 1: Now we'll see what happens there. But synthetic data might
Speaker 1: work where the output is something that you could figure
Speaker 1: out using a calculator. But when you get into anything
Speaker 1: a bit more fuzzy, like written text or anything with
Speaker 1: an element of analysis, you'll likely encounter some unhappy side effects.
Speaker 1: But I don't know if that's really going to change
Speaker 1: how good these things are. There's also a little scuttle
Speaker 1: about about where deep seak got its data. Ben Thompson,
Speaker 1: that's trategory suggests that deep seek's models are potentially distilling
Speaker 1: other model's outputs, by which I mean having another model,
Speaker 1: say metas LAMA or open ais GPT four H, which
Speaker 1: is why deep seak identified itself as chet GPT at
Speaker 1: one point spit out outputs specifically to train parts of
Speaker 1: deep Seek. This obviously violates the terms of service of
Speaker 1: these tools, as open AI and its rivals with much rather,
Speaker 1: have you not use its technology to create its next rival.
Speaker 1: And open Aye, by the way, has recently reportedly found
Speaker 1: evidence that deep seek used open AIS models to train
Speaker 1: its rivals. And this is from the Financial Times, although
Speaker 1: it failed to make any formal allegations, but it did
Speaker 1: say that using chat gpt to train a competing model
Speaker 1: violates its terms of service, and David Sachs, the investor
Speaker 1: in Trump administration AI and cryptos are, says it's possible
Speaker 1: that this occurred, although he failed to provide evidence. I
Speaker 1: just want to say, how fucking funny it is that
Speaker 1: open air is going where where you're stealing my stuff?
Speaker 1: Don't steal my things? Where fucking coward, pansy bastard bitches.
Speaker 1: Fucking hell, what a what a bunch of whiny babies.
Speaker 1: Oh no, my plagiarism machine got plagiarized. Where kiss my
Speaker 1: entire asshole, sam Orman, you little worm, you fucking embarrassment
Speaker 1: to Silicon Valley. You should be ashamed of yourself for
Speaker 1: many reasons, but so much this though. Where Yeah, oh no,
Speaker 1: you stole from use my plagier is the machine that
Speaker 1: requires me to steal from literally every artist and author
Speaker 1: on the Internet. The thing where we went on YouTube
Speaker 1: and transcribed everything and fed it into the machine. That's
Speaker 1: that's not stealing, that's good. But you using our model
Speaker 1: to generate answers. That's just not fair. What a bunch
Speaker 1: of babies, you guys say. I'm almos worth billions of dollars.
Speaker 1: He has a five million dollar car. Cry more, you
Speaker 1: little worm. Personally, I genuinely want open ai to point
Speaker 1: a finger at Deep Seek and accuse it of IP theft,
Speaker 1: mostly for the yucks, but also for the hypocrisy factor.
Speaker 1: This is a company that, as I've just very cleanly said,
Speaker 1: exists purely from the wholesale industrial larceny of content produced
Speaker 1: by literally a fucking everyone, And now they're crying way.
Speaker 1: I'm sam Olman. I'm a big baby. I've filled my
Speaker 1: diaper because someone stole from my plagiarism machine. Kiss my ass,
Speaker 1: Kiss my ass. These companies haven't got shit. Open ai
Speaker 1: doesn't have shit. They they don't have anything, They don't
Speaker 1: have a next product without reasoning, they haven't got anything.
Speaker 1: And now they don't have that disgusting justification that overspending
Speaker 1: the fat, ugly American startup culture of spending as much
Speaker 1: as you can to build America's next top monopoly. They
Speaker 1: should be fucking ashamed of themselves. They shouldn't be billionaires,
Speaker 1: they should be poverty stricken. They should have to pay
Speaker 1: everyone they stole for And it's just it sickens me
Speaker 1: seeing the reaction from some people on this, seeing the sinophobia,
Speaker 1: but seeing this level of defensiveness of a company like
Speaker 1: open AI or Anthropic, And as I'll get into next episode,
Speaker 1: we are really running out of time here, and I
Speaker 1: think Deep Seek is really I think it could be
Speaker 1: really the end of days for these companies. I don't
Speaker 1: know how much they've got left time wise, or even
Speaker 1: money wise, and I'm not sure how they even raise money.
Speaker 1: But in the next episode, I'm going to deep dive
Speaker 1: into Deep Seek and I'll tell you how they sent
Speaker 1: the US tech market into a panic and what it
Speaker 1: actually means the future of open Aianthropic and the hyperscale
Speaker 1: is backing them. This has been a crazy few days.
Speaker 1: I hope this has helped, and on Monday you'll find
Speaker 1: out more. Thank you so much for listening. The support
Speaker 1: I've got for the show has been incredible, and the
Speaker 1: emails I've got about Deep Seek. I've been trying Okay,
Speaker 1: I've really been trying so the fastest I could do it.
Speaker 1: But I'm so happy to do this show, and I'm
Speaker 1: so grateful for all of you. Thank you for listening
Speaker 1: to Better Offline. The editor and composer of the Better
Speaker 1: Offline theme song is Matasowski. You can check out more
Speaker 1: of his music and audio projects at Matasowski dot com,
Speaker 1: M A T T O. S O w Ski dot com.
Speaker 1: You can email me at easy at Better Offline dot
Speaker 1: com or visit Better Offline dot com to find more
Speaker 1: podcast links and of course, my newsletter. I also really
Speaker 1: recommend you go to chat dot where's youreaed dot at
Speaker 1: to visit the disc and go to our slash Better
Speaker 1: Offline to check out I'll Reddit. Thank you so much
Speaker 1: for listening. Better Offline is a production of cool Zone Media.
Speaker 1: For more from cool Zone Media, visit our website Coolzonemedia
Speaker 1: dot com, or check us out on the iHeartRadio app,
Speaker 1: Apple Podcasts, or wherever you get your podcasts.