📆 ThursdAI - Jan 30 - DeepSeek vs. Nasdaq, R1 everywhere, Qwen Max & Video, Open Source SUNO, Goose agents & more AI news

Welcome, everyone.

Welcome to Thursday. January 30th. we had a lot of Thursdays this January. my name is Alex Volkov. I'm an AI ventures here with weights and biases. And I'm very, very, very energetic and excited today to bring you, Thursday. I, our weekly AI show that's recorded live on X on YouTube and LinkedIn, everywhere that you can possibly see us. So thank you everyone. Who's joining throughout all of these different platforms as we record live. The show is also then turns into a newsletter and a podcast that you can catch everywhere you get your podcasts. with me today, my cohost Wolfram, welcome back Wolfram.

Hello, I'm glad to be here again.

I missed the last episode, so really happy to be here with all the news.

You missed it, but you met with some of our

community as well, which is great.

AI Dev in Cologne.

That was a German AI developer meeting. It was great. We had people, someone even came from France. Friends to us because of the show, because of the podcast, because we mentioned it.

Were you able to talk about anything else

besides DeepSeek is my question.

at that moment, it wasn't even the big news there.

Yeah, it was more about operator agent and stuff like that. So the big news happened a bit later, I think.

Oh, Yeah.

That was like Thursday. And then we talked about our one on our show last week. you weren't here, but you guys already covered this as well. It came out, we just talked about this and then the whole weekend. And until now, there's so much deep sick in the news that we have to at least. We're going to do a meta analysis. We're not going to talk about what they talked about. We're going to talk about Nistan, go ahead.

We had a whole hype session about it in November.

I was watching the November, 20th or 12th show. And, Jan was in it and R1 preview had come out and we all got excited and stuff, and then just like forgot about it. And, yeah, didn't expect that to completely blow up after.

Absolutely.

We covered, dipstick R one. Back then it was R one preview. It only came out as deep think in the interface and then they didn't launch anything. We always knew that Dipsy is coming. They told us we're gonna release this, then they released it. We're very happy. Ta-da, new reasoning model. State of the art and then the world lost its goddamn mind with deep sea. It's incredible. My mom knows about this. Like grandma's know about deep sea. It's number one. Yeah, exactly. It's number one on the app store. It's it's everywhere. The, our AIs are for United. States talks about distillation in a non exactly completely correct way. There's so much about deep sick this week that we're not going to cover like what happened this week with deep sick, we're going to do like a meta analysis, what like people thought and some reactions, Nvidia commented back and Microsoft commented back, there's the special opening, I know to release some stuff. So we're going to have some, breaking news this week. but not only deep sick. So deep sick was definitely. Another week highlight of the news. so like rarely do we have an occurrence like this where something comes out. And folks, remember we cover week to week, we're a weekly show. We cover pretty much basically only news in the span of the week. I have a bunch of people who tell me, Hey, Alex, I want to come on the podcast. I was like, okay, what did you do this week? I'm sorry that I sound like Elon, but what did you do this week? Because if it's not this week, there's other shows for you. Rarely do we have an occurrence such as DeepSeek R1 that we can talk about from week to week afterwards. Very rarely. And DeepSeek has taken more of the airways this week than last week, I feel like. Even though, yeah. So it's been crazy. But, besides DeepSeek, there's just so much that happened this week. on top of this news, and in addition, there's a bunch of reasoning stuff. we should definitely cover. So let's just dive in folks. Let's dive in. I want to hear from the folks. We now have a bunch of folks in the studio right now, with us, in, into our spaces live on the video as well. So if you're just stepping in, please go to the bottom, And tell us if your mom knows about deep sick. I want to know, my mom knows about DeepSeq, WolframMom knows about DeepSeq, and tell us if your grandma knows about DeepSeq, it's very interesting. Or if your parents to kids of 13 and up, do they know? because it absolutely broke through the AI kind of bubble area, which is very interesting. Meanwhile, I will also say, if you have other news that you're excited about, please tell us because I definitely have a bunch of news that I'm very excited about. And for that reason, we'll do a TLDR. as we always do for folks who are new to the show, we basically run through super quick about. Maybe 10 to 15 minutes about everything that we saw that happened this week. this serves two purposes. One of them, if you don't have the two hours to sit with us, as we're here for at least two hours lately, it's been crazier and crazier. It's been harder and harder to not be here for longer. as you sit with us, if you don't have that time, you can just. Absorb the TLDR, know everything that happened, and then our commentary about this you can follow up later. Another one is, we often don't get to talk about the things we want to talk about because there's breaking news. For example, today, we already have some breaking news from Mistral, we already have some breaking news from Refusion. But also, I'm pretty sure we're going to have breaking news today from OpenAI. I don't have prior knowledge, folks, even though, OpenAI and Wysys, we work together. I don't have prior knowledge, to be very clear with you, about a specific date. I just saw that, some influencers, that are not me, that did get some embargo, broke that embargo, right? if I have an embargo, I won't tell you about it. Trust me, I will try not to screw this up. However, we have a bunch of other stuff. So let's run through the TLDR as we always do. And I think it's a very dense piece of commentary right now. So let's run because there's a lot to talk about besides DeepSeek. obviously we're going to cover DeepSeek, crashing the stock market, reactions, Dario Amodei's reaction, Sam Altman, Microsoft, like all of these reactions we're going to talk about. On the wave of reasoning in DeepSeek, a bunch of stuff came out. One interesting thing is this open thoughts, reasoning data set. So folks started saying, Hey, it's great that we have this like state of the art reasoning model, but we need more chain of thought data sets. And so these folks released on open thoughts, a large scale reasoning data set, it's 114, 000, distilled versions of chain of thought, like reasoning, I would say. connections or I don't know, traces, and we're going to cover that this a little bit. we're now in the open source LLMs kind of TLDR. We also just have breaking news from just today from our friends at Mistral. Finally coming back to us, understanding that, hey, Maybe one of the reasons why the world jumped on DeepSeek is open source and Mistral has not been releasing open source lately. And you guys know we love the Mistral friends. and finally We got some open source from Mistral Mistral today releases Mistral Small 2501, which is a 24 billion parameter model that, gets like 81 percent on MMU. It fits neatly in the 4090 or a 32 gigabyte Mac. great model looks, and finally for the happiness of all of us they finally understood that not mentioning Quinn in their evals is a no, in the community, they're going to get the goose. You guys know the goose meme was like, where's Quinn? Where's Quinn? Finally, the ad Quinn 2. 5 in their evals as well. so thank you, Mr. Al for listening to the community and understanding that we need this comparison. We're going to do it anyway. Might as well put it in there. but congrats on Mistral, it's an Apache 2 license that you can download finally from Mistral, very happy about this. we also had two replications that I could follow of R1 0. something we didn't mention super quick last week is that R1, DeepSec R1 released basically two main models, R1 and R1 0. And the difference between was, R1 was trained with human, chains of thoughts as well. R1 0 was, Almost entirely RL'd and almost entirely synthetic, as far as I understand. And so we had two replications, one from Berkeley, one from Rajen. Rajen is a, reproduction from some folks who were at DeepSync before, actually. And, they both tried to reproduce R1 0. and, Berkeley is called Tiny0, and they claimed that they reproduced it for 30. That's crazy. That's absolutely crazy. They didn't reproduce fully, but they claim that you can get that aha moment. If you guys remember, we talked about deep sick, getting that aha moment where the model starts reasoning and like researchers like, Whoa, holy shit. It's reasoning. So they claimed that you could get to this aha moment with under 30, which is incredible. And Rajan is also another repro of our one zero. They claim to be the first. And, testing on Jim Sokoban. So like very small, they, I don't believe they even release models. but, Berkeley's one, they released the weights and biases logs, which I love. you can per use them. So we're going to take a look at them. anything else in open source that I missed Wolfram, Nisten, folks in the audience, anything else in open source that's important for us to cover?

think that covers it.

Yes.

Yes, there is.

There's a big news, Tulu 405B.

That's right.

From our friends at Allen AI, we have another huge model just like literally breaking news from today. I just like barely saw it. Tulu 405B. do we know anything? What are the stats? Do we know? What do we know?

it looked really good on the benchmarks and, the

thing about Tulu is that you can apply it to anything you're doing. Any model here because they are actually fully open the data even the code I think the logs yeah, it's i'll post some stuff as I find more. I don't want to say anything.

And also Nistan coming to the actual studio, we need

to hear you and see you as well. Alrighty. so yeah, TULU 405B from our friends at Allen Institute for AI. And as always, the stuff that Allen Institute does is always completely open and it's great. And training this model to 405B, I'm assuming that's not an MOE. So it's like a dense 405B model. this. Could be a fine tune of llama. Yeah, we'll take a look. So definitely breaking news in there. Ah, this is all an open source. Folks, we have a new category called agents. It's about time. 2025 is going to be the year of agents. And so far agents we like, I didn't give them a specific, area. But now we have a specific area. It's called agents. So let's talk about what's new in agents this week. There's two things. First of all, operator, if you guys remember last week, Wolfram, you weren't here, but we watched the operator news live on the show, as they happened, like we were speculating is operator going to come up in the I launched operator, their agents, and we tuned in live. And that's why the show last time took so long. and then we watched it and then. I hit that button of 200 bucks immediately and I spent the 200 bucks for you guys and I play with operator the whole week. So I have a bunch of thoughts about it and I also built like a Mac OS app for it. I have a bunch of thoughts about operator, so we're going to talk about them, but also in the news of agents blocks. with, Jack Dorsey's company, also released a open source agent called project goose. And, it's a framework, but also an app that you can run and it has extensions and can tune in. you can use any other kind of. via open router, you can use pretty much anything you can use. All llama, you can use a GPT four. You can use like a bunch of models. If you have the API keys, they have extensions. It's pretty cool. Does not work as well as you might imagine, like conceptually, it's cool. So we're going to talk about goose in the world of agents. folks, the world of agents moving like super, super fast. I will say I, If you're following the Worlds of Agents, closer than I am, please drop in the chat everything that happened in the World of Agents that I may have missed. Those are the two main things for me, but I'm no I know for a fact that, the Agent world moves sometimes as fast as the rest of the world. so please drop Anything that happened in the world of agents, in, in, in the comments, or we'll be able to cover this moving on to big companies, LLMs and APIs, which is like also a big thing. the reason why wait, no open source. We missed one big thing. One huge thing. okay. We're going to cover this because, this is open source LLMs. We also have a vision and video category where we also cover open source. And another thing that happened in open source this week was DeepSeek. Dropping Janus pro. So not only did they drop like a model that changed the world and grandma's know about this and is number one on the app store. DeepSeek also dropped Janus. Janus is very interesting. very interesting. It's a model that generates images. So like a diffusion model, but also understands images. So it's like a duality model that understands. I don't know why fireworks are flying behind me on the video stream, folks. It's very funny. But, DeepSeek just released also a state of the art model, state of the art among the models that can understand and generate images. It's very interesting that they released this. it's Janus Pro. So they already released Janus before. This is Janus Pro. also Envision Video, also an open source. And also something we covered last week that they yanked it. NVIDIA brings back Eagle 2. Eagle 2 is a high resolution multimodal understanding model from NVIDIA, and last week we tried to cover it, and the models were yanked, and so we didn't cover it, but Eagle 2 is back, so they re released it, so we didn't get the wizard LM treatment, and now we have Eagle 2, which is great, and it looks like we got three state of the art vision models, because Quen 2. If you guys remember, Quen 2 VL was one of the top vision models in the open source. Quen decided to give us a Chinese New Year present in the form of Quen 2. 5 VL, which is a vision model, from Quen, based on the Quen 2. 5. It's, they claim state of the art OSS vision. As well, and in vision and video, we also have two video models, big updated one. And the high Leo released a director model. You can actually tell it what to do. So those are the open source. I think we covered everything in open source.

And another video generation model that was released

Yes, but it's not an open source.

there's more,

Mr. Which

one?

Listen,

There is, one, million context models from Quan also, when 2.

5, 72 B multimodal one, which, people already started using as an operator, as an open source, 2.

5 VL, right?

2.

5 VL, but also the 1 million complex models. no, they are, fully open source. they're on hugging face. I already started using that. I was going to post some stuff. actually we're trying to deploy and get enough compute.

All right.

And so we're in the TLDR. We're going to cover them. once we go to the actual full coverage, then we're going to cover them. big companies, LLMs, this is our next kind of like TLDR section. Alibaba, Quen, the folks from Quen, not only dropping open source models, they also dropped Quen 2. 5 Max. Which is their kind of like main reason there. If you go to their chat, you can play with their model. This model comes close to GPT 4 and beats GPT 4. I think it's also available in the API. a great model overall with like very good performance. 87 in MMLU, 89, 69 in MMLU Pro. very good model. Very powerful model. However, the one thing that's interesting to me is that they also dropped a video model that nobody talked about. You can generate video on Quen now, folks. They didn't say anything. Nobody knows about this. They didn't announce. There was no announcement. But you can generate images and video, and those look good. very good. Very close to state of the art. shout out to our friends, Junyang and Baiwan and like other folks from Malibu Kuen. They're usually here on the show. However, I will say, they are on their very well, deserve rest right now. So they're not on with us on the show, but, yeah, Kuen dropped a lot of stuff before the Chinese New Year. also in the big companies and the APIs, there was an earnings call from Meta where Zack talked about multiple things that are interesting to our crowd, including the fact that Llama 4 and is going to come as a mini. So it's going to be Llama 4 and Llama 4 mini. very interesting because they usually just release like smaller models, right? But now they actually have a new name for this. we got llama in smaller models, but now we have llama for mini. Not only that, he mentioned that there will be a reasoning model for llama, which makes sense because well. as part of the, we're going to talk about the reactions to DeepSeek, but as part of the reactions, there was this like famous blind post of somebody that works at Meta claiming that there are executives in Meta that get paid more than what DeepSeek was trained for, which, probably an exaggeration, but it was like very funny thread. Zach is talking about Llama4 mini and talks about how. open source is very important. And I heard also rumors that they're going to implement deep seek inside meta. So that's very interesting. as we said, OpenAI is likely about to launch O3 mini today, according to some, influencers that broke embargo, not, we don't know anything before, but yes, this is going to be very exciting and we're going to talk about this, O3 mini. Once it launches and also we potentially gonna get some new Gemini 2. 0 news soon it's been leaked But also grok 3 is starting to show up to some people and Mr. Wolfram like we had folks in our chat That they talked about the grok 3 showing up for them So we don't know anything, but we know it's coming. We know it's coming and We know that potentially There's going to be a reasoning model for Grok as well. Somehow we also know this. So reasoning is all around. Anything else in big companies that I missed?

I posted something in the chat.

it's not a model, but it's a Lama stack for meta. That sounds very interesting to compose an application on all.

stack is like an OS that you can run a bunch of stuff.

Like it is the whole. get started with Lama's things. They released like an

integrated environment in a way.

I

found it very interesting.

And that is an open source actually.

So we can cover this in open source. anything else in speculation with them usually do a speculation. We can skip, let's see what. Else. Ooh, yes. And this week's buzz, this week's buzz. So last week we talked about how weights and biases got state of the art in sui bench verified with our WNB programmer, agent. this week I can tell you that Sean Lewis, the CTO went on Latent Space podcast with our friend Suix and they actually talked about how they achieved this. So if you're interested in that, definitely check that out. I will also tell you that again, if we're talking about Suix, our friend, AI engineer. Conference summit is coming up in New York and I'm going to be there with somebody who's going to sponsor this. we just closed the deal and I'm going to be there together with the whole team. So if you're coming to AI engineer, which you should, if you don't have tickets yet, I think they sold out, but don't trust me, go to AI that engineer and check it out. I think there's maybe one more ticket a day after this though, me and another friend here, the Canadian friend over here that sits with me here, we're going to meet up because I'm coming to Toronto folks for the first time since I started my journey, for the, I think. So if you're in Toronto, we know there's a bunch of people, AI thinkers in Toronto. There's 800 people signing up for each event. we're going to do two workshops about evaluation. So if you're in Toronto, you're listening to this, go sign up to AI thinkers. We're going to get an email very soon. We're going to come and do some workshops. and this is going to be there probably also. he doesn't know about this yet, but you're invited.

You picked literally like the coldest time where your

eyelashes like freeze outside

I'll bring

come visit Yeah

I'll bring all my layers folks all my layers So

hopefully in the university of toronto is gonna be auditorium. We're gonna talk about evil's gonna be awesome. I think we're still in the TLDR, folks, because there's still like a bunch of stuff that happened. Voice and audio category is popping up because we finally have open source Suno, folks. It's crazy. I don't know if you heard about this, but UE7B is a 7 billion parameter model that comes fairly close to Suno and Udio and Refusion, which is a new one. And you can generate full songs. It's non commercial, but it's like, it runs a 7B model. It takes a while. But at the end of this, you get like a nice sounding songs. I think it's the highest quality that I heard in open source, absolutely highest quality that I heard in open source. so OpenSUNA we'll call, we'll just call this OpenSUNA for now. OpenSUNA from HKUST, HK University and, Art Projection, Multimodal Art Projection, those are the releases, but also we have a new contender for the music generator of the year. It's called Refusion with Fuzz, and that one is free. Until their GPU burnout. So if you want to generate music for free, you don't want to pay sooner or audio. go to refusion. They just released it today. I think also breaking news, but we're obviously more excited about the open source one and nothing in AI art in the fusion this week, probably, but. there's another category called tools and because everybody is now integrating R1 into the reasoning. I don't know why they didn't do this with R1 because the API was there. oh no. It's pricey. Yes. First of all, it's pricey. Second of all, it wasn't there. one API was only available for tier five and tier four. And DeepSeek is now yeah. So everybody's integrating DeepSeek basically. and not only that, because it's not in the whole point is like DeepSeek runs on our servers and not in China, perplexity with R1 is really good. My perplexity, like subscription lapsed this week without me noticing. I probably would have canceled it, honestly, between us, friends. but it lapsed and I was like, okay, fine. I'll another year with perplexity pro this time I actually paid for it, but they added R1 reasoning and it's really good that you have to drop down. Exa, which is a search API that kind of like. You can use for your agents also add the deep seek chat as well. And it's also does deep research for you. and also Cray, our friends from Cray, I also had the deep seek as a deep seek. So everybody's adding reasoning now because it's possible. so this is in tools. I think that this is our TLDR folks. I think let's actually start talking about this stuff. including, oh yeah, somebody mentioned this and I want to add this as well. In agents, browser use, is a alternative to operator. So we'll add this to agents. browser use is an open source browser. Use, thank you, Quinton. browser uses an open source alternative to agents, and operators specifically that you can run and you can change different LLM providers. and as we know now, Denison just mentioned, Quen 2. 5 VL is agentic to an extent as well. And, Lama is going to be agentic as well. So we're going to have agents and reasoning agents and, all kinds of things on the show. So with this, I think. After quite a while of an intro, it's getting longer and longer to the DLDR. if there's not anything else, folks in the audience, and if there is, please tell us about this. Let's get to open source. Let's get to open source.

Open source AI. Let's get it started. Welcome to. The open source segment of Thursday. We're going to talk about LLMs mostly. we're, we have to start with DeepSync folks. We have to start with DeepSync because it's been everywhere on the news this week and not necessarily from the way we cover DeepSync, which is, Hey, there's this reasoning model, it's test time compute. And it's like this on, I, AIME score and blah, blah, blah. DeepSeek was everywhere this week, not because it was open source, it was because, they also released an iOS app that started, getting, number one in, different app stores, but also because, when it released, one of the things that was supposedly True is that it was trained with around 5. 5 million, which completely threw in the disarray, the whole discourse about what it means to release models, what it means about open source, what it means when Meta has five executives. That together get paid a year more than like it used to train DeepSync. And they're spending billions of dollars on infrastructure. And why? this caused a panic, market. NVIDIA market crashed a little bit by 17%. I found this very interesting stat that, NVIDIA crashed by five intels. So in one day, NVIDIA lost five intels worth of like market share. they gained it back. Let's talk about this just briefly. Like what's, what do we subscribe to this doomerism approach of a deep six release means less GPUs folks. I doubt that we are like, we're not Mr. Market, but I want to hear from you Wolfram.

they would have crashed much more if people really believe that.

I think the people who are investing in Nvidia know that, now you have a model that needs less, but then you can run more of it. And, I think that is what made DeepSeek so popular for everybody because it wasn't in use because of the stock market crash,

not

because it's such a big model, but the outcome it achieved.

And then everybody heard about it and then they went curious about it and got the app and, it went through the roof that way. But I don't think Nvidia is endangered or anything like that. And, yeah, we need GPUs and if you have the GPUs, you can run DeepSeek even better. So of course I think people have realized that by now

Yeah.

Nistan, what's your take on this whole like public market insanity for a few days?

I think there were a lot of other things that play in

the market and this just set off. Avalanche a little bit remember at the end of the day the market's always smarter than you because it is like the collective Latent space and it does tend to make better decisions than most people even though you might not initially agree. Now we can interpret that in many ways and we can say, yes, there's going to be an increase for more, a lot more GPU usage. At the same time, DeepSeeker runs pretty fast on CPUs and this is the best model. So I think it was more of a, I don't know. Emotional reaction and, an avalanche of ideas that a lot of the companies which were over financed and, overcapitalized should not have been. And, yeah, but the things that the next day picked back up. So it's, yeah, this is not financial advice. It's just the market just goes, it goes up and down, but that's what it

So some of us were able to, top up some Nvidia on the discount.

That's all I'm saying. I will comment on this, quickly and say that, yes, open source. It's open source that is coming very close to or bidding on multiple things, from China, which is very interesting. the app is free. even if you want to pay a little bit, the app is very simple and free, there's definitely some consideration of market manipulation, at least in people whether or not this app was pushed, et I haven't seen any, but I saw multiple conversations about this on X, whether or not this was manipulation. suffice it to say, it got all the way to, White House. I heard the president, commenting on this, his AI czar is talking about distillation. There was news that OpenAI found, evidence that DeepSeek accessed, and used O1 for train. two things about this. One of them is OpenAI does not expose. Reasoning traces. So whatever happened, probably happened on GPT four, et but also let's be very frank, openly, I trained their models on the public internet pretty much. And coming out and say, this, like comment, this is not my personal take. I don't care. Take my information, train on it. Let's get better LLMs for all. But some people are like raising their eyebrow and saying, Hey, you cannot like train on the public internet. And then cry about it when people train on some of the stuff that you trained on. so it was a very interesting reaction to OpenAI. there's other public reactions like, Dario Amadeus, the CEO of Entropic blog, where he talked about the increasing need in, chip exports, restrictions, because he basically said that, it's incredible what DeepSeq was able to achieve. He also talked about the fact that, while the numbers are potentially true, it doesn't mean that this is the whole number. So DeepSeq also spent some money training DeepSeq, v3. On top of R1, like R1 is related as well. So that kind of pricing didn't go into there They also shared some very interesting highlights. Finally from Antropic that we haven't talked about that Cloud Sonnet 3. 5 was trained with tens of millions of dollars billions not hundreds of millions tens of millions of dollars they also shared that, Cloud Sonnet was not a distillation of a bigger model, of a reason model. so we, we got some actually information finally from folks because DeepSeq shake the industry. So a bunch of like public reactions as well. very interesting. so after OpenAI mentioned that they have found evidence of distillation, evidence of traces being used from DeepSeq, Microsoft, a day after this, Added deep seek to Azure, which is a very interesting comparison of things. There's a, there's a selfie of Sam Altman and, Satya Nadela, talking about you guys are not ready for Microsoft and open the eyes like next chapter. And like a day afterwards, Microsoft, ta-da, deep seek R one is now in Azure. It was really just comical, almost, consequence. what else do we have about Dipsy?

Look, having an open source model is a very

Libertarian free market thing to do. It actually benefits the whole economy quite a lot. And it's very unfortunate to see that the reaction from two of the top players is, yeah, just build the great firewall of America, bro. like that, that's gonna help things. it's not, the only thing They can do now because the public knows they can run very good intelligence privately and have control over it. So the only thing they can do is actually open source some get some engagement, get a lot more people using them and just stop trying to build a spy apparatus. To just suck everybody's data back to your service The public has realized it so it's too late again. This is my Opinion my take as some dude in canada on this So this is not advice or anything, but remember the Chinese government, the CCP, the reason that they allowed them to get away with deregulation. they didn't have to open source it or anything, but the reason that they did is because at the end of the day, the interoperability. of it, which is something that actually Zuckerberg has been saying, like the interoperability that it enables, like the graph, theoretic benefit of the technical capital machine, some people like to say, that is immensely beneficial. And, it's not beneficial to an oligarchy of like only top two players and stuff, but it is very beneficial to an ecosystem. And that's why they've allowed that because at the end of the day, it just brings them economic benefits and brings them better relationships with everybody else. And, what we're suggesting from, companies here, instead of them actually contributing to get, I don't know, why are they not getting open source medical models to literally everyone so that they can just run them privately, that you can do that, just make one for that, instead, now the whole world is going to be using Chinese models because You tried to do it in a bad faith way where you were trying to do carrot and stick and trying to suck all the information in and block all the other players and try to pass laws to literally put open source developers in jail, and now you're losing. So there's, yeah, what it is. that, that's my. Take on it. You're always going to be second place with that mentality.

There were also like all of this outcry about

DeepSea being Chinese, et cetera. So there's two things I want to say about this. First of all It could

have been anything.

No, but you mean like I specifically talk about like the

clutching of the pearls regarding the information of yours that goes to China. First of all DeepSeek R1 is the model, DeepSeek, the iOS app. Yes. They send information to the servers in China and they collect, blah, blah, blah. Okay. Don't use that. Just don't download. There's also, multiple places now that host DeepSeek directly because they opened the source. So for example, together. Fireworks, Grok, folks from Grok added DeepSeek, they didn't add the full DeepSeek, I'm waiting for the full one. But they added DeepSeek, Distill from Llama70B, and that runs at 300 tokens a second. I don't know if you guys remember, we talked about how the LPUs and the TPUs of the world are going to be beneficial for, reasoning. We can see this open AI if there's anything that like open AI doesn't have LPUs and TPUs open AI runs on GPUs from Nvidia And so they have all three mini that's gonna come at some point very soon. Hopefully today but We talked about how these like new approaches to specific GPUs are going to get beneficial, additions from reasoning models, because reasoning models, the longer they think, the better they are. And if you condense down the thinking into 300 tokens per second, you can see much better impressive results for smaller models. And seeing DeepSeek, 70BR1 distil running at 300 tokens per second in Grok. It's just incredible because you get like the reasoning, like poof, it just happens. Gemini thinking as well. Listen, we talked about this briefly last week, how good this is. It keeps getting very good. I think that like people are just missing completely the fact that yes, DeepSeek is cheap and fast. their API is no longer, by the way, cheap or fast, like their API does not work right now. They're claiming to have attacks, but deeps. KPI does not work for me. you have to go through, American folks, so many are including deeps syn in their open source, including Azure right now. So you can actually use deeps syn without the clutching of the pearls. Oh, it goes to China. That's it. That's done. However, most noobs and most grandmas will not. Run DeepSeek on their platforms. So the iOS app being number one definitely means that people are willing to give up their information to, DeepSeek, et cetera. And also, two vulnerabilities were found in their services, including one completely open. Clickhouse database completely open left in the open. So like people could have, access all of your chats. So if you care about your privacy, do not use the app. Just go somewhere else. many people are already implementing DeepSeek. I think we've covered DeepSeek enough folks, like comments, please give us comments as well, but I think we've covered. DeepSeq enough again, compared to the fact that we talked about DeepSeq for an hour last week. So if you want the technical details about the R1 models, listen to our show from last week. We covered end to end, including all of the open source stuff. but meanwhile, let's move forward in the open source. Because the next thing that's very important for us to chat about is a new model from Mistral. Mistral AI, the French company, are friends who gave us once the best open source model to fine tune Mistral 7b. and then Mixtral, and then a bunch of Pixtrals, and a bunch of Codestrals, they play with this, naming all the time. They brought us Mistral Small to 25. 01. It's a 24 billion parameter model. That has a very high score in MMU pro, 81 percent accuracy on MMU, but very high, score in MMU pro it's a model that can fit on a single 4090 or a distilled version of this runs on the Mac. so almost like quite possibly the perfect size. It's smaller than a 32B from Quen 2. 5, for example. and, it gives a very good result, compared to GPT 4. So almost a perfect model. It's not a reasoning model yet. and they mentioned this, I wanna say reasoning is gonna be everywhere. Arthur Mech, the CEO of Misra mentioned. It's not the reasoning yet. So Mr. Charles is also working on reasoning. we can assume at this point, I think folks, let's call this, every company is working on reasoning models in if not, if they're not about to release one already, they're working on one. So whether it's rack. It's going to have reasoning. Lama three was confirmed by Zach is going to have reason Lama four. Sorry. Mistral now confirmed again, have reasoning, Gemini already have Gemini thinking, and that's great. So pretty much reasoning all around. Like we predicted back in December, 2025 is the year of agents, the year of reasoning as well, and probably reasoning agents, Nistan, Mistral, Apache two license model. Finally, tell us what you think. And also, Wolfram, I want to hear from you because. I know you're excited about this one.

I think the social pressure works, because yes, we

were, When they released the newer CodeStraw model, I, I went online, okay, I can't even use this. It's going to take me over a day. And then I was just able to grab, do like a yarn extension of, the QuantEncoder. 32 and it worked up until 200 K. So the approach that they were taking as a company to start locking everything down and, reducing the interoperability just wasn't working because, people just stopped building with their models. And, I still use Mistral large to do a long context summarization. I just don't integrate it into. Anything else I'm doing. It's just for, it's just for me personally, but that it makes it, the licensing makes it very hard. So anyway, a lot of people gave them like myself, gave them a lot of smack for, turning the page into a used car, salesman portal, and, 2 license. We released it.

we're very happy with this.

So shout out to Mistral finally giving us back Apache two license model. The commitment to open source is palpable. Not only that, I want to highlight this one thing that we've, we also called out. The community called this out. Mistral did not like adding Quen to their evals for some reasons. There's reasons, I think. finally, we're getting comparisons to Quent 2. 5. I think maybe now it's because they're beating Quent 2. 5, at least, coming very close, but in smaller form factor, so it's beneficial to show this off. But also, I think that, folks, this has been, like, Chinese models that we've reported on Kuen on DeepSeek, everybody's waking up just now. We've been talking about DeepSeek for a year. We've been talking about Kuen for more than a year. Jun Yang is a co host of the show. He joins all the time. We know that Chinese open source absolutely destroys like everybody else until Llama comes out and that, holds for a few weeks. And then also then they also catch up. And this kind of unease with Chinese open source from folks, not even comparing to them. We're done with this. We're done with this. we know that if R1 comes out and is like the best model possible, you have to put up every model, doesn't matter where it comes from, in your evals, and to show that you're better. And so very happy that Mistral finally acknowledged this, and then they have Gwent 2. 5 in those performance comparisons. I found this very funny. one of the ways that Mistral eval was landed for Mistral small is human evaluated benchmarks. They didn't say which benchmark they only said we use the third party to evaluate, maybe artificial analysis, maybe Something else, and they basically posted a chart that says the categories are not like MMOU Pro, Human Evil, like the stuff we're used to. The category is Mistral is better, Mistral is slightly better, others slightly better, or others better. They basically get like a grid, so I found this chart hilarious because I tried to read this, not necessarily understood. So they compared Mistral, 24b, Mistral small, to Gemma 2, to Quen 2. 5, 32b, to Lama 3. 3, 70b. and GPT 4. 0. So you can see in the chart that Mistral, I don't know. This chart is super confusing to me, folks. I don't know. It says Mistral is better 0. 1, 0. 19 percent of the time against Lama 70 B, but also Lama is better 0. 2. It's a confusing chart. So I don't know what they tried to do with this, but we need better comparisons on numbers. it has function calling has a bunch of stuff for local inference. So that's good. Wolfram, what's your take on Mistral? You probably don't have time to download this one.

not yet, but definitely today because, I've

been waiting for this last year. There was a lot of releases, but Mistral, they were completely quiet and I really missed them because they are the only European player in the open source scene that actually matters. And, it's always great. I still see the eight by seven B in production in a lot of use cases. So having a new model that is better and as a good form factor size, that is very important and I'm really happy that it is Apache tool license as well. So I think. Especially in Europe, this will be a very big release and it does the European languages extremely well and most probably better than the Chinese models, which is a big factor here.

Yep, I think that it's very important for European purposes as well.

I agree with you multimodality as always, we're going to wait and hear from you, how this model performs and going to be very excited to see your evils as well. So folks definitely give Wolfram a follow Wolfram is our resident evaluate, person. He has released a bunch of benchmarks. We also compared some of the R1 stuff that you released last week. We added to the show. So this is MS trial. Let's talk about, the two replications of R one zero that come one from Berkeley and one from previous folks that came from, deep sec. I mentioned this briefly in the TLDR. also a shout out super quick to folks who just joined us. We're also on video. If you go to the top of the space on X and that, the tweet from WNB, we thank you for hosting us. our show is also on YouTube, but also on, on, WNBweave, account. So give that a follow as well while you're there. so back to our talk about open source, Berkeley released tiny zero. It's a reproduction of R1 0 for 30. And then Ragin, was released by folks, who also previously worked at. DeepSeq, and both of them tried to replicate this next thing where DeepSeq trained and gave us two models. There was R1 and R1. And the difference between R1 and R1. 0 is that R1. 0 is completely RL and completely, I think, synthetic. and, both of these replications tried in the open source to replicate this. There's also an attempt to replicate A fully R1 replication from Hug and Face that we're waiting on. so very interesting things for you to check out. If you want to see how, we get to this, aha reasoning moment. Berkeley TinyZero released also Weights Biases logs. complete logs from Weights Biases. and, those are fun to see always. So shout out. if you release Weights Biases logs, you know I'm going to shout you out. as always. Tulu from LMEI, for example, just released, they used to release Weights Biases logs. So they got to a point where with the chain of thought training, they were able to get to this aha moment where the model starts to reason, which is what the paper talked about and what like was very exciting. anything for us? No, let's skip. We have a bunch of other stuff to talk about. let's actually cover TULU. Nisten, let's look at what we have from TULU. TULU from LNEI 405B. Tulu, which is a fine tune, somebody confirmed in the comments from Lama, 405b, was released. What do we know about Tulu at this point already? Let's take a look.

apparently it's beating DeepSeek v3 on the benchmarks, which

is, so again, for people that, because there's R10, R1, v3, All of the DeepSeq models started from, a mixture of experts, architecture, and that just means that the model can, sit on a lot of RAM, but it's only using, a small part of it. But the architecture between, v2, so if you heard of DeepSeq Coder version 2, or DeepSeq 2. 5, or DeepSeq v3, which we've been covering for the last, year or so. Those are all this new, much more evolutionary architecture, which like keeps a pool of expert weights and then picks in places from that pool layer per layer. And this is what we'd consider like a second gen architecture. Whereas the Mistral 8x7b was like a first or the first grok that, that the Elon released, which was like the gigantic file one, that's more of a first generation architecture because that just has. For each layer, only a small, a small pool that they only pick two from. So now with the newer model, the V3, they built up. So the newer DeepSeq R1 is just a, a reinforcement learning fine tune of basically. DeepSeq v3. So that's the base and that's also the, yeah, so that's the base which all these models are built on and that's what 2luv45b is comparing it to. So 2luv45b also managed to Beat the base of this now some people are saying that this is

deep secret three base.

That's what you mean, right?

yeah the v3 base, but using some of the same techniques

and the reason that we really like tulu is because completely open and, they show, everything that they do. And this benefits massively the community as well. And it also brings them back. those connections end up making them better at making models too.

Yeah, so shout out to LNEI for like open sourcing, pretty

much everything, including checkpoints, usually including everything. the one highlight for me is that this RLVR training method, they call reinforcement learning with verifiable rewards method. This seems to be like holding on and very well, even in larger sizes. so shout out to Tulu about this. Anything else that we want to add about this, or we're skipping towards vision models because we have a bunch of stuff to talk about vision models as well.

it's just going to be a bit expensive to run, but,

yeah,

I'm looking forward to doing some, data gen.

so let's talk about, open source in the vision category as well,

because like we usually split, but I doubt that there's a need to split. Sometimes there's a very stark comparison. I will show that, The thing that we have now to cover in the open source is DeepSeek released Janus Pro in the beginning of this week. Janus is a very interesting model. Janus is a state of the art model that understands images and generates images. If you guys remember a long time ago, Greg Brockman posted this, image of a person writing on a blackboard and, the writings were, like, looking like a chalkboard. And then they basically said that GPT 4 can generate images natively, not via DALY integration. And this never came. we never saw any of this ever again, but it looked like super, super cool. this model is basically that. This model understands and generates images together. So it's like a combined architecture. and compared to models that, it's not the first model that does this, but compared to other models that do this, this one does this very well. and so we can. We can take a look, the architecture uses SigLib and then, uses LamaGen for generation. it beats, I think it's a 7 billion parameter model and it beats, yeah, it's, there's two models, 1. 5B and 7B. it beats SDXL. At least on the low resolution images it generates. So this model that, is MIT licensed, by the way, shout out to DeepSeek. this model was trained on like lava, et cetera. It beats like DALI 3 and lava. You guys remember lava, the llama with vision. So this tiny model with 7 billion parameter beats DALI 3 on images. And beats lava and image understanding, which is to me is like absolutely crazy. I don't know why people are not talking more about this. I don't know what use cases you would use this for necessarily, like, why wouldn't you use like flux, for example, for image generation? but yeah, that's pretty cool.

It would be interesting if you could input an image, have it

changed by the AI and output as an image. That could be very cool. Image manipulation through the AI, through natural language.

Yeah.

So that's like what Sora does, right? You upload the video in Sora and you like, tell it like, Hey, understand what's going on. I believe that, somehow GPT 4 when only it knows, like it can do this. Like we play with GPT 4. We uploaded a copy bars and whatever. and different raccoons. I remember the raccoon on like a skateboard was one example that you can upload and it understands the image and it generates. You can even transfer the understandings from the understanding into the generation. So this is Janus. Dipsy kept releasing a bunch of stuff. This is Janus Pro. Janus 1. 5b was the previous one. Janus Pro is the existing one. we also have NVIDIA Eagle 2. NVIDIA is, so DeepSeek, like one of the highlights of this is like the combined understanding, but compared to a specified model for, vision, they're not competitive. Gemspro is not competitive to something like Eagle or QuenVL. So we had like just a array of multimodal models or vision models, coming out. And NVIDIA finally brought back Eagle 2, which they released last week and then yanked for some reason, and now it's back. Maybe they, they confused something. the model is there. papers there, and the demo, you can play with the demo. It's a high resolution, multimodal understanding. like it understands 4k videos. This is, I think, one of the first, this is the highlight of this model, understanding 4k resolution video processing, for OCR and object detection in the video. they have a multi expert architecture, long context support, 16k sequences for video analysis. Very interesting, released in three sizes, 1b, 2b, and 9b, including the paper, and they claim they match 70 billion parameter competitors with this 9b. this is a shout out from the paper. Anything else? Oh, by the way, we have LDJ here. LDJ, welcome. Welcome to the stage, man. have you looked at Eagle at all? What'd you think about Eagle?

Eagle?

I can hardly remember what was So I guess I don't really have much to say. Yeah,

I think that the least, like the highest unique

thing is like the very high quality processing videos that they do. and, some efficiency in training, et cetera. let's talk about,

it's still a coin model, by the way.

is a coin fine tune.

Yeah, yeah, it's still a client, a client fine tube.

Interesting.

I mean, they try

to name the architecture themselves, it's like an Eagle 2, but

then you look down in the configs and it's, oh, it's a client, it's a client 2. 5, 7b.

Yeah.

no, good. Nistan's, comments is very interesting. model tree now on hugging face, you can go and see the model tree, like which model was distilled from which version, and so this is a, on top of quen 2. 57 B. So maybe the vision encoder is there's listen, cause I don't think it's a, quen vision.

the vision is, is a Google siglet.

So it's, yeah, it's just another, VLM, like many of them, out there. I mean, it's probably pretty good. It doesn't seem like there's anything wrong with it, but it's not all that different from Lava and all the other ones Even some of the ones that Alan has made.

Yeah, that's true.

They compared themselves to QuanVL. Two, not 2. 5, QuenVL2, which came out a while ago, right? so they compared themselves in 9 billion parameter model compared to QuenVL2, 7b and also 70b. It's great. Shout out to NVIDIA. they have the Visionist, Siglip from Google, and ConvNet as well. but a more interesting model, I think, came out, which is, I think, state of the art. Quen 2. 5 VL came out. And, Quen 2 VL was state of the art, and just a second before the Chinese New Year, our friends at Quen gave us Quen 2. 5 VL, which they call the flagship version of Quen. it's really hard to now get to a point where we know what's state of the art in, vision models. this feels state of the art ish. and there's a few highlights there that I want to chat about specifically. One of them, Nisten, I think you called out. it's good for agentic purposes. You want to chat about this? I think you called this out.

yeah, I think it was VB or someone that, demoed this.

And, I'm in the process of integrating their one million thing. I wasn't doing anything, agentic, but it was basically like, an open source operator. You're going to build it with this. I think it's going to be, this model is going to be a much bigger deal than what it seems right now. we had some other ones from, from GLM, but they're not as well known and people haven't built as much with them. Also, sometimes there were difficulties with them on running them on llama CPP.

Yeah.

The quantized version of those did not work. And they even put up a notification on GitHub. quantization versions just completely fail the computer use angle.

Yeah, you're probably going to need the large one, the 72b

to actually have it reliably, Do work, but, yeah, I think in the coming days or weeks, this model is going to be a much bigger deal if you want to have your own operator, because it seems like this is the best one right now, mainly because you don't have to have two separate models, one to recognize what's on the screen and also decide, the same model can do both. And I'll put the code.

for me in QuanVL 2.

5, QuanVL 2 was like already great model. from you probably like already, you also looked at some of the state of the art, updates. the highlights for me is like object grounding. They now can show you like which objects are where I think that's significantly improved, including like specific places and labels and positions. They output them in like JSON stuff. And they're going to send what, which, what. Each one is, they also have a bunch of world information in there. They showcase the pyramids, et cetera. And they show like different types of birds understanding. But, the highlights and they have the standard, like benchmarks, they compared themselves to Gemini two flash, by the way, which is a very good vision model, very good vision model. And they, they claim. Higher stats than Gemini 2 Flash with this model. Flash is probably going to be significantly faster and cheaper though. Because this is a big model. 72 billion parameter, Flash is probably going to run faster. But they, like on, on, OCR bench, they're beating, Flash and they're like on, et cetera. However, the most important stats for this model to me, is video understanding. This model has temporal video understanding. So not only sequence by sequence images that they just sent to the model, okay, this is video. No, this model understands temporality in videos, which we talk about all the time, which is very important for actual video understanding. So this understands like high quality video, as well, similar to Eagle. and this is important also for agents as well. So this model is like we said, agentic, This model is going to be like a good plugin to something like a open, operator or like browser use. probably because not only does it like, it's already trained in a genting, a visual agent purposes. It only, you also understand video, all visual agents right now, they're operating based on screenshots only the operator from open AI. We're going to talk about the operating on screenshots only this one. understands video for up to an hour. So if that kind of turns into a real time understanding soon, this agents are just going to be better because you can see progression.

And Quinn announced, or at least there were

talks about a Quinn agent coming out. So I think that is an integral part for that.

So

you've had that in the repository for quite some

time and, and people, give a shit for, Promoting the CCP or whatever. I think it should be a bit more grateful for people to just open sourcing work to Get stuff done because that just benefits everyone and the reason that people are using this model is because all agents suck But these tools are just barely adequate to begin with so just using the best one You're just like you're using the best tool for the job, which happens to be this open source one that like fulfills the requirements of you having owning the information instead of renting it. So if you have that as a business requirement that your business has to own the information, you're going to use the Chinese model, bro, because everything else sucks. And even the Chinese model is just barely adequate enough to do the job. Yep. And, yeah. So just to let you know, if, if you don't want people to use Chinese models, open source, better ones.

I will say though, Nistan is going hard against some imaginary people.

I don't remember too many people for Thursday. Complaining about,

but

yeah, generally once it breaks through the, the barrier

of normies, then CCP shilling, commences or sorry, the accusation of CCP shilling commences. anything else that we want to add about Quen folks, Quen releases the greatest like vision models, go get those models, go play with them. You can play with them on the Quen chat app that they have the Quen chat. And the thing that I wanted to talk about the, so it's chat. quenvml. ai, I think?

is something in them because they've put a limit on, the input.

It's only 10, 000 characters. So that actually is very limiting. So you can't even, paste a large amount of code. So maybe we could. Ask them to ease that off

a little bit.

So chat. quenlm. ai And the one thing I want to highlight in this chat. quenlm. ai is that they now have image generation in there and video generation in there. And The videos look good. They look like Hauliok level, Sora level videos. They didn't talk about The video at all and now you can generate videos with quen just through the chat though. So we started talking about quen which is like a state of the art OSS vision model and We then said that to try this you can now test it from the drop down in quen and then also while you try this, there's other two models there that are like, for some reason hidden, there for video generation as well. I think that this is it on Envision Video. That's, as far as we have, that's what we have to cover there in Envision Video.

Also, they released two Apache license, one

million context models from Quan. Yeah. Which, we're trying to have up on GitHubgg for people to use. They were very fast. And, and very good. So expect some, some fine tunes on that. I think those kind of deserve their own thing because people don't really realize why they're useful, but that will be communicated as more projects.

So we covered this a little bit in last week, but

let's talk about this as well. When, before this time, Quen had a Quenturbo, which is 1 million context window that they support from their portal. But now they also release the models as well. This was last week we covered this. I think it was like literally on Thursday that they came out.

Apache licensed.

You can now fine tune Apache licensed 1 million context model.

I think that's incredible.

It's a big deal. You remember we talked about the difference between like 4k and 8k and we're like, yeah, 8k context. 1 million is a big deal. Absolutely. All right, folks, we've been at this for more than an hour. Let's do a little reset on the space. If you're joining now, you are on ThursdayEye, which is a weekly AI news show that we bring to you for the past almost two years now. We cover everything. Sometimes we cover life breaking news. For example, last week we talked about Operator from OpenAI, which we're going to mention today as well. Live as they launched it, we watched it together with you and then we used it and tested it out. so my name is Alex Volkov. I'm here, as an AI evangelist from Weights Biases. Weights Biases also co hosts the show with us. So if you're in the Twitter space with us, this is the square button with the golden check mark on the X, please give it a follow as well. We also, are on video in, also hosted by wisdom by says we've this time. so if you want to not only listen to us, but also see the stuff that we're showing sometimes for like vision models, et cetera, it's nicer to just see. So feel free to go and check out that we're live streaming everywhere on YouTube, on LinkedIn. There's five people on LinkedIn that always join and this is their preferred medium, which is great. Hey, LinkedIn folks. together with me, I have my cohost here, Wolfram. RavenWolf, all the way from Germany, Nisten all the way from cold Toronto in Canada, and LDJ in a somewhere in the Eastern Southern part of the United States Also joining us as well. We usually do a reset in the middle of the show and as always the show is not sponsored by sponsors So everything we talk about like when we're not shilling CCP. They're not paying us the only Sponsor ish that we have is Weights and Biases. And so usually in the middle of the show, we do a short corner, sometimes that's called this week's buzz, where I give my cohost a five minute break. And I talk about, the. So let's do this super quick right now, and then we'll be back and talk about what else happened in the world of AI, including agents, big companies, API, and including, voice and audio. There's a bunch of updates in voice and audio. So let's do this week's buzz.

Hello.

Hello.

Welcome back. This is this week's buzz, the corner in Thursday. I talk about everything that happens in the world of. which is quite a lot. So this week I was waiting for breaking news to actually bring some more examples of the stuff that we worked on, but breaking news did not come. Breaking news from over there did not come. They're probably waiting on it. So if there's going to be some breaking news, we're going to talk to you about them. However, this week, I want to tell you that, just a reminder last week, we. mentioned a state of the art result that came from Weights Biases. in Weights Biases, we have two main products that we do. We have the ML observability framework, ML experimentation framework called Weights Biases models that most of machine learning engineers in the world use, including companies like OpenAI, including companies like Microsoft, Mistral, I believe as well, like a bunch of companies. We also have a new tool for, the AI engineers, It's called Weave for LLM observability and, evaluation. And I talk about this quite a lot. So if you follow the show, you probably know all about, weave and evaluation, in kind of order to. Promote it better. We are becoming experts in the world of evals specifically. So if you want to test one of those models that we talk about and drop this in the replacement for your production application, and you want to test out the differences, whether or not, your production application still works with the new R1, for example, how would you know? The only way to know is evaluation. So we release content continuously about evaluation. We have a course for free. and last week we actually are dogfooding our own. Products to know how they do better, and in the process of duck footing, we achieved state of the art results in suite bench verified, which is a very hard benchmark for software engineering tasks. we are now number one in the world on that benchmark with 60. 5. 1%. And, Sean Lewis, who is the CTO for Weights Biases. He built this with a agent framework he called WMB Programmer that we're likely to open source at some point. and he went on Latent Space Podcast with Sweets and Alessio, our friends as well, to talk about how he did this. And so one of the things that happened with this week's buzz is that Sean's interview with them is very interesting and you guys should definitely tune into that. I'll add this in the show notes as well. Other two things that I want to shout out is that there's going to be a few events that are coming up and I love talking about events in Thursday night because then people who listen to the show, they come to those events. When they come to those events, they say hi to me and they give me a high five. And I always love this. One of the main events of the year for AI engineers is AI dot engineer also run by SWIX. And I've been at pretty much every event since then. we're going to sponsor that one. I'm going to be there together with all my colleagues. If you're coming to AI engineer summit in New York in January. Sorry, February 20th, 21 and 22. we're going to be there. Come by our booth, say hi to us, check out some cool stuff. Probably the chef is going to make an appearance. There's going to be a bunch of like very cool things from us. Probably talk, from us about evaluation. So definitely if you're an engineer, we're going to see you there. I'm going to probably do a Thursday. I episode live from New York. not Saturday night, Thursday morning. and it's going to be live from AI engineer as well. Probably going to do some interviews. So if you're into that, we'll see you there soon. And a few days afterwards, I'm flying to very cold Canada to meet my friend, Nistan, for the first time after two years running the show together, because we're coming to Tinkerers. In Toronto, which is becoming bigger than New York. the Tinker diaspora in Toronto is huge. their upcoming event has 150 seats and more than 700 people subscribe to that event. So it's like way over subscribed. And so we're coming to Toronto to university of Toronto to do some workshops about evaluation. So if you're listening to this and you're from Toronto, or if you want to. I have an excuse to fly down to Toronto in the middle of February for some reason. I don't know. You got tired of the palm trees and you want to come and learn about evaluation. We're going to run two workshops, the same thing we did in Seattle, in Toronto. tune in for those. We're going to add. Links probably next week because we don't have the links quite up yet, but it's coming. So AI engineer in February, and then two workshops in Toronto in February as well. I think those are the highlights we have a bunch of other features that we're going to launch. I'm going to talk to you about features, but the main thing is folks, if you're not doing evaluations on your AI applications in production, you're not. Able to confidently drop in our one, for example, and just see if your application is better. And this is what we're here for. If you're interested in anything more than this, just click on the weights and biases link, in the show notes, or just go to 1b. me slash weave and then, get started with a few lines of code. It's very simple. 1b. me slash weave. This is our evaluation and observability framework that sponsors the show and makes it happen. So if you're enjoying any parts of the show. and with this, folks, this was this week's buzz. Let's get back to our regular programming.

All righty, back to our regular programming. And, we now have the next section that we talk about. It's a new section. It's called agents. We haven't talked about agents. I don't have an opener for agents, but folks, 2025 is a year of agents. It's clear. We've talked already with Joao from Crew. ai. We've talked with, Graham Newbig from All Hands and OpenDevon, previously OpenDevon, and now OpenHands. And, last week we talked about Operator that came out as a first agent from OpenAI. it came out a week ago. It's only for the pro tier. So you have to shell out 200 for the privilege to use this. And, I think the corner for agents is going to be a very interesting one, maybe growing and growing on Thursday night. And so we should talk about some stuff that happened in the world of agents this week. So first of all, in the, world of open source agents, Blocks, the company that's run by, Jack Dorsey, previously from X and Blue Sky, et cetera, they released an open source agent called Goose. And you can run this, I think on macOS, not sure if on Windows as well. and this is a, open source agent framework that's running agent that control your computer. Very similar to Open Interpreter, by the way, very similar to like how Open Interpreter works. the thing that they have is a very nice package. I don't know, LDJ WolframDivya had a chance to, already try out Goose or folks in the audience as well. Feel free to let us know how Goose worked for you. I ran this, on my machine. And, with some success, I would say some amount of success, but, not great. for some reason, so I'll give you one example of not great. it runs on your machine. So it like uses your, keyboard and your apps, et cetera. and it uses shortcuts. So I asked it to go and fetch me some information from WhatsApp app. It went there, it opened the WhatsApp app and then it tried to delete one of the chats. And so I literally got like a WhatsApp app. It's Hey, are you sure you want to delete the chat with this person? I was like, no, I'm really, that's exactly what I don't want. so yeah, so agents are still, let's say local agents are still, they can run into issues. The highlight there for me is the extension feature. You can build extensions. and they're built with MCP and tropics, computer protocol, for connecting agents, one to another. So they already have extensions like, let me see which extension they have. They have a memory extension. They have a git extension that can read git statuses from directors you're in, they have a, obviously a computer control extension that does web scraping and file and the automations. And knowledge graph memory, and then they have a web search extension as well, and a very interesting Figma extension. so if you're into trying out different like agents, this is a very interesting, fairly neatly packaged open source from blocks, which I'm assuming the better the under, like like Nissan said, the better the underlying models will happen, like the QuanVL agent, for example, the better these type of like agent runners are going to be, and the more important they're going to be.

Which one did you use when you tried to

delete your chat on WhatsApp?

I think that Lama tried to delete my chat.

I tried DeepSeek and for some reason DeepSeek did not work. So I tried DeepSeek on Grok and then DeepSeek on, on OpenRouter. For some reason, I think because of the way reasoning tokens are shoved in there, that just didn't work with me on, on Goose. I mean, we can try goose right now if we, I can show you guys, for those who are watching, if you want to switch. So you can switch between different models, GPD 4. 0 Mini, for example. once 0. 3 Mini comes out, then you can do that. and then you can choose, let's choose a different model. let's do Cloud. Cloud is supposed to be good, right? Entropic, Cloud, Sonnet, Latest. Let's try Cloud, Sonnet, Latest. Okay, so we'll switch to Cloud. and then let's do, Sonnet, Latest, Open Chrome and go to x. com.

For computer use, cloud computer use, they made a specific

model with a specific name that is different from the normals on it. Yes. If you could use that here, that would be interesting.

So it's not that one.

I don't know if that one is available necessarily. I think it is. but I don't know, do you guys remember if that is available via API specifically, the model, the computer use model?

must be because it's being used by the API of that thing.

So we should just get the name of the model and try if we can insert it here. It could be an interesting experiment.

Yep.

so yeah, we should probably just add this to the API. so I just asked it to go to Chrome and open and it opened Chrome and went to X. com and it did this With the script with Apple script So it does like a bunch of Apple script stuff, but I would be very cautious to telling people to use this I didn't yet see the benefit in this. My benchmark for running agents is this I have a very standard, like running for the past year and a half preparation flow for Thursday. I follow all the news on X, like most of us do, and I add them to a bookmarks folder on X that's called Thursday bookmarks. And then the Wednesday before, a couple of days before I start collecting my bookmarks into the notes that you guys sometimes see on the screen. I'm looking for an agent to automate that part. I have the bookmarks. I know the process to go to x. com, go to the bookmarks folder, get them, et cetera, and then do some connection to get the link inside there, do a summary, et cetera. none of the agents that I'm able to run is able to do this for me. And this is my benchmark for an agent that can help. That's not to mean to replace me because I'm still the person who decides which news ends up on the show. It's just like the literal slog of going to X. com, going to bookmarks, opening this like a new tab, doing a summary, putting it in my notes, putting it in the right section and notes, right? We have the open section. We have the, the company's APIs. That thing is my benchmark for whether agents are here or not. none of the agents, like the goose cannot even come close to do this. operator. From the other side is another, agent that, OpenAI just launched. And, for folks who do pay, 200 bucks for OpenAI, I have built an electron wrapper for operator, which I will hopefully be able to show you in a second. I have this operator. App that I built. So basically it wraps the operator website, but it gives you a native notification on macOS. So if you are paying 200 bucks and you're playing with operator, you can download this on my GitHub. I'll add the show notes and I've been running operator for the past week. And I tried the same exact task. I've tried to have it go to bookmarks, get each link, get a summary. And, it failed with so much fanfare. It's better than Goose, but it failed with so much fanfare. It only extracted, three links. didn't really work for me. which is unfortunate. but it's trying. It's working way better than Goose. And the beautiful thing about Operator is you can run multiples of them in the background. you can ask Operator for something like a, tweet about our Live show. And you can tell operator to do something like this, and then it will know how to do some stuff. So it's very interesting. Have you folks tried any of the agent runner thingies, that we just covered? Goose or operator folks here with me on stage or folks in the audience?

Not yet, but it should be working on windows.

If you use the command line version in VSL, for example.

You mean the Goose one?

Yeah. the benefit there is that Goose is open source and runs. So even the Euro operator is not available in Europe. there was also Another company that also claimed state of the art operator like things. And I tried that and it like absolutely completely failed. so that also didn't work for me as well. So we're waiting for agents. 2025 is the year of agents. But right now, we're still waiting for some of those results.

Hrunner?

not Hrunner.

another company.

it Hive or something?

There was a French one too.

Yeah.

I think also French. but yeah, we need better. I didn't even add this to my notes. I think this time. so yeah, another agent that runs, I will add this to the show notes. there's also an open source version that tries to get to operator level. It's called browser use. So folks, if you're interested in taking a look browser use, I public on, it's very similar to open interpreter, actually, browser use is public on, and they also have computer use from browser use as well. So if you don't want to pay open and get their state of the art models, their state of the art, open operator model understands a few things incredibly well, like it can join a space. For example, it's really. not completing my task yet, but it's probably the closest one to complete my task. It was able to go to the bookmarks folder. It was able to pull out. I can show you guys some examples. it was able to pull out the tweets to put them in a new entry You can see here on the screen, you can see how many tabs are opened. So it's really funny. Like they have a Chrome browser running in there and they opened a bunch of tabs. it was not just very good at getting summarization, but it did. Open those tabs, take those things and put them in a new rentry, example for me, including getting the links out of those. So pretty cool. It took I do it faster right now, but obviously that's not a good measure of agents. If I do something faster,

what's cool about agents.

When you have a good agent, you can send it to the open AI providers. yeah. So they provide us like Claude or open AI and so on. And then the agent can use those and pick the best one for the use case. You have, that would be interesting on a metal level in that way.

So a very interesting thing is that if you go and you

ask operator, go to operator. chat. npt. com, to do an inception, say, Hey, agent, launch an agent. It goes there and the error they have is it's very funny. they have an error that says, Nope, not even, don't even try. Nice try, but no, just, no, This is literally the kind of site unavailable. Nice try, but no, they know that people will try this. so agents, corner, I think that this is pretty much, it will add another, the runner that we had, it also failed for me. It also didn't work.

you know what's messed up now that, we are going

to need the AI calling on humans. To just solve the CAPTCHA, so that I could Go by, we spent all this time making captures so that we know it's a human, using the tool and now we have to reverse that.

that's really funny.

Um, you can break, you can jailbreak operator to solve captures for you. It absolutely can do that, but it doesn't want to. And Nista, I think you're referring to the fact that, you're getting a notification that says, go and solve this capture for me, human, and then I'll continue. It's really, it really is funny. And so I think the highlight in Operator is that when you start different tasks, you have all these, integrations, OpenTable, StubHub, they don't have direct API integrations into those. Operator only works on pixel level, but they have those integrations. Assumably because they work with these companies to say, Hey, if our agent runs, then at least don't show, don't show captures for our agent because it's like really annoying. So going to Reddit and go to Reddit, let's say, let's try this. go to Reddit and show me the top two posts. I believe that it won't go to Reddit. Like I believe that Reddit already blocks something, the operator. so yeah, those examples of asking the human to. and actually, yep, there you go. So Reddit blocks the operator stuff. they want you to log in and then operators actually very good operators sends you notifications as, Hey, log into the browser. And so you can actually do this in the operator. their Chrome web runs on like a cloud somewhere you can store your. credentials. And so if you store your credentials, then it should work. It looks like you, they don't have a custom user agent as well. So you can take over and you can log in and then you're able to use the services that you want. I think the operator right now is definitely state of the art. It's the closest that came to being able to do my task. It doesn't do it well. great promise, but still not there, but playing around with this, it's. Probably worth the 200 bucks for the folks like us who want to try different things. I don't know if it's worth for a regular human to pay 200 bucks for agents.

Look, as someone who is always very critical of agents,

it's not because I don't think they're Real or they're not going to happen. It's just that they're not really doing anything automated and I just think it's going to be a lot harder, but remember when Open ai gt4 came out with function calling and we were debating whether this is a game changer And then they released all the plugins and the plugins did not have any product market fit so then that kind of died down because it wasn't really Helping stuff automate and now every model is required to have function calling ability, otherwise, it's Useless. just not as useful. I think this will become a much bigger deal. I just think that the difficulty to actually implement an assembly line that consistently works is a lot higher than What people are imagining or just getting a dopamine rush about it's, again, the analogy here is you can watch someone cook pasta at home and then you want to build like a pasta maker factory. no, that, that looks like an oil refinery from the outside. That's a very complex, complex thing. So it will take some time before these things. Actually get to a point and it's going to be a lot more work because, there's just an endless amount of edge cases to get through.

I see Simon as well with us here in the audience.

feel free to come up and tell us about your experience with operator. right now, as you were talking, I asked operator to go and play paint by numbers, color by numbers, the children games. I found it very endearing. right now the folks who are seeing us on video can see that it loads the game the main issues is it doesn't support web GL the browser the operator runs doesn't support web GL and most of these games for some reason web GL you can use Canvas, whatever. let's add Simon Wilson, our friend, Simon, will join us. And I want to hear your experience with agents generally, but also operator. If you have a chance to play with

yeah, I've been unimpressed so far.

the best results I had from it is I asked it to figure out. which BART station was closest to a specific address. And it fired up Google Maps and it measured the driving distances and gave me the correct answer. And what was great about that is that the only time I've used Operator where it hasn't just constantly stopped to ask me for permission to keep on going. if you give it a read only task, it can actually go and churn away, and it ran for about two and a half minutes. But I was doing other stuff, and then I got a notification that it was finished. So that was quite good. the other things I've played with it, it's just, because it just has to keep on asking you for permission to do this and do that and so forth. It feels like a tech demo, it's okay, I see what they're trying to do here. This is not actually saving me any time yet.

Yep.

it's an interesting thought.

It's definitely not worth paying 200 a month for. but yeah, it's been fun to play around with, but I don't know. I remain, skeptical about the whole idea of what operators trying to do because of the security concerns around it. they've got, they can detect, they actually, I've got a screenshot where Somebody did send it to a meme generator and on the meme generator page was a little bit of text that looked like a prompt injection and operator just stopped and it showed a big thing at the top saying, Hey, there's something on this that looks like instructions. Are you sure you want me to continue? Which, I guess at that, but that particular time that the filter kicks in and works. But I don't know if I'd trust most human beings to understand what the hell that even means. So yeah, I remain skeptical about it for the moment.

what's your kind of go to task to test this out for that

you would say, okay, if this passes this task, like agents are helpful.

So I haven't really, I haven't done the testing yet, but one that

I've been holding in my head is, I want to buy a, battery for my house, right? like one of those big backup batteries, which you plug in and then if there's a power outage, you can plug other stuff into it. And for that, you if you want to do comparison shopping for that, you have to go to 15 different websites of different manufacturers and gather information from all of those. And so something I've been trying, I tried this with deep research and I played with chat GPT search in the past, like trying to get a really comprehensive view of all of the things in this particular category of product, and then give me a comparison table and help me understand which one gives you the most kilowatt hours or whatever it is that feels to me like that's going to be the moment for me. It's when I can. Send out one of these research agents. I like research agents because they don't need access to my bank account. they feel like a much more sort of safe application of this. But when I can send them out and get such a comprehensive overview that I don't feel like now I've got to go and do the work as well myself just to make sure, that's going to be big. And that, I think we might actually get there this year. I feel like the research agents thing, especially the Gemini Deep Research one has been pretty impressive so far. so that I'm more optimistic about.

So that's a very interesting kind of like a comparison as well, because

sometimes for stuff like that, you may be better served with a search API integrated with like a deep seek, for example, we have down later in the show, we're going to talk about, perplexity adding, deep seeker one, and then X also integrating something like deep research as well. So it's very interesting, would you need an agent that runs and clicks on buttons in the browser to do that? Or will this be better served with an LLM with some API access, but it's also like an agentic to an extent.

exactly.

like Google deep research is just, it's using the Google search API, the internal one directly, which is what I think you need for this kind of thing. the browser automation aspect is. Much less important for that sort of thing. Although I say that, I think at some point with research, you want vision involved, you want to be able to look at, and that's the point where that's like half of the value out of things like operator is that they can do compute, they can do the vision things against a screenshot page. So it might be that you want a combination of that. You want the search API, but also the ability to load up actual web pages and actual browser and explore it that way as well. That combination would be. Really powerful. Isn't it weird that OpenAI haven't let O1 or O1 Pro access, access their search tool yet, or code interpreter?

for search tool.

Yes, yes, yes. I heard the trick today. You can go and enable web search. In a 4. 0, context and then switch to a one and then it has access. I think even though one pro has access to search like that, I saw this trick and I was like, I wanted to mention this, but yeah, I think it's weird. But there's a trick to do this. LDJ, go ahead. You have your hand raised up as well.

Yeah, I was just going to mention with operator.

I think it's worth noting. They did actually say in the next few months, they'll make it available to plus users. So 20 per month users. And I think also people should keep in mind that the computer use paradigm. It is something that. In a lot of ways, is functionally verifiable, which means that it can have a lot of implications. And, the 0103 paradigm in the future where they're integrating these types of tasks into the reinforcement learning. And I think we will see over this year in 2025, a lot of advancement similar to the 01 paradigm and specifically computer use capabilities.

Yeah, I will just say that throughout all of this time I

asked my operator to go and play paint my numbers and so far it found like five games, all of the WebGL, WebGL is not supported in that context. So it wasn't able to actually get to a game, but it keeps trying. It prompted me to ask me some questions, et cetera, but it keeps trying. So it's very interesting, operators model. They call Kua computer use agent. It runs longer times than usual, like other agents that I've seen. while absolutely I'm Simon with you, it's not there. it feels like at least out of the ones that I've tried, at least the one that can run for the longest in computer use.

That's interesting.

I mean, O1 Crow is the only model that solved the New York Times Connections quiz for me. New York Times Connections is such a good test of the reasoning agents. It's like the perfect, and O1 Crow will turn away for four minutes. But it has got that quiz right almost every time I've tried it, which is quite impressive.

So we're not in the realm yet of also reasoning agents as well.

We have the reasoning parts all in pro, does reasoning with steroids, we're going to get all three mini at some point I was hoping for today breaking news, but we didn't get it. and then we have agents that are doing it step by step also like chain of thought things, but they're not necessarily combined together. but I think once we get there. For example, like reasoning agents that we did state of the art on SweetBench, that was a reasoning agent and that seems to work well.

It's also worth mentioning that, the reasoning stuff, if

you're worried about prompt injection. There was a paper just the other day where the reasoning agents do much better against prompt injection attacks because they can reason through them. it's not, they're not infallible, which is the problem, because if somebody's trying to hack you. Getting catching 99 percent of the attempts isn't going to save you, but it's still like for the reasoning stuff, I think is going to make the browser automation things a whole lot more reliable.

Yep.

Even though probably slower as well, like even this one operator is like not reasoning and it's very slow. It's you can see it and it's Oh, I wish I'm clicking faster. folks, we're moving to our last.

I think there is a whole market for a security software

to detect prompt injections and malicious code and stuff like that.

No, I hate, everyone's trying to build that.

I hate it because that will only give you a statistical chance. It'll say, Oh, there's a 99 percent chance that this isn't a prompt injection. And that's where you get hacked because you've got adversarial attackers who will come up with that. One in 100 attack that gets through your security filters. So I feel like those are all a bit of a red Herring.

Yeah, maybe, but no protection is better.

Yep.

I will say just to, pinpoint on this, Simon, I think you mentioned, OpenAI, when they launched operator, we didn't cover operator on the show. We actually like operator launched as we, and we didn't have access to it. When they launched it, they said they have three levels malicious intent. The user can be misaligned. So the user can ask operator to do some stuff that it doesn't want to. So it will say no to the user. The website can be misaligned, which is like Simon, what are you talking about? Prompted injection from the website context that they have built in some stuff that says operator, a user, the website is trying to fuck with you. So don't go take a look. And then also just the. What was the third one user website and, Oh, the LLM itself can be misaligned. So the elements of the operator agent can like do hallucinations and do some stuff that you don't want to. So they have now to deal with all these three levels. so it's going to be interesting how this unfolds. we're moving forward to our big companies and the API is generally a conversation, and, we also have audio stuff, folks. So we have two, two new, two other, yeah, go ahead Simon.

I dialed in early.

Did you talk about Mistral small three yet?

Yeah, we did in the open source section.

Yeah, it's new and breaking news from us in the open source. we're moving now to like the latest corners of big companies and APIs In the big companies, the API is that deep six API is down. So it's really first of all, I cannot go to the deep six platform website to get my API key anymore. I think that they close it off. To non Chinese people, I believe I know registrations are out so you can only register with a Chinese, phone number and, deep sick API through open router that I still can access, is very slow. they claim to be under attack. I don't know if folks can access it as well, I just wanted to let you know.

they're going to have to change their policy because

they were boasting, about two weeks ago they were boasting they don't have rate limits on their API. And I saw people who were doing 500 parallel requests against the API and it was working just fine. I think that's probably over. I think they might have rate limits from now on.

Most likely.

And then many people are trying to get, reasoning traces out of deep seek because OpenAI doesn't provide them. so that's going to be very interesting.

The official API, I just did a test right now.

It's working, but it's not working all the time. So if I repeat it a couple of times, either it's a rate limit or just too much.

So rate limits and also like slower.

I noticed an open router shows you throughput. So you can see all of these, APIs, you can see throughput. Deep six was very high. Like deep six official API was fast. And it's not fast anymore. it's like when it does work, it's not fast anymore. I want to cover, super quickly 2. 5 max, because launched another model, that's not open source. But, it's a great model that they gave us like just a second before the Chinese New Year. And, that model competes with, DeepSeq v3, which is another DeepSeq model it's funny that this week people are discovering DeepSeq, and they still only know about DeepSeq like the reasoning model. They don't know that DeepSeq already released a great model called v3. They don't know about like other DeepSeq ones. Quen 2. 5 max is the kind of the flagship model from Quen with impressive numbers. Like they compare themselves, let's say to the previous Quen 2. 5b, it's not a model that's open source. So when we talk about Quen, it's very important to differentiate. This is not a model that they released. publicly, this is an API only offering from 2. 5 max, does 69 percent on MMU Pro, which is a very high number, compared to let's say DeepSeek does 64, for example, Llama 405B does 61. So like very good model, Human Evil at 73, also state of the art among those models. they also compared themselves to GPT 4. 0 and, and other things on Arena Hard. they get the highest score as well. very impressive model. this is a mixture of experts model that runs on, on their servers trained with, massive data. and, It requires an API key for access. I think pricing wise, it's also fairly competitive as well. so

Very competitive.

Two hours ago, they dropped the price. For 2. 5 max, two hours ago, it's now 1. 60 for a million tokens input, 6. 40 for a million output. I've not compared that with the other similar models, because I haven't quite figured out which models are similar, but I think that's dirt cheap.

Yeah, it's very cheap.

And also, they're boasting one of the only APIs outside of Google that support 1 million contacts, which is also very important for folks. but I doubt that this is competitive with Google at all. Google's like millions of contexts. It's very dirt cheap prices. Simon, we talked about how you can run your whole library of images. Through Google's like Gemini understanding and it costs like eight bucks for like your whole life of pictures, pretty much.

it's ludicrous how cheap the Gemini flash models are.

Yeah.

So it's really hard to compete with them.

but shout out to Quen for this release. Absolutely. We should remember that like when folks are in different countries, they may don't want to use Google or they want to use something else. So that's a great model. The one highlight there is, in addition to this, I discovered that the web interface to which you can try Quen 2. 5 Max. Which is the chat interface, also has video capabilities, video generation capabilities. So Quen now has a video model they didn't tell us about. They don't, no examples, just videos. it stopped working for me as well. I hope they fixed it. But definitely I saw some generations and they look very good. So to our friends who do video generations, try out the Quen. video one, if that works for you, it takes a little while, but the examples there are very good. So I'm very surprised that, we should bring Jun Yang after the New Year's break and interrogate him about, Hey, how come you're releasing video models without telling the world? Like what's going on? so when video model and image model as well, also in big company and APIs, the next thing I want to talk about is Zucks, earning call statements about llama. I don't know if you guys saw this already. I posted like a tweet about this. it came yesterday in the Meta earnings call. Meta The company was rebranded to Meta because of Metaverse. And it looks more and more like the Metaverse is taking the back seat while AI is taking the front seat because that talks more and more about the Insane amount of CapEx they're going to invest in, in infrastructure. They famously talked about while people were covering projects, Stargate, and we covered this last week, Zach and the Meta folks talked about their data centers that they're going to bring online as well. And they literally overlaid the. Size of the data center on top of Manhattan. And there's like this famous image from the next data centers they're going to build are Manhattan size data centers. So just like the whole Manhattan Island data center worth. So that's crazy. It talked about tens of billions of dollars in operation in CapEx expenditure on data centers and infrastructure, but also. Zack talked about Llama4 and I want to bring to hear like some of the mentions that of what Zack said about Llama4 Here's the quote Llama4 is making great progress in training. obviously They know about DeepSeek because there's this blind post from Meta that like they're in shambles. He didn't seem as scared by it in the earnings call, but obviously he needs to show off For investors like they, they got this. But here's the quote. is making great progress in training. mini is done with pre-training and the reasoning models and larger models are looking good. So a few things here. First of all, they used to release LAMA and just mentioned LAMA in different sizes. Now they have a different categorization for LAMA mini LAMA four mini, which is interesting. it's done with pre-training, so they're gonna release a few separate ones, but also they confirm they have a reasoning model from Meta. Now. that's a new confirmation that we haven't heard before. We assume that they will have one. we, from now on, we assume that every foundational lab will have a reasoning model because test time compute is not a scaling law you want to ignore, but it's great to have confirmation from the big Zack himself. and they have larger models as well. He also continues to say our goal with Llama3 was to make open source competitive with closed models. And our goal with Llama4 is to lead Llama4 will be natively multi modal. It's an omni model and we'll have agentic capabilities. So it's going to be novel and it's going to unlock a lot of new use cases. So it seemed like Zack just threw the kitchen sink of buzzwords about AI into this like statement, but. Omni models is definitely something we're waiting for. R one is text only. It's text only reasoning. We are waiting for multimodal reasoners and we haven't seen any, with the exception of Quinn VL reasoning attempt that they did. Right? There's one model super quick. we haven't seen native reasoners. we know the Gemini thinking. Simon, I think you had some comments on this as well.

was just gonna say, is Gemini thinking, multimodal

Ah, we know it's multimodal so it can accept images,

but we don't know if it's omni modal. So we don't know if it's vision examples for Gemini and then they feed this via text. I haven't gotten the confirmation. Gemini 2 flash is omni. They talked about this, so Gemini 2 Flash can like natively output a sound. It can even draw images. They haven't released this, but they talked about this publicly.

seen a preview of that.

Like the Google AI studio has a preview of it. But that's not the

reasoning model.

That's the Gemini 2 Flash model. So we don't know if we still have reasoning. multi modal models, and probably they're coming and Zach just confirmed that there's is going to be, It sounds

a bit like the VL that we talked about before

the 7TB with the capabilities.

So that one, it has agentic capabilities.

At some point, Quen released a version of theirs. of reasoning with vision. Oh yeah.

Yeah.

Yeah.

so that one, but I don't believe QVL

QVQ was it?

QVQ.

Yes. QVQ. So when quill, but like with visioning, encoder on top. Um, What else do, oh yeah, the last thing I wanted to mention is that, Zack continues to say that glasses are the perfect form factor for AI, and that Meta Ray Bans is a significantly well received, device. I have ones here, I didn't put them on because it screws up with the Bluetooth and then you guys can hear me through the headphones. I love these. I love my Metaray bands, absolutely love them. I love the AI integration. I wish that, there was more memory. I wish that I could access reasoning models in there. I absolutely agree that this could be a thing here. This could be a breakaway category that we can walk around with glasses and then they will have AI on them. I think that's it on Lama, on Metal Lama. that's the update. We didn't get the models. We don't know the timelines, but we know it's coming. And then, Lama 4 is going to have a mini model and it's going to come, it's finished pre training as well.

I'm especially curious about the licensing because

their multi model stuff hasn't been, available to Europe before. So I hope they are going fully open again now, because otherwise Europeans just have the quen stuff or Chinese stuff that would be pretty limiting them.

That's true.

so I have this here and, openly, I was supposed to launch all three mini, according to some leaks from BindoReady and from Tio. It didn't launch Ultra Mini yet, so we may still get Ultra Mini today, but for now, so far, Ultra Mini was not launched, but it looks like it's coming pretty much, pretty much undeniably either today or tomorrow, like very soon. Um, So let's move to voice and audio. I think that this is a very interesting one. Voice and audio. there's an update here. So first of all, let's talk about UA. US7B, Wolfram, if you can pull up an example so that you can play, because for me, it's a little harder to play with like audio. US7B is a music generation model that's open source. And we have breaking news, folks, from this model, because when I did the notes today, the model was CC by NC, so you cannot use this commercially. They changed the license right now, so literally like breaking news an hour ago. The license is Apache 2. 0. So not only is this model competing with Suno and Udio and like all of these in the open source, it's a 7 billion parameter model. it's also now, Apache 2 license. So that's great. That's we celebrate open source as much as possible. and, we can play one quick example. I think, I think I can send you a link to that one.

Yeah, can you give me the link?

Yeah, let's see if I have a link for an example.

used it before and, it didn't generate the

audio for me, but the sample they have is, it was very impressive.

Yeah, let me actually play this here and see if this works.

You guys hear this?

Yeah.

Yeah, I doubt that the space folks can hear this.

I have a little bit of a technical thing. Wolfram, I'll send this to you. let me send this, hold on folks, we're gonna play you an example. I'll send this to you from XChat. One second, Wolfram. There you go. Yeah. So here's an example, folks. So again, this is US7B. It's an open source AI model, for full song generation from, HQ University and multimodal art projection. It turns lyrics into vocal and instrumental tracks. And, it works across languages and genres as well. so super, super cool, full song generation, lyrics to music pipeline. they have a bunch of stuff. They have a chain of thought, reasoning in there for song construction in context learning. they can, mimic, Style from uploading audio references. So that's also super cool. So you can upload like your favorite music and then say, Hey, I want something like this, and we're going to play you an example, the thing is this model is, even though it's 7 billion parameter, it's slow, so it can run technically on a 40, 90 or H one hundreds or eight one hundreds, but, I was able to run this via foul and it took a while. It took I don't know, like 900 seconds or so. but now it's Apache to license as well. So now you can use this for whatever you want, for commercial as well. and let's try to see if we can hear an example of this.

Okay.

Let's go.

a keeper.

We're living out my dreams with this. My can.

To waste time gonna, but I brush it off to the negativity. Something strong. Mama waking hard. Wanna make it a proud Michaels of a prince cutting through crowd.

7 billion parameters, just did whatever this is.

It's crazy. Like the words are coherent. The music is separate. This is folks, we're getting there. We're getting like multiple things from open source. so I'm very impressed with this. so shout out to folks from you, Y U E is how you write this. yep. we'll add this to the show notes so you guys can generate music as much as you want. another example, from the voice and audio is refusion fuzz, refusion, released, it's a competitor, like a. Commercial competitor to Suna and Nudio. And they just launched Fuzz today. And the highlight there is it's free. So Suna and Nudio now they require money, et cetera. They have like advanced things. Suna v4 is incredible. Some of the transitions here on the show are made with Suna v4. refusion fuzz is a new one. I haven't been able to this out quite well openly, but it works. It works well. It's fast as well. and then, it's all free. So you can try that one, as well. I think this is it in voice and audio. I think it's pretty much it. The one last thing that I want to mention in voice and audio, this is not from this week, folks, I keep getting, amazed by how Julio's audio. The one with the emotions one. If you guys remember a couple of shows back, they launched it. And I talked about how emotional this is. since then last week, I also played you an example of Woody Allen, that I cloned his voice in whole year's audio. And like it, it had the emotions in it. I keep being amazed by, I think that they compete with 11 labs. on emotional sound. I think it's really good. folks. I think we're going to chat about the last two things super quick and we're going to finish the show. The last thing that I want to bring up is perplexity with R1. So Simon was just here and he was talking about deep research. That's from Google. I think that's only if you pay Google, for Gemini pro. I don't believe that this is like free access. perplexity is also only for pro, users perplexity added perplexity with R1, they highlight the fact that this is only, this is like the model is in the U. S. So it's very funny that they really highlight this in the model itself. So this is hosted in the U. S. because people are still worried of sending their information to China. But, let me see if I can pull this up, show Windows. Oh yeah, perplexity. Let me show you a perplexity search. It's really good. I've been like making fun of perplexity a little bit because, they renewed my thing. You have to go on the little dropdown where it says pro and then say pro with R1, but then it is really good. I asked it, for example, what was the last episode of Thursday Eye about? And it gave me the right examples for the first time ever on perplexity. what was the last Episode of Thursday. news about, it shows you all the reasoning as well. So they implemented the reasoning, hopefully this is going to replicate now in the example, an additional tool that I want to show you is XR AI, which is a. search API for agents. They basically did the same thing and that one is free. So if you don't have perplexity, if you go to, demo. exa. ai slash deep seek chat, they also integrated a similar thing as well. so we can go and take this and see, Then paste it here and do a small comparison. So back to, back to perplexity super quick. this is the result, the most recent episode of Thursday News released on January 23rd covered groundbreaking developers in AI, including major open source, deep R one, $500 billion infrastructure bill and video, desktop supercomputer. That wasn't from the last week's show. that was the two weeks before. So not perfect, but open the operator.

It looks very good.

Yeah.

Excuse me. and, the XR one is also really good. Also pretty much did the summary. That's a bit of Thursday. I was about major news, deep seeker one operator agent, 500 billion project by dense UI, TARS and updates on Google. so really, great results. yes. so what else do we have in the tools? The last tool that I'll cover is Craya AI. I have friends from Craya AI also had the deep seek. If you notice. The three tools that we shout out, we don't usually shout out the tools, all integrated reasoning and all of them now have reasoning in there, thanks to DeepSeek. So whoever, whoever didn't DeepSeek because it's Chinese origin can go and make a better model and show us what they have, because right now we got you wouldn't be able to get this with a one is going to be like too slow and too expensive for folks to run. All of these tools are now using reasoning because. The Chinese lab, they used to be quants and now they're like the most crack team, in AI, at least in China, gave us the reasoning tools for free.

Quick correction, Perplexity did have

all one reasoning in it, but it was limited to 10 users per day. So you could pick it, but you only had 10 users. Yeah, they removed it now completely and it's only a R1 and interestingly, they even removed croc and some other models. So now it's just, GBT for all, R1 and cloths on it. That's all they have right now and it's their own model.

one last, yeah, I think, so this is, I think, the

three tools that we have, Craya. ai is like a multi modal art generation thingy. They also added like a DeepSeek, generation as well. And I think that this is it folks. I think that this is a show for today. I have some more folks sending us different things like, CMD instructions written by, R1 in GGML, and different things. So that's super cool. Agents are writing more code, but I think that we pretty much covered most of the important news except one. All three Mini did not launch.

LDJ wanted to say something.

LDJ,

go ahead.

Yeah, I think yesterday me and Aaron, we put out this, chart.

Yes, I want to talk about this.

Let's talk about this.

You and Aaron Komusuzaki, the guy who posts about all of the releases and papers, you did some work. I want to hear from you like what the effort was.

Indeed.

yeah, so yeah, I guess if you have the link or do you, I'm going to send you the link right now. I got it. I got it. Okay. yeah, I mean, we were talking with various researchers, spoke with, I wouldn't say we, we got like necessarily insider info, we would never do that, but, yeah, there's like various things. for example, if you make some kind of baseline assumptions I would say a majority of researchers that we spoke with agree with, for example, assuming that GBT 4. 0 likely has at least similar data set efficiency during training to Lama B and various assumptions like that. And along with, assuming certain MOE sparsity is likely happening in certain models and, and others like you, you do end up quickly calculating things like, Like llama 3. 1 to four or five B it's fully open source. We already know all the details. That one's very easy to calculate that, that one we're super confident.

So that's coming in at 50 million.

Yes.

And to break down like what that cost actually means, like we saw a lot of the headlines of, talking about DeepSea costing 5 million. First, we wanted to address the people that feel like that's like a lie or something, based on the way they're calculating, which is a fairly typical way of calculating training costs that it does seem to be correct. And, basically the cost is really calculated as like essentially the market rate of H one hundreds times the amount of H one hundreds in the amount of hours you're running that training for. And that's basically if you were to rent a cluster to basically train the specs of that model, that's how much it would cost you.

I have a question for you, though.

They don't have access to H100s, correct?

Yes, correct.

They don't have access to H100s, and the original GPT 4 here, listed in the middle, it also didn't train on H100s. It only had access to A100s. That's true. And so for both of those things, conveniently with DeepSeq v3, it actually still ends up like a similar price as H800s, because The ratio of kind of the, it has less flops compared to an H 100, but then it also costs less per hour than an H 100. And roughly the amount of flops per dollar you get from the H 800, when you optimize it to have similar flop efficiency ends up, basically around the same flops per dollar. And then, for GPT four. It would have cost significantly more with a 100. It's likely more like a hundred million dollars somewhere around there, with the original 2022 training costs, but because of H one hundreds now being significantly more cost efficient than a one hundreds back then, per flop. Yeah. You end up with 30 million training costs there. And, for Oh one and GPT for, Oh, it seems very likely, based on various people working on RL and things like that, that I spoke to that, the cost is Around becoming equal to the cost of the pre training of the base model, pretty much. So that looks like something like maybe 10, 15 million for GPT 4. 0 and another 10, 15 million for the RL that makes it.

then, and same

thing for VV three and R one

and then R three as well?

Probably.

Yeah.

And for deep seq, R one, we actually don't have, there is no published cost of R one. They only publish cost estimates for V three. And so R one is something we're extrapolating based on, other analysis that we did.

So this $5.5 million that everybody keeps

quoting, they didn't publish it.

So the $5.5 million they did publish in for the deep seek V three model.

That's the non reasoning model.

Really?

I thought like the 5. 1, 5. 5, like people attribute this to R1. Nisten?

it is what it is.

They got H100s, bro. All the H100s on eBay. I don't think it matters. I want to draw a distinction though. And you still lost second place. I appreciate both

attempts, but I want to draw a distinction.

LDJ and the run console like it did, and I did exercise to try and estimate. There's a calculator that you guys built that will add to the show notes as well. You can input the active parameter count and token counts and then choose which estimation of. H100, B200, Nisten, doesn't know whether or not DeepSeek has H100s or not. but he's a very, like he's estimating as well, but a different level of estimation.

not estimating.

I'm just saying that. Speculating, yeah. they definitely have H100s.

Yeah.

Anyway. I think maybe one of the most interesting things here, too, is I don't know if you noticed the giant bar at the top, but, so that For folks

who are not watching, what is the giant bar?

the giant bar at the top is it's labeled upcoming 100 K H 100 scale models.

And it's basically 10 times higher than all the other bars below it. Like Lama, four or five B cloud 3. 5 on it, GPT, four, et cetera.

Which model is that GROK three

multiple models, actually.

A lot of people, asking things like, Oh, where's GPT is late, or blah, blah, blah.

where is it?

Yeah, and to make a long story short, over the past two, three

years, cluster sizes have been largely limited to around 24, 32 K max GPUs. This is for many reasons due to fault tolerance mitigations. You have to start implementing at larger scales and interconnect limitations. And a lot of these have just largely been worked around in the past few months. And so you have reports of Microsoft and open AI now training on a hundred K H scale models. the past few months, as well as grok three, of course, that's publicly announced. And then now even Amazon, announced building 400 K training and cluster for anthropic, which is equivalent, roughly 200 K H one hundreds. And meta publicly announced over a hundred K H models coming soon as well. So you have many models all in this year, suddenly going to be around 10 X scalar more than the frontier models of today. And I think that's really exciting.

Yep, and we're about to see whether or not this works out with grok

Supposedly already showing up for people in the chat and then potentially they're gonna also have a reasoning model as well thank you for this research folks We'll add this to the show notes as well So you'd be able to see this incredible like deep insights that you guys get on Thursday from the expert folks here on stage with that I think we're at the end of our show after two hours and 15 minutes I wanted to Shout out a few things. So first of all, thank you everybody for joining throughout, wherever you join us, whether it's on Twitter spaces, whether it's on YouTube, on LinkedIn. Hi, the five folks on LinkedIn join tell other folks on LinkedIn we should grow, but also folks who are just tuning in via the podcast. thank you for tuning in for me week. hopefully this is, helpful for you if it. Is there's a few ways you can support the show. One of them is, if you're listening on any podcast, like Apple podcast, Spotify, give us a five star rating so that we'll get exposed to more people. If you're tuning in via the sub stack, the forbidden sub stack that you cannot mention the next. if you're doing it and you're reading the breakdown of the news and the links, share that sub stack with your friends as well. Leave us a comment on sub stack. Let us know that you're reading this, give us feedback about whether or not. You want longer, shorter, et cetera. it takes a while to produce all this content. And it's very appreciated when somebody comments and says, Hey, this was good, or even with a critic, to help make it better for everybody else, that's also helpful. with that, thank you so much everyone for joining. Thank you, my cohost Wolfram, LDJ, Nisten, Simon Wilson, join us as well. and, thank you everyone for, joining and giving expertise. we. Don't stop this week was crazy. The next one is probably going to be crazy as well. We waited for a GPT 0 3 mini. We didn't get it, but probably maybe very soon. there's also some other things that we speculated we didn't get like Gemini 2. 0 Pro and other things. It looks like all of them are going to come out very soon because people are stressing out about deep sick. So with that, I won't tell you bye bye. Thank you. And we'll see you guys next week. Cheers, everyone.

Cheers.

Bye. Thanks.