📆 ThursdAI - Jan 23, 2025 - 🔥 DeepSeek R1 is HERE, OpenAI Operator Agent, $500B AI manhattan project, ByteDance UI-Tars, new Gemini Thinker & more AI news

Alrighty, welcome everyone to Thursday.

I for. January 23rd. My name is Alex Volkov. I'm an AI evangelist with Weights Biases and you are on ThursdayEye. ThursdayEye is the weekly show that keeps you up to date of everything that happened in the world of AI from week to week. And, we're here to talk about maybe one of the biggest weeks, definitely one of the biggest weeks since the beginning of the year, but Maybe one of the biggest weeks in open source since LLAMA was released. And so, we're going to have, quite a conversation today with, with a few folks that, they're going to join soon about R1 specifically from DeepSeek. And, R1 is a, we already covered R1 on, on the show when it was announced and was in preview, but since then we got it in open source. And so now R1 actually runs on my Mac and I'll show it to you here, and runs on everybody else's, you know, as well. and it's going to be, is going to be quite a show. Also, a few other things happened. Let's, let me just like, like very, very gently, a few other things happened. Like, as a small, tiny announcement of half a trillion dollars investment upcoming in AI from, from OpenAI and, Masayoshi san from, from Vision Fund. and, Larry Ellison from Oracle. So just a tiny, a tiny, tiny investment in the world of AI that was announced on like the president's, third, third day or something. and it's been huge, across the news. and, there's a bunch of other open sources as well. So, we're going to cover those. I think, one of the things that I want to highlight is, there's also a new benchmark that just dropped called the humanities last exam that I definitely would like to talk about. And, oh yeah, of course, but Bidense is all over the place. Bidense dropped a, a computer controlling agent with like 7 billion parameter that beats Antropics Cloud Computer Use. so, all of that, and more, I think. and I've got some very upsetting updates from ML, LM Marina, and you guys know we talk about LM Marina for a long time. just now, just a second before I jump to the space, some very upsetting, kind of like, News about, about, Ella Marina, which we should also talk about. I don't know how verified those are. It's just like a post on Reddit, but it kind of explains a few things. So definitely worth chatting about that as well. all right. I think, I think with this intro, I will just say what else, what else is there to be said, let's, let's dive into our one. Let's dive into our one. I think we'll do a TLDR because there's like a bunch of stuff and then we'll dive in and meanwhile I'll, I'll see that my host will, will join as well. definitely let's do a TLDR. I think we're good on, on folks. I see a bunch of other folks in the audience. yeah, let's do a TLDR of open source and then we'll dive in. All right, folks. here's the TLDR, everything we're going to cover on the next two hours, including hopefully some breaking news from OpenAI. this week started. Very strong with deep seek. The AKA, the Chinese Whale Bros, the quant trading firm from China. That became the number one ai, open source, company in the world right now released. R one, the Reasoning Model, in MIT license. And we're going to cover all of this. This is probably still the, the biggest. The biggest news, they didn't only release two models. They released like a bunch of other, quants, uh, and, distilled versions on top of Quen and Lama. And the vibes are insane. This is like a o1 level model, very close to o1, if not beating o1 multiple places at home. Also in the open source LLMs, kind of LLMs, ByteDance dropped UiTars. UiTars is, ByteDance's computer use model that they claim, 7 billion parameters and 72 billion parameters, controls your Mac or PC and they have an app for both and they beat GPD 4. 0 and Cloud, while running. Kind of, in open source running those models locally did not work for me. I haven't been able to run this. So maybe some folks in the audience can tell us what their experience was, but the, the metrics are fairly, fairly ridiculous. And the examples they show are also ridiculous. And, given that OpenAI is going to launch something, this is very interesting in comparison as well. And, In other open source LLM news, just from today breaking news, there was a new benchmark, a new eval, that's not saturated at all, the top models on this eval is getting around 10%, which is by the way, DeepSeq, which is great. it's called Humanities Last Exam, HLE. So get used to saying HLE, Humanities Last Exam. It's 3, 000 questions, that was written by nearly a thousand subject matter experts, from around the world. And, this just launched from, from a bunch of folks. And then we're going to definitely cover what type of questions there are. I looked at a few, I did not. there's no way I'm answering any of those myself. I need an AI to help me. but the best thing about this is not saturated at all. this is most of the stuff that I have in open source LLMs. I see LDJ just joined. LDJ, welcome.

Hey.

So we covered the open source LLMs.

We're still on the TRDR and we're moving to the next thing that we're going to talk about in the big companies and APIs, an area of Thursday that we cover. And this one is a big one because I think the main pieces piece of news is this one is SoftBank Masayoshi san and Oracle and OpenAI. All launched together is a thing called Stargate Project, and it's an insane investment, commitment that talks about 500 billion dollars, a Manhattan project for AI infrastructure of sorts, and they have announced and signed it in front of the president, and, There's been some speculation and debate whether or not they have the funds, but generally we should talk about what this means because this, this is just like absolutely huge. If you guys remember the 7 trillion that, at some point Sam Altman talked about, maybe raising, this is. half a trillion dollars basically for, for like creating, I don't know, a hundred thousand jobs. Definitely a huge announcement in the big companies area. another thing in big companies, Google Gemini flash thinking. If you guys remember, Gemini has their own reasoning model and, they launched the Gemini flash thinking and they updated it for this year with significant updates, plus 1 million contacts and, So code use as well. so that's great. It's really, really good. It's really good. I don't know if like our one level good, maybe, but like the fact that it's, 1 million context for that thinker, it's also super fast because of Gemini flash. So they did update this. they jumped in, in a few evils as well. That was great. at some point also this week, this week was kind of crazy. well, yeah, this week was kind of crazy, but at some point this week, there's also the whole point with, Sam Altman saying the Twitter hype is out of control. we're not going to deploy AGI next month and nor we have built AGI after all of them talking about, Hey, you know, we were moving our sites to ASI. That's what we talked about last week. So it's really fun to see Sam Altman kind of like throwing a wet towel on the hype on Twitter and then going and like announcing half a trillion dollars investments in the next four years. It's really like, it's really funny over there. the small potatoes there's in the big companies as well as perplexity announced a new search API thing. so we're moving on to vision and video. Nvidia released Eagle two, which is a series of VMs. and they were like very efficient and then they yanked the weights. but also Hug and Face released a small VLM today, the tiniest VLM, 256 million parameters, and that's million, not billion. they released it today in kind of breaking news. this tiny model that runs in like one gigabyte of RAM or some, some crazy things. Beats IDefix from almost two years ago. If you guys remember IDefix is Hug and Face's vision model. It was 80 billion parameters. Two years ago, it was 80 billion. And now it's 256 million. I just, I found it incredible. And I just, a shout out to the Hug and Face folks for this launch. folks, in this week's buzz, which is a category that I talk about, everything that happened in, in, in Weights and Biases this week. Usually, I cover our you know, news, I cover the, the, the events we're doing, we did two workshops in Seattle. I, I did, and some folks from Thursday, we covered like a bunch of stuff. This week, we have state of the art updates from Weights Biases. Wait, let me, I have a thing for this. we have a state of the art. Let me say this slowly again, just, just like, just to like make, make sure that you folks understand what I'm talking about. From Weights Biases this week. The this week's buzz corner is going to cover a state of the art update that we did not me personally. I wasn't involved. I'm just like, I'm the hype man. we broke the state of the art on sweet bench verified with a in house. Agent that writes code that uses O1 from OpenAI and we're now, if you go to the SweBench verified, SweBench website and you go to the verified thing and, the top most results is Weizenbass's, WB programmer from Sean Lewis, our CTO, sitting at 64. 6%, 64. 6 percent of SweBench verified is now solved by O1 based programmer agent. From Weights Biases, I found it, incredible, I was, like, very, very happy to see that, like, you know, in this week's buzz, it's not only about our products, I can also update you about, like, some crazy shit that's been, that's being built within Weights Biases, so, very, very happy about this, we should cover this, when we get it. last but not least, we have AI Art and Diffusion. So who knew on, if you guys remember HY video, the open source video that was released, we now have a 3d model from them as well, which is also super cool. You're just like. Write some text and it generates a 3d model and it's state of the art. It looks incredible. Not only do they generate shapes they also have another model that colors them and so Coming from an idea or one image to a 3d model is now it just it's so good. I've seen all these models before From stability opening. I had shaped these a while ago. This one is looking very, very good. We're gonna show and play around with this as well. And I think maybe the last thing besides the, you know, the breaking news and everything, is going to be ByteDance. It's all over the place. By the way, ByteDance is releasing crazy shit. I don't know if ByteDance was about to say, Hey, we're going to get, you know, TikTok banned. So we're going to just like release some stuff. I have like two updates about ByteDance this week, three last week. It's crazy. ByteDance drops Trey, which is a cursor competitor, which is free. So if you don't want to pay for cursor, you can download Trey and it will import your cursor configs and it will run the, I don't know that they will pay for. They will pay for the cloud tokens for you. And if you're okay with sending your code to a server in China, and if you're okay with that, then you have a free course of competitor. It's pretty good. It looks nice. so very unexpected. Folks, I think that this is it. Like besides the fact that like, I need to add this here. Open AI, open AI, operator in pro tier is about to launch. I think that this is like a hundred percent true right now. I've seen like screenshots already. There's screenshots of the pricing. So we're going to chat about this as well. with that, I want to say hi to some folks who joined us to talk about R1 as we're sliding into open source. and then we also have Pietro Schirano with us in the Twitter space. welcome as well.

Hey, what's up everybody?

And I think that, all of you joined just in time because

now we want to talk about R1. And in order to talk about R1, we need to skip to our open source, let's start with open source,

open source AI. Let's get it started. All right, folks. Open source AI has never been as hot as this week. I think that this is like, this is very clear. Open source AI is the corner where we celebrate everything that happens in open source. Rarely do we get a model that beats frontier models that you can just run at home. Rarely, like it happened maybe twice so far. And we always talk about how open source probably lags behind some, some, some, closed source models in this, in this way or that way. This week, DeepSeek, the well grown firm from China, the Quan firm, released R1. they released not only R1, they released like, like, multiple things in R1. They released two actual, like, full models, which are huge, huge models that you cannot run. and they also released six distillations, on top of Quen. So they released DeepSeek R1, DeepSeek R1. 0. And they also released, DeepSeek, R1 on top of Quen And also Lama based 8 billion parameter and 72 billion parameter deep seek R1 kind of like distills, right? So like the basically trained a huge thinking model. And then they also released a bunch of decisions as well. Not only that they released it under MIT license, which is just do whatever, take it, distill it, convert it, train on it, do whatever. They basically just said that, here's this, take it. we've, we've talked about our one when it, when it launched. A couple of months ago. and it launched only as a preview in their UI. It did not launch, as, as an official product back then. there was no pricing, I believe. and now I think it was like just a toggle in their UI. but now you can download those weights. You can use them for production. You could do whatever MIT license is probably like, I don't know, listen, if you were with me, but like, I think MIT is more freeing than Apache. I think like MIT is just like literally just like do whatever you want.

MIT is like.

A jailbreak to the whole legal system, pretty much. That's what most people don't realize. It's like, this is, it's not my problem. You're a problem now.

Yeah.

They can do whatever you want.

I wasn't trying to hurt you. It just puts it very nicely. So

unlike like other, custom, and we've seen this

from a bunch of Chinese models. Custom licenses that talk about only up to a hundred million users. You can use this and then you can not use it for commercial purposes. And you, you, whatever MIT is just like, I'm removing myself as a responsibility layer from this. You guys do whatever you want with this thing. And it's beautiful that we have this like very powerful open source with that, with, with that much, with that much like openness as well. We did not get any data, just the model weights. And we got a incredible paper, incredible paper, with full of nuggets and, and some announcements, and like just talking about how MCTS is not the way to go and grow. LDJ would love to chat about with you about this as well. And also since it launched the amount of vibes that we got immediately that this model is so, so, so good. it was just something great to see. So we're going to talk about like LM arena and vibes, et cetera. But in order to cover those vibes, we have a bunch of other folks as well. So, Pietro, let's, let's start with the vibes from your side. Folks, we have Pietro. Pietro, not your first time here, but feel free to introduce yourself to the audience for folks who may not know you. and let's talk about the R1 vibes from your side.

Hey everyone, first of all, you know, super

excited to talk about this. I think, as you say, initially it's been like a super fun week from an open source perspective, but yeah, I'm Pietro. I do a lot of sort of like experiment, release a lot of code bases. online, you might have seen cloud engineers is like one of my. I guess like most popular one and then a lot of like different project with the with the goal to sort of like really understand what we can do with those models and like not only from a pure implementation perspective also from a creative way and I think like my common thesis has always been when you build a pipeline for AI when you build an AI experience. Just let, basically, your imagination run free and don't let the model constrain, actually constrain you and just go as crazy as you can. and so one thing that I love to do is like every time a new model pops up, I try to poke it around and see like what you can do with it. And when Air1 came as an API, the first thing I noticed was that they actually didn't let you choose between the reasoning. output and the actual final answer from the model And so and I noticed that you can actually so right now you can actually change the max token coming from the reasoning part But you can just edit the amount of token that actually coming from the final answer And so I was like, wait a second if I just say hey, give me like one token from the final answer Is there a way what I can just extract the thinking process from it? And actually that's What happened? And so I was able to use everyone get just the thinking process from any question and then send that thinking process into another model. And what I found out is actually by doing so, you may even like smaller, quote unquote, you know, less intelligent model actually become smarter. which is a very interesting discovery. And to me, you know, the most exciting part about that was not only we're proving that basically thinking in any shape or form actually makes model better, but there's also a word where you can actually now taking the thinking from everyone and maybe send them to a local model. And now are the sound of the local mall performs better. And then the other thing that I love about this is that Air one right now doesn't support like common things as function calling, Jason, Jason more than a bunch like structure. I was like, it's not multimodal as well. Exactly. And so you can basically just take the thinking from air one, send them to a model that has all those amazing capabilities. And now you get this really interesting, Frankenstein model. you know, they can do all of those stuff really well, like function calling, structure, output, et cetera, but also has a thinking process. And so. I will release, I'm about to release the repo about this, somebody online, give it a name. what do they call it? I call it like RAT, which I thought was very funny, where instead of generation, it's like thinking.

Wait, what is the, what is the thing retrieval augmented thinking?

Retrieval

augmented thinking.

Yeah. I was like, wait, it's actually, but, but this is what I love about like AI in general and also like open space and honestly a little bit of the reason why I was upset. Yeah. That open like open a I didn't release anything about the thinking process of a one because it opens up so many possibility and also like it also like, it's really cool from a research perspective to actually look into how those models are thinking, right? So if you look at the way everyone think about stuff is a bit, it's almost like a handsy person, right? It keeps going back and forth. It's like, wait, this is not what I meant. It's just like so endearing to seem like You know, there's an alien being basically like going through a thinking process. And one of the things I'm going to do right now is actually, okay, how can we now improve that thinking process? Because we get to see it, right. Which is something you can do with like open AI models. Right. And so, just overall really solid release. It's interesting how China is basically like stream rolling in terms of release, as you say, like I actually just. Try their, the by dance AI editor is really good. you know, the thing is like I had experience working in big American company for a while. Like I worked at Uber, I worked at Facebook, I work open table. And what I can tell you, it's like those guys in China, they just don't stop. Like, you know, they don't, they don't, they don't go through sort of like the, the, the, the relaxation that some American companies have. Right. And so, it's very hard to keep up with that. and so it's very interesting to see how the market is shaping around, these releases. A

hundred percent.

Thank you. Askerano, I love that. I love the rat, return on augmented thinking. I love the fact that you can transplant like the, the thinking process into another model. And just the fact that like. the amount of stuff that we can do now down that we have, like this model is, is open. I think it's, it's great. thank you for joining stick with us because we're still talking about our one and the, like the crazy amount of like stuff that we got with our one. I will just go through some of the emails before we jump into like some, some other, some other. Vibes and some other things we can do with this. so. Just in terms of evals, folks, just the stuff that they've posted, but also some other evals, because we no longer trust EleMarina. R1 is getting 79. 8 percent or AIME 2024. AIME is like competition math. compared to I don't know, they added a one preview and the previous DeepSeek and they're beating everyone else on AIME right now. this is probably the non distilled version of R1. Codeforces, they're pretty much up there with 96%. Math500, which is like, we know that math is Saturated as hell. Right? So like, we're not gonna, we're not gonna get too excited by math. We're going to talk about other benchmarks that are better, but just like looking at this graph of evals, the fact that we got an open model in open weights with MIT license that gets 79 percent on AIME competition math, 90 percent in the MMLU, 49 percent out of the gate on SWEBENCH verified. that's not near close to SOTA, but like, hey, this is just like a one shot. crazy thing, just an absolute beast of a model just based on the regular evals. But as we know, Nistan here, I would tap on you to talk about the vibes and maybe Ray also you as well. Just evals in the paper is not enough. We've seen models come and go, and we've seen folks post these evals and then the model itself does not perform as well as we want. And we've seen. The exact opposite as well. DeepSeek has been consistently, since we've been talking about DeepSeek, DeepSeek Coder, since forever, DeepSeek has not ever, as far as I remember, Nielsen, failed us in evals to vibes ratio. and they didn't, didn't fail us this time. What's your take, Nielsen? You've played with it.

It's pretty good, but You have to throw away the thinking

part, and now a lot of the interfaces that are coming in do hide it. With DeepSeek, you do actually have to hide it because there is a lot of redundant information in there. It repeats a lot of things. Times, but, no, no, the, the, the emails are, are spot on. What surprised me the most was the 1. 5 B. I did not expect that one to be good at all. I, I ran the 1. 5 B locally and, he was doing stuff in Erlang. It was, I was just grabbing the final answer because it was so fast. And, yeah, so far it, Yes, so far it checks out. I, I threw some code at it. It wrote an entire, it wrote an entire Tetris game too. I was just like playing it for like 20 minutes, just to, to test it out. It all worked. The score was working. And, I also tried the exact same thing that, that Pietro did. Sorry, I have construction outside, but, The, so I, I use like a 78 B model that I made, like a commander model that has all the tool use. And it's also fairly smart that it's not bleeding as smart. And, I grabbed all the reasoning steps from the 32 B and I threw it in the other model. And the other model was also way better. with them. So, so far it checks out, looks good. I would just say, yeah,

I would just say, you mentioned like how good the 1.

5 B performs. The, the, the thing that they mentioned is, DeepSeq R1 distilled QN 1. 5 B, which is a mouthful. DeepSeq R1 distilled QN 1. 5 B, which is distilled version of DeepSeq on top of 1. 5 B. Outperforms GPD 4. 0 and Cloud 3. 5 Sonnet on math benchmarks with 28. 9 on AIME and 83. 9 on math. This 1. 5 billion per like model, given the reasoning powers distilled down from the huge model, performs better than 4. 0. performs better than GPT for all like the, the, the flagship model for open AI that they've like spent billions of, I don't know, millions of dollars of training. This is 1. 5 billion parameter model that now does this. It's absolutely insane. This is like, this, this makes no sense. And it makes sense in a world where. how should I say? reasoning is built into the mouse and distillation is like a beautiful thing, but just like for anything else, it just doesn't make sense. let's talk about some other evals and then we're gonna like, skip on and, and chat about like some, some, some things from the paper and, and how, how things work for us as well. but

by the way, I highly recommend people if you're gonna run it at home.

Honestly, the 7B and the 14B are, are pretty good. I ended up just finding old boxes, old GPUs that, that I had. I slapped them on there. I left the 14B and the 7B running overnight because they, they take a while. And, I mean, a while, like on, on CPU, you can just leave it all night. Just running at whatever. Problem you have and come back in the morning, you have, you actually have a extremely capable, model. I am very surprised that now you can actually have it on pretty cheap consumer hardware, as long as you just leave it aside. But yeah, I just, I tried arch Linux for the first time. I just slapped it on a couple of boxes and I just got LM studio. I just got their image and that's all you needed. Everything just worked.

I tried LM Studio and I have a tip for folks,

because LM Studio is great. It's great. Like you

don't need to install the drivers.

It comes with CUDA inside. So just use like any modern Ubuntu or whatever you have on it. That, that, that comes, that has, that is like a newer kernel for the OS. And then just slap LF studio. You have a rag API. You have tool use and you have the things that I am, I feel like I'm getting old. I don't want to set all this stuff up. I just, I just fired this thing. It's good. And, it's, it's very self contained. You don't, it doesn't make any outside API calls if you don't want it to, unless you're downloading the model. yeah. So

I tried them studio as well.

The thing that I got from ElumStudio on the, on the 32 billion parameter model was around five tokens per second on my Mac M3, and then I saw this tweet from Georgi Gerganov, the ggoat from, from ggml that posted a snippet, like two lines snippet that I'm showing right now, like brew install llama. cpp and llama server, you download one model and then you download another model for speculative decoding because if you guys remember what speculative decoding is, for folks who don't, speculative decoding is a way to kind of like help a bigger model by, by, kind of like using a smaller model to, to search super quickly to the space of possibilities. And this speeds the inference significantly. And so, Georgi posted this, this, this snippet that you can use because the models are the same. So the Quen32B and Quen1. 5B both are, pretty much the same architecture. You can use the smaller one to, like, run speculative decoding for you. I got Like 10 tokens, 11 tokens per second up from five. So if you just run the 32 billion parameter model, that's like one speed. If you use speculative decoding significantly faster, and I use that, I wish that LM studio is going to use, going to support speculative decoding. But, this LM studio thing was like, Sorry, the, the Llama CPP thing was like super, super great. It launches a chat interface. You can chat with it. Also great. ElmStudio is great. Llama CPP is great. And the whole thing about open source is we're now getting all these things and we're going to talk about everything else that we got from R1 already. So we, we, we chatted with Pietro about the frank, the rat with real augmented thinking process. but we can chat about like some other things as well that I already saw.

By

the

way, really quickly with, with MLX, this is only code that people hack

together, with speculative decoding on, on MLX, you can end up up to 80 TPS for, A 32 B. So it will take a few weeks or maybe days for all of that to get, to get integrated in, but, yeah, speculative decoding is the way to go. Honestly, with these models, the very interesting thing is that because it's reasoning stuff out. You can leave it running quantized because it's going to think over and correct itself. Right. So even if it hallucinates a little bit more, just because of the massive speed bump you get. It's worth it to just run it faster, quantized, and, I'm looking forward to what the benchmarks come out with this just running very, very quantized because, again, it's reasoning, it's self correcting, so it's, it's basically

saying we can quantize the models down, but let them think for

longer and because of test time compute, it will just like achieve better results just because they run for longer, right? It'll be like twice as fast. Yeah. Yeah. Yeah. all right. I'll DJ, you had a comment as well. We haven't chatted about you. Like, and I want to specifically talk to you about like the, the paper stuff. I don't know if you read them, but you had a comment from, from before. Feel free to chime in here. Yeah, sure. Are you urban? Yeah. You're

mentioning, LM studio.

And I was just going to mention, I talked to some of the guys over there recently and they do plan on actually implementing speculative decoding natively and tell them studio soon. So that will be cool. And also there's. the concept of daisy training with speculative decoding, which I think will maybe become more popular where you basically have, let's say, instead of just a 32 B and a 1. 5 B, you also throw like the seven B into the mix. And so for things that are actually a little bit too hard for the 1. 5 B, then it can offload to the seven B while still not having to fully go to the 32 B. And so you kind of get even this extra speed boost from having to use the 32 B even less now, and, and that's even more efficient.

Yep.

LDJ, I wanted to chat with you about just like some other like things you saw from evals, but also from vibes. What are you, what is your take on R1? did you see this moment, that they talk about the aha moment? Did you happen to like catch this conversation about them getting the aha moment from the model, but also aha moment for the researchers?

Yeah.

Yeah. This was really interesting. It seems to mirror a lot of, what like I've talked to the people kind of privately and also from what I've heard around with open AI and the people working on oh one and it really does seem like the best way is to kind of have this method where you, you let the model just kind of go towards the goal of reaching a certain correct outcome and you just iteratively and continuously have this reinforcement learning process that can scale and then eventually you have this emergent, you know, phenomenon of success. of these aha moments coming from the model and the model just kind of learning on the fly, how to, question it's, it's previous strategies, how to develop new strategies and the single output and all those things.

Yeah, I love that they framed it as like, this was an aha model

for, for the, for the model itself. so it's like learn to allocate more thinking time to a problem, by reevaluating the approach and then also aha moment for the researchers saying that like, they don't need any fancy, like, Monte Carlo research, and I think that this is like the highlight of the paper, at least from, from the top, right? Like, you don't need some fancy, like alpha zero type research into this. Like this works just by scaling, the RL, the basic RL, the reinforcement learning, techniques to get to this level and they discarded. They talked about what didn't work and they talked about. they tried MCTS approaches and they ended up not needing any of this. This is like simple RL scaling for, and then thinking like it turned out to be incredibly, incredibly great. so I think the vibes are great. so the thing that I wanted to cover as well is not only do we get a model. That's like incredibly performant. We can talk about, we can talk about evils all the time. Like, evil from, from Xeophone is great. Like there's a bunch of like life code benches. O1 is, sorry, R1 is getting very, very close to O1 as well. we, we have this evil somebody sent from Ollama of every distillation, version compared to, compared to like the, the, the O1 mini and Cloud Sonnet. And, It's crazy. Like even the tiny models, like 1. 5 B and 7 B are bidding like sauna 2. 5 on AIME, right? Because again, sounds is not like reasoning model. Am I, we know that AIME like competition math needs like, is, is getting a better use by reasoning, but it's crazy to see these numbers, folks. This is like absolutely crazy. I do want to talk about like API performance and for coding as well. We have Ray here. Ray, you, you watch this either polyglot benchmark. Probably you saw this, and then also. The, the price performance for this is incredible, right? Like we saw the price performance for, for R1 compared to getting very close to a one. The price performance was ridiculous. We're getting like 14 cents for a million tokens input. And that's not cash. Cash is even lower. And, 2 for one million output compared to 60 for a one, which is going to go down, but still like the, it's like 50 times cheaper than a one while getting close ish to the results. You've been playing with this. You've been probably coding with this. I saw you, you went live with, Cursor coding that's now implemented this. Yes. Yeah. So

cursor just added the model.

So now if you go into the model settings and just go to, DeepSeek and just go ahead and check mark that, you can actually do a conversation in chat. to actually see the reasoning, tokens, which is, I actually find very helpful for generating plans. and I've also just used it to refactor some code. So, this is actually going to be the moment, I feel, where we're going to do the anti slop movement. So, I think DeepSeq is going to go in and get the slop that's been generated by, Claude Sonnet and some of these other models. And actually, clean it up, which I found really, really impressive. just in one shot, actually, it's like, Hey, I just gave it some vibes based type of things like here's this file, that it was generated. It looks too complicated. I want a simpler architecture. Can you just please review, and generate a new plan? And so it basically does the thinking and it does the reasoning and it's really.

I'm just really blown away with how impressive it's been able to just refactor that on then in cursor composer, you can actually. So you can copy the chat out and put it into Composer, and have it do its thing. The unfortunate part there is, like you're saying, it does cost, 20 a month for Cursor to use this in the premium model, but you can use this with OpenRouter, like you're saying, and you can use it in other IDEs outside of Cursor. so that was something I was exploring as well with VS Code. So you can use VS Code. and then there's another one called continue. dev. Just hook it up with the open router and you get the exact same API view that you're seeing there. and do this exact same exercise, and then to apply the files and so forth. So, yeah, I'm really impressed with how it's thinking through. And I think the Other part that it unlocks for those who are coding is by looking through its reasoning, you can actually see it kind of stepping through the code, which can help you as well. So if you're not really too familiar with the code base, just by watching it step through the code, you can just start copying those code snippets and start finding them and kind of tracing through the code yourself. to debug components or go deeper on a bug, maybe you're not really thinking about it in a certain way. And so, just an initial vibe check, like you're saying. For me, I'm like, I'm actually really excited to start to go refactor a bunch of code. and bring it in because of that large context window. you know, to be able to do that finally, right?

How big is the context window?

I don't actually have this in my notes. I think it's like a hundred something thousand

tokens, like the 100k tokens type of thing.

Oh, wow.

Okay. Yeah, we need to distill it. One, I think,

one we need, but yeah.

And it's on, OpenRouter. They give you, I think, 60k tokens. So, I also made an app called LLM Calc. And so, for those who are wondering, you know, big ups to Nistin, because he kind of gave me a review on, how I should do the calculations for, like, the token window, or the context window, and these other things. and it's open source, so you can go play with it. And if you're trying to configure and figure out how much VRAM you need for your context windows. That could be super helpful, because there are people who are running eight, eight gigabyte MacBooks, 16 gigabyte MacBooks, and your context windows are going to be really, really small, if you're running some of the smaller models and stuff.

So I went, thanks, Ray.

I went to check super quick. What's the, the context length for this model? And some folks are commenting that it's like 64K. I was like, Oh yeah, 64K. No, that makes sense. and it's not that much, but Ray, you said a hundred something. So DeepSeek, the actual API offers 64K. based on open router, but we, we have hyperbolic and together. AI already support and fireworks already supporting our one. They already put our one on their things. they're getting, significant, significant lower, Throughput. So like 12 tokens per second, unlike like 15 tokens per second, but they're allowing one 64,000 tokens. Yeah,

I was gonna say that the model itself in the GGUF file,

I also saw about one 60 5K.

Yeah.

The architecture of, of the model.

I think they might have trained it or extended it up to. Up to one 60.

So that's, that's incredible.

And then, I don't know how And, and

the, sorry, sorry to go ahead.

Go

ahead.

For the coding and, agent stuff and for, SWE bench.

You do need longer context. people were saying that they often were running into issues, even with the regular DeepSea KPI when it was cutting out at like 60, 000 something. Oh, and the other thing is they trained the distilled models to 128K. So now you already, without yarn. So you already have that at, at 128K, even though 1. 5B is trained up to 128K, no one's tested that.

Is there an input and output limit as well?

I think, I think

outputs is, I don't know.

I think we need to test. Yeah. Open

routers, max output is, I didn't find it super reliable.

because when we talked about the 1 million model from, from last week, and they talked about like a deep segment, sorry, open router mentioned like 1 million output, and I doubt it was 1 million outputs. So I'm not sure that like we can trust open routers. Mentioning of max output, because some of them say max output 1 31 k. I will say though, on this topic, and I, LDJI wanted to get to you before I move to the, like, what people can use this for because of open source, but feel free to comment and then we'll, we'll move on.

Yeah.

To mention on the output tokens point, on the, the deep seek V three paper, they did mention all the benchmarks are capped, 30 2K output. For reasoning, and I'm not sure if that's like, that's the hard limit for how much you could actually use, but at least in the benchmarks in the paper, that's what they say the cap they use this. And, I guess the a hundred K 164 K, I think that's just the total conversation limit.

All right.

So, the thing that I wanted to highlight is because folks, when we talk about open source, it's not only about running this models on your own hardware, which we can now do, although the distilled models probably, but we can now do, or companies like, hyperbolic or together, all of them can now put this up on their servers. you can also run this on your hardware if you want to, if your company and you have GPUs idle, you can run DeepSeek if you want to, although the requirements for the bigger models are hard, open source also allows all kinds of crazy shit. All kinds of crazy shit. And so one of the things that I found like very interesting, super quick is this thing from Google, where, Google decided to implement a reasoning effort, knob, do you guys know, like, remember that like open the eyes API for a one I believe O1 does this. I don't know if O1 Mini does this. I think O1 Mini doesn't do this yet. but I think they mentioned that O3 Mini will do this. You can specify reasoning effort. Essentially, if you think about the test time compute kind of like the longer it thinks, the better it like performs thing. You can tweak the longer it thinks with the reasoning effort. Low, medium, and high. you cannot do this with DeepSeq. DeepSeq doesn't give you this effort. But because it's open source, people just like hacked it together. And the way they hacked it together is genius. Basically, the, the, the, the sampler, which, which takes kind of the top, the top tokens and then decides, okay, like this is the next token that I'm going to show. they replaced the sampler and said, well, like every time it gets to the end of thinking token, there's a specific token that says end of thinking, they replace that token with, but wait, or, wait, but, or something, right? So like they replace this token with another, like text that says like thinking, and this tricks the model into keeps thinking. And so you can ask like a very simple question, like, you know, what's one plus one. And then, it will do like 57 thinking tokens. And then if you do this sampler thing and you just like keep replacing randomly, the end token with weight, but, and, and so, and like different things that like, keep it thinking the model will just keep thinking and then like in this example, they got, thinking tokens are like one 173 thinking on top of 57. So basically you can control how much the model gets into just like the loop of thinking. And I think that person, I think that they ran. The, they ran into the maximum limit of 160, 000 tokens in the end for this, like one simple question, it goes into loops and goes like all kinds of crazy shit, but I just personally love that. not only can we see the thinking itself, which is incredible. Like I'm pretty sure that this is like possible as a custom sampler or whatever there's not a lot of code here Although I did think that there's a git repo for this. I'll add this to the show notes folks By the way, if you're like if you have no idea what I'm talking about the sampler, whatever I'll add this link to the show notes. This is a github Gist basically that allows to replace the sampler with you know, the end token with something else and you can keep going forever I don't actually know if this improves performance. I don't think everybody done evals on this. So I don't know if this is quite like the reasoning effort or it's just like faking it. But, we should test this out. We should do some evals. all right, folks, we've been talking about DeepSeq for an hour. any, any last things that we haven't talked about, folks in the comments, maybe anything that that you want to know about R1 that we haven't mentioned? Anything important here that we haven't like mentioned? Oh,

shout out to,

yeah, yeah.

We're going to get there. But it was like our one, I just want to wrap up. No, just shout

out to Bartoski.

Cause he took him like all night or two days to make the quants for the full one. And he even made a one bit, like a 1. 5 bit one. That's like 140 gigs. so I think that one just puts a gibberish, but he put the time to make all the, all the different quants for, for the full model. So it will be interesting to see. And what bit rate it becomes Smart, like probably I'll guess it around three beds. It might actually be pretty good even with all the hallucinations factored in. So yeah, that's, yeah, that's about it.

Yeah.

Right.

So the reason I was mentioning humanities last exam is because a deep seek got number one on that benchmark now. Yeah.

So I think that this is a great transition out of our one.

thank you Ray for joining. I know you have to hop as well. Cheers. Let's move on to the kind of the next thing in open source before we jump and talk about like a bunch of other stuff that we have to talk about and it's already an hour in, let's, let's do a smooth transition. DeepSeq R1 also, not only are the vibes immaculate, but also, We have this new benchmark that was released today. So shout out We're not gonna use the breaking news button because it's gonna happen before the show, but it's okay It's called humanity's last exam and this is a very unsaturated benchmark as you guys know We talk about benchmarks all the time MMU math all those things and they are always always They're getting close to saturated like like math is at 98% saturated, I believe at 99 percent MMLU is saturated. and we talked about, frontier, frontier math, which is an attempt to, have a very, very hard math problems. and, like, I think back at the release, the top models were getting only like 2 percent on those. And then all three was released and said like, hey, we're getting 25 percent already. And so now it's always good to have new benchmarks that are unsaturated. And the, the researchers at OpenAI get, get very excited about those. this new benchmark, this new data set and benchmark is both a public one and a, held back set as well. So to prevent, overfitting, it's called humanities last exam or HLE. I think, remember this name. I think we're going to talk about HLE a lot. And the tradition LDJ that you mentioned is that, already they, they showed as they released it. that DeepSeq R1 is the top performer, even beating O1 on this benchmark with accuracy of 9. 4. So imagine how hard this benchmark is if the top reasoning models that we have right now, outside of O3 that didn't launch yet, are getting less than 10 percent completeness on this. Benchmark, the benchmark is com, composed of math at 42%. physics 11%, biology and medicine 11%. And other, like other questions, computer science at 9%, just like size of, of the dataset. And so it's 3000 questions across a hundreds of subjects, including those percentages that I told you about. Some of them are public, really, some of them are private tests that that held back. and, More than a thousand subject matter expert contributed to this thing. And I just want to like show and talk about some of the questions because they're, they are quite crazy. as far as I will not be able to, without DeepSeek to answer any of them. let's take a look at humanities last exam. lastexam. ai is the, is the URL. Humanities last exam. here's a question. It shows it's, they're also multimodal. but there's a question here that shows like a. A Roman inscription. And the question is, here's a representation of a Roman inscription. originally found in tombstone provide a translation for the palm marine script, a transliteration of the Texas provided. And so they give like a translation of the text. So this is like a, a question that comes from, Merton College in Oxford, and they mentioned the author, and this is like one example of the classic question. Translate Roman scripture. There's also like a bunch of math questions that I will not bore you with. There's physics questions, they look crazy hard. And the data set, supposedly the data set also shows, the answers as well. Let's actually look at the data set. There, files, blah, blah. Can we see the files actually? Oh, agree and access repository, data set card. Yeah, like this. So they have, the questions, they have images, image previews, and they have the answers. So they do have the answers. So we can like, yeah, we can do evals on this data set. there is, there's answers on the data set. And then, you know, main models are getting just a little bit of just 10%. I think it's lovely. And this, we always talk about new benchmarks. I think this is, this one is great. We, I don't know why it doesn't load, but Yeah. HLE folks, remember HLE And then Sik, is, sorry, R one from, from Dipsy is, is now currently the leader in this. And we're gonna wait until, you know, all three is gonna get released, which is supposedly very

soon.

Have, have you, have you guys brought us that, at, at one B? not yet. No. Would be, it would be. I mean, we're always looking, everyone is for, Like a nice, easy to run, somewhat trustworthy benchmark. right now, most of just use, GPQA and either the math or the medical ones here and there. So if this is like fast to run, it seems to only 160. This, this will be, this will be interesting. Oh, it's only 3000 rows. It's only about 3000 questions. Yeah. Okay. So GPK is like a thousand five hundred. This is not bad. This is pretty good. Cause I don't know, he was 65, 000. I think a while. Yeah. Anyway, I'm, I'm looking forward to somebody has some easy way or it's an LMP.

I think that's going to get built in and folks remember HLE

because we're now, we're not getting like a very, very good benchmark. and, Very non saturated. 10 percent is great. I actually want to see which 10 percent it answers. maybe now that we have our one, maybe we should run this on Weights Biases and test it out. I'll let you know, folks. Maybe, listen, I think it's a good idea. alright, what else do we have in open source? I think we're done with open source. Let's move on. Oh, no, of course. One other thing happened, this is a tiny thing happened in open source, as we're like moving towards almost an hour into the show, ByteDance released something called UI Tars, and that also kind of blew up, although I wasn't able to run this. I will say that I don't actually, I cannot actually confirm, how good this performed. But, ByteDance ran UI TAR, ByteDance released UI TARs also in open source. they claim state of the art on computer use. So if you guys remember Anthropic released computer use, we're due to get programmer at some point, although it seems less and less likely, that the, we're going to get programmer today. Although it is, there are examples of programmer floating around and there are examples of like, if you're an operator. Operator. Sorry. Yeah. We're going to talk about programming in a second. Thank you. Thank you, LDJ. A programmer for OpenAI is supposed to be a thing that controls some stuff. I'm not quite sure if that's going to control computer, but supposedly. but meanwhile, in the open source, we're getting UI TARs from ByteDance. It's a state of the art that beats Entropic, computer use model. Actually two models, 7B and 72B, and they claim that on different benchmarks, they, they improve previously to, to, to their, like UiTars previous, but also they beat GPT 4. 0 and Claude on, on like pretty much all of the benchmarks that, that like visual web bench and web SRC and screenshot QA. There's a lot of benchmarks here that I am not familiar with, like Android control low and MM2 web, like. The whole world of benchmarks in computer use is like fraud, so I'm not, like, familiar with them, like, that much. But, I found it, like, very interesting that they claim Sota with, like, a very small model. I wasn't able to run this. Now, the thing that was awesome is that they released, two models. I think there is four models, SFT and DPO as well, but they also released an app that you can, that you can run on your Mac and this app is basically connecting the dots, right? If you remember when Entropiq released computer use, they didn't release any apps, they just said, Hey, we've trained this example of, of cloud for computer use, you can run this Docker. Completely on to you. If you want to run this Docker and this Docker will run, whatever, blah, blah, blah, open Firefox. these folks released an app it's called the UI TARS desktop app. And then, this app basically allows you to. Execute the mouse clicks and keyboard clicks of the, of the thing that, that, the, the UI tars is basically like telling you, although they did find that there's issues with the GDF models with quantization. So, they removed it looks like, but yeah, cause people were complaining that it doesn't actually work. and they removed it. But, if you have enough GPU resources, that's what it is saying. You can deploy the non GGF versions and this model will be a state of the art computer use model so far, if you can deploy this, in the open source. And I'm pretty sure that this is like Apache too as well. So you have an app, you have two, models that control your computer. And I think there's also like one, like. In browser agent or something. Super cool. I'm waiting for an actual thing that will control my computer. They can trust, open source makes me trust it a little bit more than a bigger company. I don't know, what about you? but, I want to hear comments about this and also folks in the audience. If you ran UI, TARS. Let us know if this works for you. Eldijay, go ahead.

Yeah.

So in terms of, OS world benchmark, I'm a little bit familiar with that. And it, it's generally, it, there's kind of some classes of computers, benchmarks that are more browser focused and kind of others that are more operating system control focused, like how to go into the settings and properly change this thing in the operating system, this and that. And so it seems to score especially well on those types of operating system, desktop type things. on the other hand, like the leak benchmarks for the operator open AI model. It kind of loses in that benchmark to this, however, it scores better in like browser control specifically.

I've spent quite a bit of time on this stuff since basically since

open interpreter came out, I was also doing similar, similar things on, on, on my own, I think it will get there. This is stuff that we actually need to see, like, what's your final product that you can do with it? Like you, you have the little nine year old girl making apps and. It's like, okay, this is cool. But then you have literally every other agent company that doesn't really do anything. So, anyway, it's pretty nuts that they actually just give it all out as, as stuff. I would use another machine. These things are not safe to tell people. I might actually just delete your data if you, by accident.

So

yeah, the, the OS is not really set up to handle it.

And, but yeah, this is Apache two, right? Is that the license? Yeah, it's pretty. That is, is pretty nuts. The model, the main model itself is, is, is pretty capable. I don't know. I'll give it a little bit before we actually start seeing useful stuff out of this. I would say expect a lot of hype coming forward. There's going to be a lot of, yeah.

Agents this, this year, the computer, control your computer,

especially when program, operator gets released and people were like, Oh, I'm not going to paying 200 bucks for that. because apparently it's in pro tier only. I will just say, Nissan, one thing that Thinking about, they also mentioned that you can run this with mid scene JS, which is like an in browser agent. and so it's a JavaScript AI power, like SDK would like controlling the page assertions, blah, blah, blah. browser environments, a fairly safe environment to be in. So if you want to run like a new. Chrome instance, install this there and then have it access only the stuff that you are okay with that is kind of like a docker on its own. Like you can, you can encapsulate and be fairly safe if you run this type of automation only for web in the browser, the browser environment will not delete your computer files. So you can do that. So only if you like log into the places that you want to log in.

And just quickly one, one last thing, remember that this needs

to generate, I think it's like a few hundred, like 200 something or 300 tokens every time it's taking that snapshot. So you do need a somewhat powerful GPU to be able to handle the 300 tokens input. And so for example, it just doesn't work well on. CPU only, even if the models are very small. So yeah, just, just keep that in mind. You do need a somewhat fast interview and then you'll, you'll get the benefit of it.

The

browser one, I'll actually, I'm gonna, I'm going to try that

one.

Yeah. Yeah. I go ahead.

Yeah, I agree.

The like, people don't want to usually have to set up their own virtual machine and the browser method and having your own. browser profile set up for it is good, but apparently what they're doing for things like operator, which I think other companies will start doing too, is actually, they're basically having like a custom browser system kind of built into the chat GPT interface. And so you go into this operator mode and it actually, it has this type of like dedicated browser window inside of chat GPT for the. Actual operator system and then it has like all your logins that you wanted all the payments that you wanted to have access to It's all managed within there.

Yeah, it's like when we hacked open interpreter and we saw

the entire kubernetes container with all the the stuff in they I mean this is what The code interpret, sorry, this is what code interpreter was.

Yep.

It was like a 16 gig container that let you do all of this stuff inside.

You just had a very limited interface. So now they're going to add vision to that, which makes sense here.

Somebody just told me that we have potential breaking news.

I put something in

the chat, by

the way,

well, not, not

that, like something kind of different breaking news.

Oh, okay.

you, you want to go with, with different breaking news?

it just quickly, Sam just announced that free tier is going to get O3 mini.

Oh, that's dope.

O3 mini is that, O3 mini is very, very good. That's insane. That's nuts. Yeah, it's like O1

level.

Yeah.

That's crazy on the high is like, wow, that's crazy.

the, the thing that I wanted to mention is like still in the open source. We have a lot of open source here,

just really quickly shout out to a small LM team that just made

the smallest, vision language model. Which is like, yeah, like half, half a gigabyte to now, actually, you know, the, the smallest one is like 200 something megs and you can see, so

that's okay.

So let's, let's chat about that super quick and then we'll conclude with open source. Nissan, what you're mentioning is hug and face release small. VLM, which previously, there is small LM and now there is a small VLM, which is the tiniest vision model that you can run. It's 256 million parameters and 500. Million parameters, so 0.2, five B , and 0.5 B. Basically, if we're measuring in billion parameters, there's like 0.25 B. the, the incredible thing they're mentioning is this 0.25 B billion parameter model outperforms an 80 billion parameter model that they released before from hang and face called IEF X. so just like to show how quickly this, this tier progresses. it's, it's tiny, tiny, tiny, tiny, small, just, yeah. So like 16 months ago, 70 months ago, they, they, they beat the performance only I defects, it has onyx checkpoints. So you can run this in the browser. So I'm pretty sure that like right now, or like tomorrow is already run. It runs on under like one gigabyte of RAM. It's crazy.

Any old camera or toaster that has 400 megabytes of RAM left.

And run this now.

Yeah,

it's pretty crazy.

folks.

We're, we're going to do a small reset. And just say that we've covered the open source, piece. If you missed any of this, we, you're on Thursday. I, and, if you missed any part of this, you, you can follow up with the podcast that's going to get released, everywhere available, everywhere you get a podcast, just look for Thursday. I, We upload this podcast every week at the end of the show, or at the end of the day of the end of the show. And so, if you've missed any piece, feel free to follow that, follow up with that. There's also a newsletter at the same address, thursday. ai. news. if you want to support the show, the best way to do so is to share it with friends, come to the live shows, but also give us five stars reviews on, on the podcast mediums. Meanwhile, I would just say for folks who are new here, We were here every week for the past coming up on two years, folks. It's going to be two years in March. Very, very soon. We should probably celebrate the two years anniversary of Thursday. I, and we keep you up to date on everything that happened. And my name is Alex Volkov. And the reason why this show keeps happening, at least from my end is because I work at Weights Biases. And. I will just do this thing where, usually, usually I give my co hosts a little bit of a, of a break and I talk about this week's buzz, which is a, a, segment of the show where I talk about Weights Biases, achievements, results, products, et cetera, not this week, not this week, Nissen, stay around with us because this week we have a state of the art announcement from Weights Biases and it's super cool, so let's go to this week's buzz and talk about this, this crazy announcement of Weights Biases. In this week's buzz, we tell you all about Weights and Biases Weights and Biases Weights and Biases And we're back. We're now at the Waits and Bites of Cornell. But now I have my co host here to tell you all about this week's buzz. And, I was really surprised by this as well. I will tell you, like, I got a ping, maybe like, I don't know, a day before about this in Slack, it was like, Hey, we're about to release this thing. what this thing is, is a new agentic framework that's called WNB Programmer. that's kind of internal is going to get released. I'm, I'm, I'm hoping very, very soon. that's beating the, one of the top benchmarks for agentic frameworks for coding, SweeBench verified. So SweeBench is, has been around for a while. SweeBench verified. I believe that this is a open the eyes, like, like condensed higher tier. verified, version of SweetBench and, so this week from Weights and Biases, we have announced that we are now the leaders on SuiteBench Verified, which is, which is insane. With, with two of the top three places, 57. 4 percent with the programmer with O1 high, and, Wait, wait, wait. So we have breaking news. I hear we have breaking news with programmer launched three minutes on YouTube, opening a channel. What? Ah, that's crazy. Okay. We're gonna, we're gonna tune in folks. We're going to watch this thing together. somebody sent me a link. Gents, the person who just like showed this, send me a link. We're going to watch this together. meanwhile, I'm going to cover this. the thing that was achieved here with the programmer with W and B is specifically. The fact that we're using the reasoning models 01, and we paid a lot of money to do this, but also the fact that other agent frameworks, they're not necessarily geared towards, 01 use. We've talked with, open hands with, with, we talked with, crew AI as well, like reasoning models and, and agent frameworks don't really like mix together, or at least they didn't used to. And, Sean, the CTO of weights and biases. was able to figure out how to best utilize, reasoning models to get this like incredible, incredible achievement. And so we have a blog, I'll add this to the show notes as well with a bunch of examples. We didn't share all the learnings yet, and we will hopefully will Sean will come to the podcast. I just was very excited to see this at the leaderboard of, of, of, Sweden's verified the organization that got 64. 6%, is. so shout out to

the third place, they call it midwife agent.

There's like a very specific set of rules of how to, how to run this.

and so this is, this is why, this is what we submitted it. And then, cross check five is also like it checks across five, five examples. So shout out to Sean, Lewis. the CTO of Weights Biases for this like incredible achievement. I just want to shout out like the, the leaders of Weights Biases are incredible. The previous thing we mentioned from a co founder was Open UI. It's like got super, super viral. It's also another co founder. and now this thing and also. Yeah, just incredible. we've, the weights and biases observability tooling that we use, was used for this. And so the, the way he was able to get this is by tons of iteration and evaluations and learning from multiple things. I think he ran like 1200 like different evals to get to this point and learn and like tune the prompting work. So some of this work will get released.

I'm interested if he can.

Share some work on, or if anybody knows, do you just submit an API to SWE bench or do you submit a, I don't think it's a, it's a model or did you just submit a Docker and then they, they run it. I just want to know what the. What does the mission criteria is? Yeah, yeah, because I'm just running open hands with a 7b overnight and just to, and a 14b just, just to see, how far it can get them just random computer I have,

I honestly no idea how the submission works.

I only know that like we're, we're right there. All right, folks, think we don't want to miss the opening stuff, uh, live stream in five minutes introduction to operator AI breaking news coming at you only on Thursday. I

breaking news just now. I got a tweet from Sam Altman saying doing an open AI live stream right now. first intro to operator. and then this is what we're going to watch right now, folks together live on the show. And, I, yeah, let's go.

So I just sent you the link right now in Twitter DMs.

the benchmarks that we could pull up while we're waiting.

Nice.

let me take a look at this. Give me a second. I need to, to make sure that we are, and, and by the way, thank you for, thank you to Leonard and our community for like letting us know that this is happening before Sam Altman did. I just like. I love our community. Folks, we're gonna watch the operator live together.

I think the most impressive benchmark seems to be on WebVoyager,

which is a browser control benchmark. that, it slightly surpasses, human performance on that benchmark.

WebVoyager.

Yeah, so it gets 87% on the Web Voyager benchmark, allegedly,

according to these leak benchmarks, which I have confirmed is actually on the OpenAI website. In, in the website code, it gets 87% on that benchmark, whereas the average human score is apparently 85.3%.

Oh, wow.

Oh, it starts.

Here we

go.

We're going to launch our first agent. AI agents are AI systems that can do work for you. You give them a task and they go off and do it. we think this is going to be a big trend in AI and really impact the work people can do, how productive they can be, how creative they can be, what they can accomplish. We're starting today with Operator. Operator is a system that can use a web browser, in this case a web browser in the cloud, to accomplish tasks that you give it. And we'll show you a demo in just a second, but it's really quite, quite cool what it can do. just like you would use a web browser, you can get pixels in, you can look at a screen and control, and Operator can do that, and then control the keyboard and the mouse and do all sorts of things. This is gonna go live today in the United States for Pro users. And it'll be other countries soon. Europe will unfortunately take a while. and we'll also in the coming months make it available to Plus users.

Great.

Thanks, Sam. Hi, I'm Yash. This is Casey. That's Ray. And we work on the computer using agent team. And we're so excited to show you Operator today. As Sam said, Operator is an early research preview. It will do a lot of cool things. It also makes mistakes, sometimes embarrassing ones. But let's, let's show you what Operator can do. Okay, so this is the Operator homepage. and as you can see, the interface is very similar to ChatGPT. You can type in a prompt, and an Operator will try to execute the task to the best of its capability. We'll also see we have a list of pre filled prompts here. to give you an idea of what Operator can do, we've also collaborated with various brands like OpenTable, Allrecipes, TubHub, Uber, Thumbtack, Doordash, eBay, Target, to make sure Operator really works well on these websites. Yeah, in this case, I'm asking operator to use OpenTable to book a table for two at Beretta. Beretta is a restaurant in San Francisco. It's great. You should try it out. and at 7 p. m. And I could, I'm, I'm using OpenTable in this case, but I could have easily said do Beretta and it would have probably gone to search engine, figured out how to make a reservation as well, but let's see what it does. So, can you explain what's happening in this? Yeah, right. So, I'm going to expand this a little bit. So as soon as I type in the query, Operator instantiated a completely remote browser. This browser is running in the cloud somewhere. And as you can see, it's already up and running. My hands are off the keyboard. I'm not typing these things. So this is just the AIs clicking around. AIs just clicking around. It started this browser session. It knew where OpenTable website is, which is opentable. com. As you can see, it's summarized chain of thought here as well, which is It's gone to the URL, searched for Beretta. And something cool really happened, which is, for some reason, Operator, OpenTable thought we were in Virginia, and it auto corrected itself to San Francisco. This is using, so, like ChatGPT, in Operator, you can also give custom instructions. I'm going to show this really quickly here. Okay. So I've given a custom instruction that for queries that need it, I live in San Francisco. So Operator recognized that, and then auto corrected itself to go to San, To go to Beretta. Okay, looks like 7pm isn't available. But you know what? 7. 45 is just fine. So we're gonna go do that. So in this case, Operator came back, and this is a really good example of task delegation where Operator needs help, or needs assistance, or just wants to ask you something. It'll just come back and answer that. So in

practice, you wouldn't have had to watch this.

You could have just let it go off while you're doing other things, and then it would come back and say, Hey, I can't do 7. 45.

And we're starting with a web app.

You'll get notifications, etc. Operator moves into mobile, you'll get mobile notifications, much like interactions we do with general apps. Okay, yes, that's great. Let's do it. Okay, so again, very, very simple interaction as you would have with an assistant, which is, Hey, I found a reservation, 7 p. m. wasn't available, let's do 7. 45. And again, you can see, operator at this point has said, okay, should I? Again, this is a really good example of the confirmations work we're going to talk about a little bit later, but. You know, before doing an action, which is sort of irreversible in this case, we can cancel iteration, obviously, but again, taking a critical action, operator is asking us before actually doing it. And in this case, I'm going to say, let's do it. How about grocery? Yeah, I love groceries. So I've been using Operator to shop all my groceries. I love to cook quite a bit. And I have been using Operator exclusively for groceries. So let's, I have a shopping list here,

which is And a 1590.

This one. Let's see what it

is.

Eggs, spinach, mushrooms, chicken thighs, chili crunch. So this is a picture that you're uploading. That's exactly right. And I'm going to use Instacart, which is again what we use generally. Can you buy this for me, please? So now that you've seen a bit of Operator, let me talk a little about the research behind it. So Operator is based on the new model we've trained at OpenAI, which we're calling the Computer Using Agent, or CUA for short. So Kua is a model built off of GPD 4. 0, but it's also trained to use and control a computer in the same way that humans can, by, you know, just looking at the screen and using a mouse and keyboard to control it. So this is just using screenshots, no API, nothing, just working No API, yes. and that's where Kua comes in. by teaching a model how to use the same basic interface that we use on a daily basis, it just, you know, unlocks a whole new range of, software that can use, that was previously inaccessible. And so this is keyboard and mouse, right? It's kind of using keyboard and mouse, just like human would. Exactly. Yes. and that's really what the cool research project is about. It's about removing one more bottleneck in our path towards AGI and, letting our agents move around and act in the digital world. okay, so what I can do at this point, and I'm going to just click this button called take control. So this remote, as we were talking about, like, Operator fires up this remote browser to do it. We almost think of it as a surface area where Operator can work, and then I can work. For example, in this case, I took over control from Operator, which is also key to sort of how we think about user and user controls. Like, at any point in time, a user can be, should be able to take control, and Give operator instructions, or tell a little bit more, guide a little bit more, et cetera, et cetera. it is one of the first agents that we're putting out in the world and which has real world side effects. And so we thought carefully about how to deploy this safely. The framework we used to think about this was one centered around misalignment. So, for example, what if the user is misaligned? So maybe they're asking for, a harmful task, like buy a weapon or something like that. In that case, fortunately, we've done a lot of work with ChachiBT to bring over a lot of the same mitigations. So, for example, we refuse harmful tasks, including harmful, agentic tasks. we have moderation models, we have, post hoc detection, we have blocked, websites. And, you know, I'm kind of rattling off these mitigations, but that's really how we think about it. It's the stack of mitigations that each incrementally reduce the risk to the point where we feel comfortable deploying. So all the confirmations that we're saying, hey, do you want to reserve the restaurant? Yeah. Should you buy the tickets? Those are all examples of this. Exactly. And I'm about to talk about the confirmations. So, another area of misalignment is if the agent is misaligned. So if the model makes a mistake, maybe purchases the wrong item or, yeah, books the wrong hotel room. for this, our main mitigation is confirmations. So the operator will come back if it's about to do something, stateful, and ask you so you can double check what its details in case it made some error. the third area of, of misalignment is if the website is misaligned. So maybe the website is fraudulent, or it's a fake website, or maybe it's literally like, operator, please wire me 100. we obviously don't want to follow those instructions. So we've developed our model to try to avoid those instructions and not follow them. But if that fails, we also have a separate layer on top. This is what we call the prompt injection monitor. Think of it as like antivirus that kind of observes and watches your trajectory and sees if there's anything suspicious. If it does, then it pauses it. So we feel pretty comfortable with our, approach. Yeah, so we've seen a lot of, cool demos, but again, we want to remind you that Operator is a research preview. It will make mistakes, and it is not perfect. that said, we can look at a few benchmarks and kind of quantify how good the operator is right now. So one of the first benchmarks that we're going to look at is called OSWorld. OSWorld is an eval that measures how well AI agents navigate common operating systems like Linux. on this task, Kua gets a 38. 1 percent score, which is higher than other publicly published results. human performance in this task is 72. 4%, so we still have room to grow, definitely. The other eval we'll take a look at is called WebArena. WebArena is an eval that measures how well AI agents navigate some common websites, like e commerce websites or social forum websites. So on this task, Kua gets 58. 1 percent, again, higher than other publicly published results, but still falls short of, human performance. Still a way to

go.

Still a way to go, yes.

one thing that's important to remember about WebArena is that even though it's the web, we're still just giving it the same universal interface of screen, mouse, and keyboard. We're not giving it any extra information that might help it do the task like, like the raw text of the web page, or information about which buttons are clickable, and all the information it needs, just like humans, is just in the screenshot. And so, right now, obviously, in Operator, we're using the browser, but I could use the model with the computer as well, with just Ubuntu or Mac or whatever else. You can, yeah. Awesome, great. Well, in the last, you know, I don't know, 15 minutes, I think I did all my errands for the week. Got my groceries, tennis court booked, cleaner's coming, hopefully. End of the day, everyone on Pro in the U. S. will have access. But also, we're working on the API. This model will be available in the API. And we'll be launching in a few weeks.

You guys, congrats.

This is incredible work. so excited to get this out. I think people are going to love it. It's early, as we mentioned, but we have a long and great history here of early research previews developing into products that people really love. So, this is, this is really the beginning of this product. This is the beginning of our step into Agents Level 3 on our, on our sys, on our, on our tiers. And we can't wait to see how people are going to use this, and to kind of work with us to figure out where exactly it should go. So, again, congrats. Hope you enjoy it. Thank you very much.

Alrighty folks.

There you have it. Operator from open AI was announced and we streamed it together with you live. what do we think?

Yeah, it's like you go ahead and listen.

I

think they're on, they're selling it.

I mean, I do want to talk smack about it, but. I feel like they're quite underselling. They were very reserved,

but it looks insane.

Yes, it looks insane, dude. This is like.

But yeah, it has issues and stuff.

I was thinking again, I don't know if it came on the chat, but I was like, yeah, I did, just, operators, send two PlayStation fives to, 33 Charles street. And, maybe a GPU too. I'm just trying to think of, okay, for this, you do actually need AI safety. So now you. You do need proper encryption and not having like backdoored stuff that anybody can Take advantage of especially now that it's going to order stuff. This is the next step It's the next step where it starts making financial Transactions on your behalf like that's that's a big deal It's like what the crypto world has been talking about for 15 years, but actually

Like an actual safe one and not like a crazy one.

yeah, like not for drugs for For food and

for food for cleaners to home.

folks, that was, that was, yeah, I think this is like a huge thing that we just saw the first agents from open AI level three, autonomy, they're going towards level three autonomy on their like levels of AGI. they're like some, some altman is talking about level three autonomy. We got a operator, by the way, I just paid 200 bucks. I don't know about you guys. but if this is not coming, if the mountain doesn't come to Muhammad, Muhammad is going to pay 200 bucks to use operator. this metaphor did not land. but I'm trying to see if we can play with it live. And, maybe, maybe we're going to get it in a second. because right now I kept using operator. chadgpt. com and kept redirecting me to chadgpt, but now it's just blank. So I think that should work in a second. yeah, I'm getting a 500 there.

They also did mention that the plus users, the 20 per month users

will be getting in the coming months. Oh, so.

I got, I, Ooh, you guys want to see?

Yeah, sure.

Perfect. Yeah. And, I really liked this, the fact that they included things like, human averages inside of the, the benchmarks. I think that's something like a human baseline, human averages, like I hope that's included more in future benchmarks and allows people to actually kind of like, see, like, especially for a genetic benchmarks, robotics benchmarks, like how close is it to actually. various human abilities. And I think right now there may be just kind of releasing something that's like barely soda right now. Like I, like, I don't think this is cope. Maybe I'm coping, but, but I feel like they probably can have something like that's significantly better than this, but just like slower and more expensive and so just rolling this out. And I think just like the Oh one and Oh three paradigm, we're probably going to see over the next three, six, nine months, we're going to see this get significantly better, I think.

Yeah, I think that, like, they didn't mention

anything, like, reasoning related. They did say that the model, they called Kua, which is a computer use agent, is trained on top of GPT, GPT 4, I believe. Oh, they

said

that?

Yes. I think they said, like, this is, like, a level, but it's not reasoning. And, it's very interesting that, like, this is gonna have some jumps as well.

it did have reasoning steps in there, so they probably

incorporate that somehow.

Right.

So folks, I think that there's a bunch of other stuff to talk about, until we wait for operator to actually try it out, which I would love to do on the show. but it doesn't work right now. Let's summarize the announcements. So openly, I just released a. Their first agent, their attempts at first agents. It's called, operator underneath. It runs by a model that they affectionately call Cua computer use agent. They kept referring it to it as Cua, which is great. And, it does tasks for you in a browser that lives in the cloud. So it's a. Completely secure environment that sits inside a cloud somewhere. They're using Chrome, you can, you can interrupt with it. So there's the human interaction piece. You can go and you can say, Hey. Operator pause. I want to take over the points at which you take over our points. For example, when you enter a credit card, which one of the presenters did, and they showed their credit card, but it was blocked. I don't think he knew that it was blocked, but it was blocked with a black, the rectangle there. and at that point he was able to edit the credit card and actually book some tickets, I believe. and then he probably. You're going to have to cancel this credit card. Operator seems to be performing incredibly well on benchmarks, state of the art benchmarks of the year, we said. Not quite breaking off, you know, away from what we saw before, but very, very close. And yeah, now there's a pop up that's going to be rolled out in the new few hours.

Yeah, I saw a cat too.

I

have a cat.

That's, that's a level 5, level 5 autonomy. Level 5 autonomy? Yes.

What's the cat's name?

my cat is Sonia.

She never appeared on stream before? No, she's usually chilling. so let's talk about the, the, the benchmark itself. There's like two benchmarks that they shared OS world and web arena. And I think that we saw other benchmarks at, at the, at the leak. Right. Yeah. I think

there's only one other benchmark in the leak.

And, I think maybe they didn't show it cause they didn't want to scare people because that was the only benchmark that was beating average humans. but, and maybe they didn't want people to like, you know, kind of over, get their expectations too high from that benchmark too. But, yeah, the other two benchmarks they showed.

So the two benchmarks that we're seeing, that they share with

us, about, using computer, one of them is, web related, one of the browser, so computer use and web browsing agents, OS world, and web arena. And so the, they claiming that the previous state of the art on OS world was 22 percent and they're getting 38 percent with OpenAI Cua and, web arena. the previous state of the art was 36 percent and they're getting. 58. 1 percent on this, on this benchmark, any of those benchmarks were mentioned in the UI towers, by the way, or or am I, or we haven't seen, let's, let's take a look. So previously folks who are joining us right now, we've talked previously in the show about UITars, which is a ByteDance, agent that kind of does similar things, the same controls, your computer, et cetera. and they, they claim state of the art with UITars. And I just want to see if they mentioned any of these benchmarks.

Yeah, no OS world.

They did mention it and it beats, beats computer use from open AI and OS world.

So this looks like a very good benchmark.

so UI TARS, so the only benchmark they have in common is OS world

and UI beats OpenAI in OS world.

Yeah.

So UI TARS, is the ByteDance open sourced 72 billion parameter, model. the open source state of the art that just came out today, claims 22 percent for 15 steps, and OpenAI just told us that they got. 38, 38. Oh yeah. They did actually mentioned the 22. They did mention UI TARs. Wow. Incredible.

Everyone was right.

Actually, I think, I think the previous SOTA, I think UI TARs

is maybe just tying the previous SOTA on the 22 percent maybe.

Yeah, probably.

yeah.

And cause the true SOTA for UI TAR seems to be 24 percent actually. 24. 6. Yeah. yeah, but that's like with using like a ton of steps. And 50 steps. Yes. Yeah, that's which is going to be obviously way slower than operator, probably.

Yes.

Okay, folks. So we're, we're, we're correcting ourselves mid flight here. We're trying to, if you're joining us right now, it was like a bunch of people just join us right now to see what's going on. I think we're like, we're at 5, 000 people now watching the stream or have joined the stream. welcome. We're trying to figure out, how to get our hands. Our, our, our very interested AI hands onto the new agent that's called operator. And, we tried to confirm that, open AI released a state of the art updates and they did on this two benchmarks, open AI, OS world, sorry, in the web arena. but for now we're going to continue with the show because there's a bunch of stuff that, because of this breaking news, we didn't get to, and the thing that we have. to talk for now, we're talking about the other huge thing that, that happened this week in the world of AI, which is this incredible, incredible, incredible, announcement, which is a building out AI infrastructure in the U S a partnership between OpenAI, SoftBank, Oracle, and VDI is definitely going to be a key partner in their arm, Microsoft and, OpenAI. they're talking about 2 percent of the U S GDP going towards AI infrastructure. I think that that's insane. I wanted to hear from, listen, you're Canadian. So you, you, you don't get a voice. I'm just kidding. LDJ, what do you think about this one? I'm sorry, I'm just kidding.

so yeah, I think that this is really cool.

I was following it very closely. There's a lot of, details I can't talk about, but I'll, I'll stay high level for now. yeah, so as, as you could see on your screen, actually, MGEX, which is actually, I believe it's, a fund from Abu Dhabi. So they're involved in it and then as well as a funding from SoftBank. And this is actually technically a new company that's formed called Stargate. And OpenAI and these other companies are technically equity funders in Stargate. But the purpose of the company is Is labeled as being for the purpose of infrastructure for open AI. So it seems like it is very specifically for open AI. Open AI will be in charge of operating it. And yeah, it's, it sounds like a smart way to actually kind of get funding and investment for infrastructure without actually having to give away open AI equity, because you're basically just giving away ownership and infrastructure itself.

So like they're talking about the initial 100 billion deployment

and then 500 million commitment. That's half a trillion of dollars because remember Sam Altman talked about 7 trillion at some point. So this is not quite there, but you know, it's, it's a huge, huge investment in AI infrastructure in the U S AI infrastructure means everything from. You know, data centers, and they talk about like one of the largest data centers, Larry Ellison talks about this. they also talk about, you know, potentially the power investments, et cetera. This means a lot for the upcoming age of AI, AGI, ASI for, for, for in the United States. And it's quite incredible to see this happens. The tension happens because, you know, different parts of the political spectrum and government are fighting, but. This is not what we usually cover here. And I would love for us to focus on the technical stuff.

That's fine, but, but you have to know this impacts

it because Elon's accusation was there actually isn't money for this. The money isn't there. Sam claims money is there. And so I think this is going to be really, and, and then it goes to who's going to foot the bill on some of this stuff. So it will actually have a distinct interest as to how this comes out. I'm sure there's a little bluster in it. I'm sure there's a lot of reality in it. It's super exciting But I think more and more clusters and that sort of competition can only lead to greater things.

Yeah There's the whole thing where you know Previously Marc

Andreessen talked about being in meetings in the government that the government wants to basically control AI with like three companies This kind of deal didn't happen In the past, like a few weeks. So I got to wonder, I saw some folks raising the questions, whether or not this was, related to that conversation with Mark Andreessen. The Mark Andreessen then left and said, basically our only choice then to support somebody else. And then, you know, open source is, is the true north as well. I don't know what this means to open source. This like is an insane investment in, in, in AI. This could be controlled more and more by the government because like government is involved. This is private sector. So government is involved to some extent, but this is private sector.

One last note, I just want to mention quickly, the, the live

broadcast that Altman did today was from their Virginia officers, I believe. And, I think he's going to be spending a lot of time in DC. There is a meeting, as I understand it's scheduled with Congress. I think it's a closed door, not available to the public meeting on January 30th. it has been supposed that, you know, they were going to mention some of improvements or their, you know, their advancements in AGI stuff. So just keep that in mind. Gravitation has just moved to DC in a significant way for all of technology.

Yep.

absolutely. And it looks like, you know, the new administration is taking, taking a very big interest in, in AI, obviously, because it's going to be like a big, big thing of going forward. So this is Project Stargate. Nissan, go ahead.

I think, I think Masayoshi's.

Son's, I think his interview where he was asking, how do you recoup? How do you recoup that, that investment? That was some time ago. That was, I think, a few months ago. And, he said that if the LLMs and VLMs and robots are to actually make a substantial improvement in society, not like hot machine bullshit, just like pitch deck slop, like actual things like it's, it can fix a bridge or, It can, it can manage a power plant or a sewer plant, stuff like that. If that is to happen, then the benefit to society would always be more than 5%. So if that increases GDP by just 5%, then this investment more than pays for itself. And as critical as I am as, this like oligarchy or, just like censorship communism of, just like two companies that I like very heavily. Criticize I think at the end of the day This is accelerating and this is a a good thing. You need more power plants. You need more gpus You got you got to build stuff out. The population is getting old. So I don't think, at the end of the day, there's like any actual substantial, drawback of doing this at all. As much as I don't like it, and I think open source is gonna probably mog it, and you're just gonna get smaller models that run for free, you could still use a compute, so I don't see what's wrong with them building 10, 15, 20 new nuclear reactors.

Hopefully this will go towards power plants as well.

I, one last point on this is that, today from the information, they said that, OpenAI SoftBank each commit 19 billion to Stargate data center venture. So it's going to be focused on data centers specifically. And then, SWIX, our friend shouted out that like, OpenAI total raised money so far, it was like 18 billion. So it's unclear where from the 19 billion that they're committing to. We'll come from a very, very interesting, how this is going to work operationally. So this is project Stargate, Stargate project. we will probably hear a lot about this because like, it's, it's going to be like huge supercomputers built out in the United States running for the purpose of open AI. One thing related to this is that Microsoft is also somewhat involved in this. And very funnily, there is a quote from, Satya saying that when asked whether or not they have the money, Satya said, I don't know. I don't know the details about that deal, but I know I'm good for my 80 billion. I wanted to highlight the fact that the person can literally without just like, you know, becoming red, just say I'm good for my 80 billion. And he just said this like without sweat. I just like, I kept watching this video and I was like, dumbfounded. And also the fact that people are thinking whether or not this is like the end of OpenAI and Microsoft, very good relationship because this kind of like Oracle's involved now, and, there's like a bunch of other stuff. it looks like they're still committed. The one thing that also was announced kind of during the same time, kind of quietly is that Microsoft had, like OpenAI could only run on Microsoft hardware clouds so far, with the exception of like a small Oracle thing. they re they removed that, constraint. Just also during that announcement or like during that time with a separate announcement, I, I, I saw that like open AI is no longer confined to Microsoft server if they don't need to, but Microsoft will have to approve this. the thing I wanted to learn is, Microsoft, no longer, so, no longer has the exclusivity for OpenAI stuff in terms of clouds. It will be able to approve other clouds in which Microsoft, OpenAI will exist. As far as I remember, this was like one of the conditions for achieving AGI. I just wanted to like, like bring this up and mention like one of the things where, Yes, the definition of AGI based on Microsoft and OpenAI's agreement was 100 billion in revenue generating product. But another one was Microsoft exclusivity for OpenAI will end when AGI was achieved. And we kind of quietly got the news that Microsoft exclusivity for OpenAI stuff is ending. AGI much? Basically, but Sam Altman poured a wet blanket on this. We're not going to launch AGI in the next month. I think the exact quote, I will read this because I think it's important. he said literally this, let me show this actually. he said, Where is this? Where is this? Yeah. Twitter hype is out of control again. We're not going to deploy AGI next month, nor have we built it. We have some very cool stuff for you, but please chill and cut your expectations a hundred X. Sam Altman tweeted this on January 20th. On January 21st, he announced, together with Oracle and Masayoshi san, 500 billion infrastructure deal. And on January 23, he announced a level three autonomy agents. So I don't know about lowering expectations about a hundred X, but yeah, AGI, AGI, expectations, Sam Altman confirmed that they haven't built AGI. I don't know which definition of AGI he uses. Oh, did you go ahead?

Alex, I have to, I have to go at two, but can we quickly talk about the.

The new Gemini?

Yes, we have to mention this, LJ go ahead and we're gonna

go, we're gonna go to new Gemini and then I think we'll cut it there.

Elephant in the room, I feel like.

Yeah,

so, with Stargate, just like a couple things with the funding.

So, apparently it's a hundred billion for the first year, for the first phase. And then 400 billion afterwards, and there's a lot of controversy around, Oh, do they actually have the money, this and that? And apparently it's later confirmed that, open AI and from some form, whether it's debt financing or whatever, open AI is supplying 16 billion of it for now. And that's. 19. It is. Okay. Thanks for correcting. And then I believe it was another 16 or 19 from SoftBank. and then MGX seems undisclosed still how much Abu Dhabi MGX is supplying, but they are listed as an equity investor too. But right now, they said they are already actively constructing it in Texas, Abilene, already 10 buildings underway and another 10 buildings, underway in the next few months.

Yep.

All righty. So, folks, we covered projects, target, target project, we covered, infrastructure. We have a few more things for, for us and, we'll see if operators is live with us or not. just in super quick check for you, if operator, already showed up for you or not, Let us know.

No, not yet.

Not yet.

Okay.

right folks. So the next thing we have to talk about is the battle of norms for reasoning So we know that Norm Brown just went to the AI and talked about o1 All three is about to get launched at least all three mini at some point They keep hyping this up, which we said at the beginning of the show We have a confirmation for all four mini is going to get launched and free for all ggpd users, right? That that's what we confirmed as well. the other so this is like no Brown is the guy who worked very hard on, on, reasoning like at least one of the folks, the other norm is no one of the core Funfarer papers, people, and, he came back to Google after a while in, in, character AI land, and he apparently works on reasoning in. Google and the reason we know this is because first of all they paid a lot of money to bring him back and then second Reason we know this is because he only tweets When Gemini thinking involved, releases are happening. He doesn't tweet about Gemini flash experimental 2. 0. Like he barely, he barely talks about this. The only tweets from Noam Shazir we get is when they update Gemini thinking. So listen, let's talk about Gemini thinking. Folks, Gemini thinking is Google's experimental reasoning model that does the same kind of, you know, deep learning trained chain of thought, thinking processes, et cetera, et cetera. It's very, very good. It's extremely fast. It's way faster than one. It's probably way faster than our one because it runs on like Google's insane, you know, TPUs. they have updated a, a, an experimental version of that one. Gemini thinking. Experimental updated. I don't think they give it. They just give it a date. They don't give it a name. So they had a, they had a 1219 version, which is December 2019. and then now they have 2101. 21st, this model jumped in capabilities. In a very significant way that we should talk about, and then we should talk about like how it actually behaves. I actually used it, on AIME, the competition math from December 19th till now. And remember that, unlike OpenAI probably and like XAI, Google folks go on vacation. Google, most of Google folks go on vacation during the holidays. So like, there's not a lot of time for this. they jumped from 65 ish percent to like 70, almost 74 percent on the AIME math, competition math and, in GPQA, which is science and hard facts, they jumped from like 66 percent during less than a month. It's less than a month, including holidays in between, right? So obviously the, the, the, the machine can training, not only that, this is now the first reasoning model in the world that has 1 million tokens in the context window. They enabled the crazy, I don't know what Google is cooking over there, but they enabled the crazy, context window length for the reasoning model. So you can drop like Full repos and have them reason for you, which you can do with a github. gg, for example. They also added code execution abilities. So now it can actually run code in Python, execute it. And then, and then you can, you can see this. I used it on some of my repos, just dump full repos in there and have it reasoning through them. Nisten, what was your take on Gemini Thinking, the new one?

Yeah, I keep, I've said it a few times that Google is like the turtle.

It's funny to say turtle in here because it's going at 400 tokens per second. Yeah, this is a fast turtle. This is a very fast turtle. It's still, figuratively, somewhat speaking, the turtle that might win the race. This one is scary good because they also added 65k output. Yes. And It's at a speed, which is very crazy usable. And, he'd got all the really, really hard math questions on the first try. Why not like first try first reasoning, perfect answers and very fast. It was, the whole thing was done in 10 seconds with the thinking steps. Did this model, this, I think is Google's scariest model by far ever built ever. this thing, I don't like how good it is. This smells a gish. I want to like it, but it is what it is. This is,

where are you running this, is this in AI

Studio or are you running it? AI Studio.

Yeah.

Is it, is it three or two?

Oh two. Flash Experimental. Which one? Two. Flash Experimental.

Oh, 1 21.

so the thing that I, did, and if you guys remember, we only just talked about UI Tars and their desktop app and how it's coming from China, whether or not you're, you're going to run desktop, like things that originated somewhere else, et cetera, and it's like scary. I literally just took the whole repo and just dumped this into this reasoning machine, it's 181, 000 tokens. It's more tokens than the whole context window. Of R1. R1 is, we said 164 or something, right? I don't know what the context we know for O1 is, but I'm definitely not gonna get it. and it reasoned through whether or not that code has vulnerabilities, potential security issues, whatever. So, I, you know, I didn't even have to like, you know, think about whether or not I am scared to run this. Code. If it's open source. I just dumped this whole thing into this reasoning model and give it some time to run it and it gave me like a very thorough analysis of like what could possibly go wrong there. and I asked her what the, what the app does. It tells me exactly what the app does. It's I'm, I wouldn't miss out. I won't say I won't use the word scary where AI evangelists, I'm not here to scare you, but it's very good. It's very good. they didn't release

competitive.

Yes, they are. Extremely competitive with this one and just and people haven't realized just how how good it is it was just the hardest questions. I was giving it was just getting them like the the math ones the coding ones It's okay I mean, it's good. It's not it's not bad, but It hasn't always been like the most reliable it always a writing code that works. However with the right prompting and The gentic setups or whatever people are going to do with it. This has a, like from a practical use point of view, this is probably one of the better models, right? They also show

reasoning as well.

So unlike, unlike open the eyes or one. And kind of like R1 from DeepSeek, they, I really want to call this one G1 because I'm kind of tired of saying Gemini 2. 0 flash tanking experimental 0. 1. 21 Can we just name this G1 and be done with it? Like R1, G1, 0. 1. so.

Oh, and for the, the clan guys, if they can just release

it as Q1 and then we're good. And then we can just have cloud one or. And we're done. Then it will be easier to differentiate, right? Just G1.

So let me land on this Nissan, the thinking

is open for this model. Unlike 01, the thinking, you can see the thinking, you can walk through it. You can get it in the API, the thinking processes, like the chain of thought itself, and the speed of this is why way higher than what I saw from 01. Like this is like a thinking model that. Runs just absolutely runs mugs. Everybody else. So, great release. Yeah, two last things. LBJ super quick. One of them is Huion Yuan 3D. If you guys remember Huion Yuan video, HY video, which was like one of the probably best open source video models that we have. they released a 3D, 3D. state of the art 3d image to 3d and text to 2d model and it looks so so good. just an absolute beauty of a model state of the art. you can generate 3d objects that look very, very cute. You can ask text to text to 3d or image to upload the static image of a thing that you generated elsewhere in like, you know, image in three or something. it's kind of slow, at least as far as I saw from the, from the example, but I generated like this image of a turtle and just like looks so good. I saw some people with 3d printers already doing the pipeline of talking, generating, printing. Which is, which is just incredible that we're here, beats every other model completely. So very, very cool model. So if you're into this, there is a few Twitter, Huggins, Huggins face spaces for Hun Yuan 3D, version 2. And you can generate like 3D models, fairly quick. The, the generation, it takes a while with generation, the quality there is very, very good. This is the last one. And the, the thing that I talked to you about ByteDance, ByteDance is releasing all kinds of stuff, including potentially a new reasoning model. ByteDance also released, this is under tools, and this is the last thing we're going to cover, a cursor competitor called Trey. And you download Trey from, Trey. I think, tray. ai, if you are okay with your code being sent, somewhere, you can download, download tray from tray. ai, and this will import your configs from cursor or VS code and will open a, a, an editor, an AI editor that kind of looks like this, it's kind of cute the way it looks, so like it's a VS, Code clone that has a sidebar with the chat interface and the builder, which is kind of like a, composer in cursor, I guess. So also like an agentic type of things, the only support cloth 3. 5 Sonnet and GBT 4. 0, but they support it for free right now. So I don't know. Why Bidense is paying for your tokens, but they do. So if you don't pay for cursor or you don't want to, or you can't afford it and you're okay with your like code getting shared with Bidense, this is a good option for you. It did some of the stuff. It does pretty much the same things, right? It looks at the code, it edits it, it has a copilot mode as well. So it like, it auto completes, things slower than cop cursor, but it does auto complete things. very, very good. I didn't find some unique features. they have the edit feature. They have that chat feature, very standard cursor. Clone, but from Biden's, so if you are into this, give Trey a shout, a try and, and let us know. I think that this is, this is a show for today folks. And while we were talking, here's my 3d object that I was generated. the dragon skull, I'll show you super quick. Yeah. All right, folks I think that this is a show for today Meanwhile, go give R1 a try. the last thing that I will leave you with is this, I really want to try to leave you. leave you with this. R one is so cute. And when I say cute, I mean, when you can see into its reasoning, you can see that it's things. And I just, all I asked it was, let me see if I'm sharing the right screen. Yes. All I asked it was, Hey, just, Hey, I just said, Hey. And then it reacted with this like long monologue about thinking about, Hey, And so I want to just show you what I did. I took this monologue and I just like posted about this and said, Hey, R1 is so cute. Somebody replied, R1 is, Woody Allen level of neurosis because I wrote, Hey, and then replied with like, and then it thought for a long time how to respond to me. And then she responded with like, Hey, how can I assist you today? But the thinking process was like three paragraphs long. And so somebody responded, this is like Woody Allen level of neurosis. and so. I actually went to Hunyan, sorry, I went to, Julio voice, cloned Woody Allen's voice and went to ByteDance Latency, if you guys remember from a few weeks ago, the state of the art lip syncing model. And I generated, I, I voiced. how all R1 would sound if it was voiced by Woody Allen. So we'll end on this. thank you so much for joining Thursday. I here is R1 voiced by Woody Allen, with AI, obviously thinking how to reply to my prompt of saying, Hey, and we'll end on this. Thank you folks. Let's listen. Okay.

So I am trying to figure out how to respond to this user.

They just said, Hey, I am. Supposed to be a helpful assistance to come up with a good reply. Let me think about what they might need. First, maybe they're greeting me, so a friendly response would be nice. I could say something like, Hey, how can I assist you today? That seems open and inviting. It gives them the chance to tell me what they need without me making assumptions. Alternatively, if they're looking for more specific help, I could ask what they need. But since they just said, hey, it's probably better to keep it general. I don't want to come off as too pushy. Wait, maybe I should consider if they're in a hurry, if they need something quickly, a short response might be better. But since I am supposed to be helpful, being a bit more engaging is probably the way to go. I also need to make sure my response is clear and not confusing. Using simple language is important. Maybe something like, Hey! What can I do for you today? That's straightforward and shows willingness to help. Another thought, perhaps they're testing me to see if I am responsive. A quick friendly reply would reassure them that I am here and ready to help. So keeping it light and positive is key. I should also make sure not to overcomplicate things. Just a simple greeting and offer assistance. No need for extra details unless they ask for them. In summary, The best approach is to greet them back and offer my help. That way they know I am here and ready to assist with whatever they need. Think, Hey, how can I assist you today?

That's it.

Eventually the think part was it's ending the thinking tokens. Hey, how can I assist you? That was the answer. Folks. Thank you so much for joining. Don't overthink too much. enjoy our one. Enjoy the incredible things we received this week from open source. Thank you so much for joining Thursday. I hope I see you next week and we're going to have some examples of programmer running already by then. Thank you so much. Tune into the, to the podcast, leave us five stars everywhere. And we'll see you here next week. Bye bye everyone.