📆 ThursdAI - Sep 18 - Gpt-5-Codex, OAI wins ICPC, Reve, ARC-AGI SOTA Interview, Meta AI Glasses & more AI news

All righty, let's go.

Welcome everyone to Thursday I for September 18th. My name's Alex. Ow. I'm in Ai Avengers with Weights, & Biases from CoreWeave with me, my trusted cohost, Wolfram. What's up Wolfram?

Hi man.

You remind me that I need to get some glasses.

Oh, yeah, yeah.

We're definitely gonna chat about this. I'm sitting here wearing my, meta Houston and, yeah, I just wanna, say to everyone, welcome to the stream today. such a busy week, man. Such a busy week. it feels like the slump of the summer is over and we're well into the fall craziness where people supposed to lock in and bring a lot of ai, updates. But, definitely a lot on my plate. we'll wait for some other hosts, but I will say that very interestingly, we have a great interview today. You guys should stick around for, we're chatting with Jeremy Berman at the end of the second hour. Jeremy has just broke the top score on R-K-G-I-R-K-G-I is this Kind of, multi model test for ai. he broke the top score on first one and he used GR for it. And then, space Uncle Elon reposted everything that he did. so that's, very interesting and we will chat with him about, why grog specifically and how he did it, et cetera. So we're definitely gonna have a very interesting conversation there. Besides this, it's been a whole week full of very interesting things. not withstanding the Meta Connect presentation from yesterday with the AI glasses, which we've talked about before, which I'm donning right now. but I don't have the newest generation of obviously, so we're definitely gonna chat about that as well. and also always a great week when there's a new model from OpenAI. So we got GBT five Codex, which we're gonna also mention, probably more and more stuff.

open source stuff as well.

And then they didn't talk about it for a day and then it was

leaked and only today like ta yeah. what is your, go-to AI update for this week? What is the one thing that we absolutely must talk about?

The biggest highlight of the classes.

I've been waiting for this for years now. I've been big into VR before AI even came out. So I, trust me, Zak, that he knows what he's doing and, uh, it takes longer than expected. Maybe I've even been a fan of the Google classes when they were announced a decade ago or even longer. So I was hoping for this stuff and now meta seems to be the first to really deliver something really great and, that is, the next generation of the interface after the mobile phone. That is my personal opinion I'm convinced of. So, yeah. that is the most exciting for me.

I have signed up already for a demo because you cannot just

buy them, you have to go get a demo and then buy them, apparently And, probably 'cause they wanna teach you the Kind of, the band control.

Oh no, not, no, that is el stuff.

what is it called?

think Neural band or something.

Band. Yeah. Neural band. Yeah. Uh, so folks who are listening and didn't tune in yesterday to the Meta Connect, the new glasses that they've announced have a screen. And the way to control the screen is with this Myo neural band. It's a company they bought like a decade ago, I think, that just, reads Mixtral electric signals from your wrist muscles, and then Kind of knows what you wanna do, which is I think incredible. It's like, you know, like an accessory. But we'll talk about all of this. I would say my highlight of the week was definitely Codex GPT Five Codex. It's a new, Finetune of GPT five specifically for coding. And, I think we've come full Circle because, I remember when the first coding Paradigm came out, OpenAI had a separate model. They had an autocomplete model, for I think it was GPT three before instruct version. And then they had the Codex. Model specific, fine tuned model for coding. Codex became a whole thing, became a CLI became an editor. It became, what is, oh, they did the hosted codex as well. so now there's another Codex model. And this one's great. And we're definitely gonna chat about this. Folks are switching back from cloud code to open the ice codex in droves, I would say. So a lot is happening, there. And I think it's time for us to maybe take off these glasses, for now and then get started with the TLDR because we're really gonna have to chat about like a ton of stuff. I'm ready Wolf. You ready? Ready. Let's, let's go. Let's go.

Yeah. All righty. Here it is, the TLDR, the section authors, the eye, where we just run through super quick everything that we found notable, and, not that much discussion, just like we're gonna run through. And then if you're interested to stick around with us and then, you know, hearing our commentary about this, you're more than welcome. But if you don't have the two hours to sit with us, you will be up to date. This is basically the purpose of T-T-L-D-R. Another purpose is structure. Folks are asking me like, why, et cetera. this helps us structure the show for you in such a way where we know what to talk about. structure is fairly rigid. We love open source. Then we talk about big companies and LMS and APIs. There's a lot of news there. this week's buzz is a corner where I talk about everything. Weights, &, Biases, and core wheel related. This week is gonna be awesome with some examples. I'm looking forward for that. Then we have vision, video, voice, and audio, art and diffusion, and 3D. but for now, let's just run through the open source stuff, So in open source, we have a company called Perceptron AI announced Isaac 0.1. It's a 2 billion parameter perceptive language model. What does this mean? It means you can ask this model to see things and ask it where the things are. And unlike, YOLO, for example, which is a dedicated, model for someone like this, this is a language model. And so we know, for example, that Gemini 2.5 can do this, at moderate length and moderate success. This model is a tiny model, 2 billion parameter, and it's really, really good at detecting like small things in pictures. You can ask it where something is, it will put a box around it. it's really fun to play with. They also have a cool demo. the folks at Tonge Deep Research, I think Tonge is also Alibaba. So Tonge released a something called deep research. A, I think 30 billion parameter, 3 billion active perimeter open source web agent that ca claims parity with OpenAIs, deep research on multiple tasks, including humanities. Last exam. Folks, This is crazy. the amount of times that we are here and we're telling you, Hey, there used to be this multi-hundred millions of dollars trained model from a huge company and now there's a 3 billion active parameter model that comes close on this, one specific task. Every time we come around this as like, I wanna stop. It's like, what the, what the actual, you know, deep research from Togi, 3 billion active parameter web agent. It comes close to deep research from OpenAI. So we definitely wanna chat and see how that happens from Togi.

Mistral released, two new models actually,

but only one is open source. magistrate Small though Magistral is their reasoning model. And they released an updated version, 25 0 9 or the version number 1.2 from them. it's Apache license and available on market phase. Of course it has, multimodality so it can work on text and images. It has a 15% improvement on market coding benchmarks, such as Amy and Life Code Bench. better at tool use. like web search, code operator and image generation and it should have clearer, more natural and better formatted response, I haven't used it myself yet, so I hope it doesn't do too many bullet points stuff maybe, but if they improved it, that is a good thing. And they have the magistrate medium 1.2 as well, which is only on the API for now

Yep.

Thank you Wolfram. and then we'll definitely chat about and see what's new there as well. Anything else that I missed on the open source stuff?

Did we cover this one that just blew up at the top of hugging face?

Oh, SRPO is, we got SRPO.

Okay. We got it in together. Yeah. Yeah. SRPO. we definitely must talk about you. You wanna just pick it up while we're here and just like keep from there. I literally just saw it. Oh, okay. Cool. So we have something about this. So, let's continue here in the AI Art Fusion. We'll go back. Uh, so Ian, which is Tencent, released something called, they've been on fire lately. Last week we talked about a Tencent, image, right? Yam, and then, there's tons of stuff. So this week they have something called SRPO. This is a shortcut for semantic relative preference optimization. this is a new fine tuning method, folks like this is like, they release the model and it's blowing up, but it's mostly about, reinforcement learning technique, for fine tuning, super quick diffusion model. Super quick, if you guys remember, like Lauras and different things. This is like a technique to Finetune. There's some examples of the model that they released. I think they fine tuned flux or something with only 10 minutes, and then less than 1500 images, and then significantly reduction of cost. And so there's a comparison thread will show you, but basically there's a new technique in the works from Tencent and also a new model, to go along with it to show that you can Finetune, diffusion models super quick. Tencent also released a third version of their Ion 3D and we can generate 3D images with a bunch of graphs. So definitely we'll chat about those. and while we're talking about 3D, we must show you a demo of World Labs Marble. We've chatted about World Labs. We chat about Dr. Fefe Lee, one of the pioneers of ai. world Labs released, examples. And also I got into the wait list of Marble. So also I was actually thinking about, if you have a origin image of the spaceship that you're sitting in, that we can maybe turn this into a 3D model and try that out live on the show. Oh yeah, good idea. So, if you guys remember we talked about Genie three before. This is a little bit different. Genie three generates Kind of like a playing world. they generate a, splat, glatt, that you can gen with ai. It's like super cool. and then ve have you guys heard about Reve Re Reve? No. Okay. there's a new, model in town, visual platform. It's called vy. one of the folks, Christian Rell, who was, a chief product, I think it's stability at some point. He, he's leading Kind of. The, the charge there. Vy is their new editing model. So folks like, this is like the third week in a row. we're getting close to state-of-the-art image editing AI tooling, I think the third week in a row, right? We got nano banana like a month ago, I wanna say, or maybe three weeks ago to a month ago. Then we got, SEAD Dream. the, the last Sead dream was at four, and now we're getting Revy. And, I planned the, the thumbnail for the show I created with Revy and I recorded it myself. And we definitely should play around with this, all of us on stage. because it's just, it's just incredible the, they have besides the model itself. They have a very innovative interface of creating ai, art. You can like drag objects around the tech object. It's super cool. I will, I have to show you, and then we're going back, and here and Yam you wanna, you wanna share about this?

sure.

So, uh, OpenAI gives us, GPT five Codex, which is a fine tuned model from GPT five, for coding. just like we had, the initial codex, but there are like five different things called Codex, but you get the point. We had another Codex, Finetune model specifically for agent decoding that is absolutely successful, by the way. And now we have an updated one, based on GPT five.

we talked to folks here, Codex is definitely

worth at least two sentences. We chatted with you guys and told you that many people are switching away from cloud code into Codex, the CLI tool or the extension. even before this new model, the world has changed and people like moving towards, and we had this discussion now, and now even more 'cause they have like a, you know, they're gunning out from a topic.

Mm-hmm.

the thing about this is that what is special about this model just from experience, is that it doesn't lose track. Even if the task is extremely long, it can go on for long and not lose track of the task. And you know, with itself, you just, you just look from a far and make sure that nothing breaks with Claude, you know, your computer might be destroyed in the process pretty fast and everyone knows that you are absolutely right. Kind of thing that Claude is doing this because, well, Claude's making mistakes that you correct it and everyone already saw that you are absolutely right from Claude anyway. absolutely ghost, right? It's absolutely great. G PT five Codex for sure.

many folks have tried it, enough so that OpenAIs said,

whoops, we're outta GPUs, we're gonna lower the speed for two x. And then they brought it up and then they gave everybody back the quota.

I think there, it has been a huge feature for Open.

I actually, because I had some other stuff noted, for instance, that, GBT five achieved, as I know the first model, superhuman coding ability with a perfect score in the I-I-C-P-C word finals. Have you seen that? So that is a premier college GA programming competition where the top university teams around the world, compete to solve complex algorithmic problems. Mm-hmm. And both Google and, open AI became finalists, but it was GBT five that achieved the perfect score of 1212, which is the best score and better than what,

1112 and the new unreleased reasoning model.

Yeah. At the, the 12 question, it

was a

special model for this, but it Achieved the perfect score, achieve

the perfect score, that than any human, where non-human achieved this before. I think one team and that person that works in open AI achieved the perfect score before, like a decade ago. and both of them, everybody says it's, it's impossible. We'll definitely get a chat about this. It's unbelievably

Unbelievably hard.

Yeah. It's not just another benchmark that's unbelievably hard.

We we're basically there and AI achieved superhuman ability

to code and, last year it wasn't able to even solve like two questions. It's insane. It's insane. let's, it's not a harness. GBT five, we're definitely gonna cover this.

And there was another big announcement about teen safety,

freedom, and privacy that will affect everybody who's using open eyes. GGBT app, Android. Because, if you've seen it, there have been issues where their Principles come in conflict because they want privacy and protection, user freedom, but at the same time, team safety and, we should definitely talk about this and what it means and the measures they are using. I think that is important for everybody.

they want privacy, but at the same time they

wanna get, IDs, for example. from teenagers to know that they're teenagers. Right. so there's like definitely, complications there.

detecting if you are a minor by how you use the ai, that could

also be very interesting, outcomes.

Yeah.

I also thought that they are offering user configurable

thinking time for the web chat, GBT five thinking because it's, I have been not using, been using it all that much because it takes so long. So for many questions, you just want a quick answer. this is also very welcome. Oh, you have it on the list. okay.

a shout out to the folks who are just gonna

listen to this on the podcast. I do a lot of work to prepare all this Wolf M just goes off top of his head and like, bang, bang, bang, bang, bang. All the items I have on there.

everything from Open since you went to Meta.

I think.

one last thing on the big companies and NPIs, Meta connect. So Meta's yearly developer conference and product announcement conference happened last night for some reason at like 5:00 PM it's weird 'cause like last year before this, when I got these awesome glasses that I lost, they had this, I, I was actually invited last year. This year, my benefactor at Meta probably shuffled a, change of scenery within meta and, the presentation was at like 5:00 PM probably due to Zach's, ability or whatever. and they talked about a bunch of ai. They talked about, the way that AI is gonna join our world. in the physical form they talk about new glasses that they announced. We definitely should cover all of this because it's really cool. if you guys are into technology in latest and greatest, like the new form factor is basically. And it's from Meta, not from Apple, and not from Google, not from everybody else. It's from Meta, which is quite incredible to see them finally deliver this, and yam the glasses that you usually wear to the stream. Then I have those with the camera. so, so that's, you know, like, you know, next time you can join while having the AI assistant in your face, on your face. and last but not least, RKGI. This is a very difficult, Kind of, like it's, I think it's in third generation now. test for whether or not AI can do outta distribution tasks with multimodality. RKGI, has been beat again, I guess like it's always getting tested. And we talked about, highest RKGI scores. All three was released, if you guys remember. And they topped the charts and like then test time, compute, blah, blah, blah, blah. so, there's a new state of the ART score on RKGI from, Jeremy Berman and Eric Pang and Jeremy's gonna come to the show. very much looking forward to Jeremy joining us and chatting about RKGI, state of the art and why he chose guac four, et cetera. I think that this is it in the big companies. philanthropic not only admitted already that they had bugs, they posted a very deep and thorough, analysis of which bugs affected, which things they reiterated, that they never on purpose degrade the quality. And it feels like a little too little, too late, a little bit because like people are switching and after like three weeks of, nobody's listening to us on a topic, they did listen, but it feels like a little bit, too late. But they did fix this. And I did see people say the cloud code is back. And they also mentioned that they never do any time of day updates, whatever, which is, you know, we know that some people think they do. Some people think that like it works better at night or whatever, when there's no, no stress, but they reiterated, they, they don't do this. so yeah, on topic is, is back as well, on, on the game in this week's buzz. Super quick. Yeah, folks, TLDR is long this week. This week has been long. this week's va I just wanna, highlight that if you're doing any reinforcement learning and using Weights, & Biases, you can now send the rollouts, the Kind of, the tests of the reinforcement learning into Weave. And we've now built Weave into Weights, & Biases, models. It's super cool. I'm gonna run through this. I'm gonna show you how, you know, somebody trained an agent to do some email and then you can like immediately very easily see both the training runs and the Kind of, the rollouts. It's really something. It will change the game for you if you're doing any RL with biases. I'm super excited about this. It just came out yesterday. Vision and video humor was released from Biden's as well. Biden's also is firing on all cylinders. They just released like an image that the Sead dream before. And they have a different video model than this, right? They have the, Biden's, Dina, no, yeah, Dina is by dance, Omni omni humans also by, I'm Kind of, lost in how many, AI generated video techniques they have. But there is the new one. This is, feels like a research only because it's only good for like four seconds, but Huo is human-centric, multimodal video gen by Bidens. what does human-centric mean unlike video generation, like VO where it can also generate humans? This one is focusing on the humans and not like scenery. So for example, you can give it sound, it will generate the video with that sound. You can give it the person, it will generate this person from a different perspective. so we're definitely gonna check that out because it's cool. And I wrote a thread about this and I was very, very happy, to check it out. voice and audio. There's rec a speech and there's also another one. There's a TTS, red pen that TTS sounds like this Wolfram you sent to me. Feel free to send me in private chat. let's dive into open source, because we don't have that much time. Wolfram, thank you for the notes. Uh, let's dive into the show.

Open Source, ai. Let's get it started. All right, folks. we're at open source. Wolf, where do, where would you like to start in open source? I think you choose for this time.

Oh, okay.

so let's start with Mistral. I would say yes. but I already said so much about it in the TDR, that that's all right.

Mistral releases two new models.

They literally just announced it like today, right? even though yesterday they actually released it. Let's go take a look. I wanna bring it up while you talk about this, I wanna bring up the actual Kind of announcements.

So mis magistral is a model for mistrial, which is, the

reasoning model they have and they release the two, versions of small 1.2 and medium 1.2, or as the model is named on hugging phase 25 or nine. And it's a Apache license only the small one is available right now on hugging face. So the other one is only API for now. But I think as a business model that's absolutely acceptable and they have released some, medium models also data. So I'm fine with that. And yeah, it just multimodal. So now it has a vision encoder and can handle those text and images. It has an improved performance of method coding. 50% they measured in the A ME and life code bench benchmarks and better tool use is always welcome because, reasoning models are excellent for agents of course. And, better tone and persona. Yeah, that is probably up, for subjective tastes. Yeah. But, any improvement is welcome I think, and it can be handled with a system prompt anyways.

And we can take a look at some evals here.

AI E 24 and AME 25 was competition math. they are comparing it. Only to themselves. Actually, no. Look, Qwen is here. Hey, good job Mr. And deep seat. Wow. We, we always made, you know, we always laugh at like, Qwen, is not in the, in the Mistral comparison. Everyone

forgets.

Everyone forgets. Qwen

is a very charitable interpretation.

If they conveniently forget because their scores are lower than it's something else. But, this time MS. Trial does add, Qwen, 2 35 b thinking from July and also deep seek from May. And it looks like Mixtral Masra medium is not quite up there, but fairly close. Fairly close, but definitely it doesn't be both of them. What

That's a tiny model.

Masra.

So, so we don't know the medium size, I believe.

Do we know mediums size movement? I, I don't believe that they talked about it. 'cause it's not available in, in, I think

I haven't looked at it.

It's probably around 20 B or something. So I guess a medium should be 60 70 B around that.

is 24 B,

Let's quickly check what other medium models size they have.

Look, we know that the old one is smaller than

all the super large models. And I'm not sure which one exactly did they compare themselves to, but I think it's fair to say that Medium is extremely smaller than both of them.

They compare megaphone, medium, the new one to deep seek

and Qwen it's quite significantly smaller than both of them. it comes close on multiple things, not beating them. I wanna, can I just say I'm very happy with the chart from Mistral because Mistral compares itself to the very big, open source releases. They did do this, but like this medium is not open source, but they compare this to the big Qwen three and Deep Six or Apache two. but still they are not the best in all charts here necessarily. They're not even the best at any charts besides maybe math, 24, right? Everything else is lower than, than Deeps six for sure, and Qwen, but they still posted the chart. I commend them for this. They don't have soda, but they still released it. And the posting the chart is great. The small one, mega small. This is a 24 billion parameter. They compare this to Quin three and then, 32 B and Quin three, the MOE, the 30 BA three B, and then, the previous Megas role, small from, from three months ago. And this is a significant upgrade. So they, they boast like a very significant, a IME jump, almost 15 points, significant a ME 25, almost, 12 points. and this almost beats Qwen at Life Code bench and other things. so very impressive, because this is a reasoning model and it now has vision, so kudos to msra. All right, folks, I think, I think moving on, makes sense here because tons of other stuff, wolf, I, I wanna talk about the deep research stuff. I definitely wanna talk about this because, because, the, the Kind of, the implications are huge. I actually saw Imad Musak, the founder of Stability and the previous CEO talk about this being like the hugest thing on his week. I asked him why, he told me like, because deep research at the level of OpenAI, I think Imad will forgive me and our chat is not privileged. but he's like, the benchmarks on the model they can run on the mobile are pretty crazy. Especially if they can upgrade to something like, A three B, a DB and also Imad said, read my book. Imad had, has your book, the Last Economy. But, besides this, I think that, having deep research in 3 billion parameter active with this level of quality is insane. Let me show this actually on the screen.

easily run that on a MacBook,

Yeah.

So Tonge Deep Research is, from Alibaba, the Ali Togi lab, the lab in Alibaba. They said, fully open source web agent to achieve performance on par with open ID research with only 30 billion parameter, only activated 3 billion parameters. Basically this can run on a Mac, this can run on phone. Togi d research agent demonstrates state-of-the-art results scoring 32.9 humanities last exam. Every time somebody's gonna bring up HLEI will make sense. I will just laugh at the fact that, this is gonna get, saturated so fast. Calling it last. Something was so stupid, but, okay. 45% on browse comp and then 75% on X bench deep search. So let's take a look at this. this is a 3 billion active parameter model that beats. Deep seek on humanities, less exam bits, ischemia, researcher bits, Gemini, deep research with a significant, significant advantage, like 32% on humanities, less exam, Gemini Deep research gets 26, open the ID research get 26. It's insane. What are we looking at? browse comp is, I think the competition of like, just like browsing. They get very close to open ID research at 43. They almost beat everything else. Web walker, brows, comp, ZH, I think is, this is the Chinese version. Simple QA is 98 or this 98.6 and simple qa, which is, significantly higher than, you know, Gemini 2.5 gets 55. I haven't looked at this deeply, but I don't know what we're looking at. this is some Kind of beast. how did they do this? They show an example of, the strategy. This is basically, novel automated multi-stage data strategy designed to create vast and high quality agent training data without relying across the human notation. So basically synthetic data.

from what I saw is that they, trained for.

They train synthetically with rl, if I remember, if I'm, if I'm not mistaken with, with another release from Coin because they release, things all the time and, basically they, they have, I mean, go the thread. That's what I wanna say. I just wanna highlight that the, the way they train this model and generate the data is very unique and very insightful, and there is, there is a lot to learn from it. there is a reason this small model is so good and, it's pretty remarkable that you could do any of this with the model this size, like we said before. I mean, what's the limit? I mean, I'm not sure where, the limit on those things anymore?

tiny, man.

It's tiny. Only 3 billion active parameters. 30 billion in total. they show, I was just on a shout out that the Qwen basis, Qwen did so much for the community for releasing those models. 2.5 is a workhorse. Three is a workhorse. They, they're like, everybody's fine tuning these models. everybody used to say, if you guys remember in the beginning where we hosted every week, somebody who fine tuned on the open source arena on hack and face. We were like, Hey, fine tuning is like the big thing. And then fine tuning, like took a dive, RL brought fine tuning back with a, with a force, and now everybody's using these qua models. Everybody's fine tuning them for purposes. This is the result of like the new fine tuning. Supervised, SFT. they also, now everybody does rl reinforcement learning in some sort of ways. Here's an outline of how they did this. They started with a base, from Coin series. I don't think they mentioned which base. and they had multiple stages of high quality synthetic data, with data for like multi-step execution of these like agents that need to do something. And they had a whole system here with automatic automate, automating synthetic data creation, and then building a, a reward service for this. Basically, when the agent does what it needs to do, then they need the reward service and then send it back to the training. and then they need to rollout. They basically rollout means, they train the agent a little bit. They then wanna test it out whether or not the, this new trained version does better or worse and give it rewards. this is, and then they basically prompt the, the model that they've trained, see how it performs. Then they give its scores and then they give it back to the, to the service. So they have like a very cool setup here with a automatic rollout service, which basically makes them go faster. And then they have a simulated environment, a real environment, with tools like search file parsing, visiting a website, Google Scholar for some reason. so yeah, deep research in a small model. incredible.

people.

People are running our rail in real life now. Yes, I mean, that's our l on the real internet. before that, the week before we saw cursor doing crazy thing on policy, on real life user interactions that's pretty crazy that we got to the point where RL is so accessible that you start to run RL in real life basically. If you take such a small model and just let it randomly try stuff on the real internet, it learns if you give it the right word. I mean, that's crazy when you think about it. Absolutely crazy. It's rl but it's crazy that it's working that well in real life that is noisy with such a small model. I mean, I think it's remarkable.

hundred percent.

and the fact that RL started picking up, and then all these companies are now building RL environments and like getting, getting to different places, not withstanding, you know, with the bison. CoreWeave also, like talked about, you know, the, the folks from Open Pipe join us and we're gonna do like a bunch of RL stuff as well. It looks like at least that's, that's some of it's where, where it's heading. Although Carat is skeptical about this being the number one thing that you know, that, that will get us to a GI. But, we can talk about this. All right, moving forward, it'd be an important part, I'm sure. That would be an important part. So this is togi Deep research. 3 billion parameter active, 30 billion in total. And it beats deep research on open AI was insane. and by the way,

I checked it's QU three based, so it's very recent and I'm very

interested to check out if it you Yeah, they did that test and, in their own Yeah. mo setup, of course. So I would be interested if you use the same model, give it an MCP web search tool and see how far it goes that way. So yeah, we'll definitely look into this more.

Okay, so, we're moving to the last thing on our Kind of,

open source, at least the LLM stuff. perceptron ai and we're gonna do like some vision stuff as well. There's like a, a bunch of other open source, perceptron AI released Isaac 0.1, and that is like absolutely dope. Perceptron, is this new lab. I haven't heard about them before. And they have released a new language model, a small language model, a 2 billion parameters in total called Isaac. But they call this a perceptive language model. which means that, they call this the efficient, the, the Pareto efficient frontier of perception. They put themselves in this graph. on the left you have performance in percentage of some sort, and then on the bottom you have cost per million tokens, and in logarithmic scale. So at the absolutely right bottom corner, they have cloud opus, which cannot point, they, they haven't parenthesis. Cloud Opus cannot point, what does point mean? We'll, they explain to you in a second. and then GPT four oh cannot point as well. And then they have GPT, Gemini, flash and flashlight and like basically the, the X axis cost per million tokens. and then they put them in like 60 ish percent. They put themselves as like 72, 70 3% with a very, very low relative cost. Because look, it, it's a tiny model. what can it do? This model can do visual qa, grounded spatial intelligence, precise point pointing and localization of understanding. So you can say, what's, you know, what is the highest book on the shelf? For example, You can give it an image of, books and say, point to the highest blue book in the stack. And the stack has multiple blue books and will like basically draw a rectangle around this and point, you can build a smart system of monitoring. You know, they have an example here of a, Construction site and you have specific rules that everybody must wear a hat and, and boots, and you can like check whether or not people wearing hat and boots by simply prompting it. all individuals entering work sites should be wearing helmet, gloves closed those shoes. If all three are not worn, please say alert. And the the model comes back with alert. The image shows a person wearing a white hard hat and closed those shoes, but no gloves are visible. So, you know, this model is like very versatile, but the whole thing is this is a prompt. You don't have to build in or you don't have to like Finetune or train it like YOLO or, or other like visual performance ones. So this is a language model that can point, but instead of talking too much about this, let is just like take a look. they have this like super cool demo. you can choose the response style polygon, box point or text. let's do text for example. And there have an image here and, it says, determine whether it's safe to cross the street. Look for signage and moving traffic. The image is basically a woman standing somewhere in the cross and you run and it's fairly fast as well. we already got an answer. it says no, it's not safe to cross the street. The traffic signal visible in the image shows the red pedestrian sign. we can take a look and see the red pedestrian sign here Okay. this red signal indicates that pedestrians should not wait. but also instead of just asking determine, you can say, instead of response text, we can say point and say, find all the people all. Find all the traffic signs in this photo and segment them so we can do something like this and run, and the model will find us traffic signs. And, you can see that it says this is a crosswalk sign. This is a left turn sign that it found somewhere over there. this is traffic light and then a traffic light. How

large is that

though?

2 billion parameters.

Let's try something hard.

Alright. that Looks too good, man.

I have gotten to the point where this model fails.

I've uploaded a huge, group photo from Burning Man, like our whole camp. There's a lot of people there. That's

then that's Dan's object detection.

I'm not sure That's a little bit unfair.

you said let's try hard, right?

So like, let's try hard.

But, the model is probably trained to just detect

a couple of, instances if you're gonna do dense subject detection. I'm not sure it's fair. but I think, that's

just for folks on the, on the stream.

This is a picture of maybe. I think 70 people, just rough counting, all dressed in weird things, all very like, dense together. and I wanna say find and segment all the hats, boots and costumes. And then let's run. And let's see if this model, this is a 2 billion parameter model and, it's running and you can see his life. It's selecting hats.

Nah, I don't know how,

how accurate this is, but like at some point

it's gonna start doing boots. I think based on this, we found the failure mode for the model for sure, because like I had this before, but it does some stuff. Well, that's

impressive already.

Yeah, I mean I'm not sure that's what the model is intended for, but no, I'm

pretty sure it's not.

But that's,

it's very impressive.

it's really good.

Yes, it's

really, really

good, I wanna say.

looks very

useful.

And I did a test as well where I uploaded an image where several people were, and I was asking in German for, the ones, wearing glasses and it highlighted them even when it was a bad light exhibition. So it's impressed me a lot as well.

Okay.

I have another picture here. And this is a. Suddenly I say for my kid, this is the Harry Potter alphabet. It has like ABCs and all of the, all of the letters have Kind of, like some, some, spell from Harry Potter. So let's find, find the, spell that turns off the light. Let's see if we can do it. Turns off the light. So it needs to read this and figure out, oh my God. Oh my God. I, I asked, I uploaded So again, for the viral moment, I uploaded a screenshot of somebody's like poster with A, B, C, D, fg, like all the alphabet. And every word has a little description on it with the spell that starts with that letter. And then I asked it, find the spell that turns off the light and then it found and said the spell that turns off the light is NOx. And then the NOx is like this active thing that I can point to it. It will zoom the spell, which part of the ABC is a Harry Potter is used to put out one light, it's important spell Wizardly world, blah, blah, blah. I, this is way more cooler to me than like the previous one. It just like it read it and found it. And like I didn't say Knox or anything. find all the spell letters and segment them so you can also do something like this. And then they will try. I think it's skip two. Black Curtain Cru Descender expect, like all of these became like smart pills in the output for folks who are just listening that can click and it can zoom in. So it found, not only does it know about them, which is all multimodals models probably know about this, right? If we upload the same picture, we can ask it. All multimodal models know about this, this knows where they are, and pointed me to them as well and built a box around them. So great for segmentation, for example, and this is 2 billion parameters.

Imagine you have a picture of us, for example, and

then you have a picture of yourself and you ask, who is Alex or not? not the name because it's displayed, but, just who looks like this?

we're gonna take a picture of the three of us folks.

Today's show is a bunch of example. I hopefully we, we will have time for all this. I, I uploaded all three of us and then I'm gonna open, what's

the video guys?

Do we

know?

How much was this train? Do we know how that thing was made? I mean, it's very incredible. and that's, I don't know, we can use, people just release crazy things. Yes. Like casually every day. Just give you crazy free stuff you can use for, I mean, it's 3 billion parameters. You can run this on your phone. Like, I mean, that's crazy.

Do you guys remember Chameleon, the meta chameleon model

that has been so long ago where they had, a model that could create images and read images that is a team behind that percept Well, from your test

worked perfectly.

So folks who are listening, I took a screenshot, all three of us without names, and then I took a screenshot just with me, separate image, and it said like, Alex Volkov on the name. And then I asked it to find his name. So in the picture of all three of us without the name, it seconded me and said, and this is Alex Volkov.

I would now give it a picture of you not from the stream that

you wear the same clothes and so on. I mean, that's gonna be impressive if it's gonna work.

yeah, yeah.

That, that makes sense. We, we can test this or we can move on because we have, tons of other stuff. Oh yeah, perceptron perceptron is quite incredible. And this is just the start of the, the examples that we're gonna have here. if we're in examples, I really wanna like show you ve so let's move on. We're, we're in open source. I think we're moving from perception. This is open source by the way. you have this model on hug and face, but obviously the UI is what you, you're here for. And, this model is CC by and see the license. this is the first in their family models built to be intelligent layer of physical world. Before we move on to the next segment, I wanna move on to the next segment where, we are gonna chat about Huon, SRPO Super quick. SRPO is a new method of fine tuning, image diffusion model. Nisten mentioned this before he disappeared. this is a model that the folks in, Huon and Tencent and I think, Chinese University of Hong Kong Sheen. they released a model, the trained with this, method. this is directly aligning the full diffusion trajectory, refined grain human preference. This is a new method they call Kind of, like, deep Sea gave us GRPO, which is a RL policy method that they open sourced. this is called SRPO, semantic Relative Preference Optimization. They optimize image diffusion models for different things, specifically, training them for specific things. For example, like realism, without the AI look. Do you guys remember, we chatted about, the partnership between CREA folks, First of all. They released a model called CREA one, and there released a Finetune Flux Crea. this whole thing was basically crea folks training the open source flux models to not have the AI look. And the AI look is very, like, you know, the, the face is airbrush. you can tell, still, you can tell even though they're incredible. these folks are achieving some similar results with non-AI looks, this picture does not look AI generated, I think the highlight is, I, I, look, I'm burying the lead here. The highlight is you could, Finetune this. With just 10 minutes according to the team. SRPO makes diffusion models photo realistic in just 10 minutes, delivering 75 x efficiency gain over previous methods like, like 32 GPU setups. fine tunes, methods, suppose profitable rewards. So you can tell it what you want. improves realism and aesthetics and is open source with example code. and then you need less than 1500 images to, for 10 minutes to directly align diffusion models. I think that's crazy. I think that's crazy because, the improvement in how quick we can Finetune some of these models are just like incredible. I do wanna show some examples here as well.

and it's my prompt that's insane.

Profitable

and fewer than 1500 images.

Yeah.

Yeah.

I'm looking into, the way they did it, but it's not something you look into quickly apparently. But that's crazy. I mean, they did a lot To get to this point. I mean, that's absolutely crazy.

So I'm showing, a comparison here between flux dev,

the Finetune for realism and SRPO. And in this example, it's very clear to see which one is the more realistic. The SRPO is way more realistic. It just looks like a person that you would maybe meet while flux dev is absolutely like, you know, the skin is AI skin. You can see this Wolfram thoughts on this so far?

Yeah.

What, it's interesting and it's also a subjective thing, of course, because if you want, airbrushed images for instance, it depends on the person, what you prefer in these. But it's great to have an option that is super realistic. That is something that we have been missing.

Yeah, probably you can tell it to do whatever you want.

If you want it realistic, just tell the model. Go be more realistic and the reward function will guide you to a more realistic model. If you want cartoonish, professional shot, iPhone shot, amateur shot, that thing is crazy. You can just tell it what you want. look at the results, man. What's the size of this? can I run at home? Can I not? I'm not sure. that thing is crazy. The whole image and video field is just blowing up. every week we see a crazy thing. we didn't have this before.

No, this is absolutely crazy.

I think it was like 41 gigs in B float 16.

Yeah.

the model is, but the process, I think you can run, I think they say, you can run this on 32 GPUs. I don't know if you have 30 G GPUs in home, but that's what I have in my notes. the last thing

you can run these on 32 GPUs.

Okay. I know you can run the model hundred model.

The model's probably small.

the thing that I wanna highlight super quick before we move on here is, Somebody took this and then extracted this into alo, and you can apply this LoRa to flex dev directly. So you don't have to train like a full model, which I just love open source, dude. I just love the open source community. Ali, nerd X is, putting up a flexx dev, SRPO, Lauras. So you can just apply this. And Loras are tiny, right? All right folks. so this was in image diffusion, and I think we're gonna stick around before we get to big companies. I really have to show you Revit because like, Revit is absolutely insane. so Revit is this new lab. Probably not new, but like the, the, the release is new that have shown a incredible reimagining reality tool. And I think this one you have to see, so if you're listening to this, I'll try to describe as, as best I can, what's happening on my screen. But bre basically, Is a image editing model and creation model. Everything that you can like, add or remove stuff, and you can create things very creatively. We talked about n being an editing model, not just the image generation, but like full editing model. You can tell it. I wanna remove this, I want to add this. but ance really seed dream, right? Well, from, correct me if I'm wrong. And then, we had the Huon as well, and now we have Rev and it's their own, it's their own approach, their own trained model. But I think the most interesting thing here is not the model. I think the UX here is what's most interesting. I think they just came out and they may have re

So do they have their own image

model look or they're using something like banana?

they have their own English model and I think, what you're looking

at is basically they have some sort of image understanding Kind of, like what we saw with the Isaac, the image before and the Kind of, the picture you uploaded is in the middle, or the picture you generated with Rev is in the middle. On the right you have Kind of, like it's not really layers, but it's Kind of like layer menu. In Photoshop, it's objects. So they detect in the image which objects you have there. So if you have your face, they say, this is a person, and inside the person you gonna break it down. There's like the face, the hands. Everything is separate. all of these objects have bounding boxes on the actual image. And you can move them around. you can say, okay, this person's face in the next generation that I want. 'cause it's a iterative process, right? Look, you build up, you can take the face, move it to the top left, then you can create a new object that doesn't exist. Create a bounding box for it and say, here I want some text. and you just basically prompt each of the objects. Why is this so cool? instead of just like prompting and then reeding the same image to the prompt and asking for changes here, you visually just move things around and say, here, I want some text here, I want another big text here. I wanna move this person here. I want to add hands. And you literally just control the composition. And I've never seen like a UI like this before and I know it's possible with VO because remember, some folks decided, discovered that if you have the first frame in VO and you say, Hey, you know, this should change to that, this should change this literally with text on the first image, VO will Kind of animate this as much as you want? They basically took that concept into image generation stuff. creating something with this fidelity where you can move things away. You can literally say where you want, what you want, I think is gonna like seep through and like, everybody's gonna build this UI and CRE and everybody else. It's really, really cool. Amazing.

because always we have heard about, but the

Photoshop Killer and stuff like that, but this is an actual Yeah. Photoshop like, AI where you have much more control in the ui. So that is, that was a big step forward I think.

Yeah.

it's really, really super cool. and, they basically break down and describe every object in the scene for you. And I think the cool thing is that you can now move. basically you can see here that like I'm taking the text that the image generated and it's literally moving it in the next generation to the top left because I'm like, I want it here and not in the middle. And then, after you've done all these changes, you're like, submit and then they will generate the image and then you'll see that the text is gonna move, to the top left. Let's try to see it generate, generate, generate generates. And the text is top left. And they changed my face as well and they added a subtitle. But basically, the controllability of this is just like scary and super, super cool. So definitely try out Rev. I think it's free for now but it's broken right now. But it's generally, super cool. Should we move on folks? we're gonna have an interview soon with Jeremy. But for now, I think that before we get to World about this week's buzz super quick. unless you guys have something super small to talk about before, I think this week's buzz makes

I just think, it's funny that you have a company investing.

Tens of millions of dollars into training a model, and then someone on hugging face like, yeah, I extracted Laura. and then you can just move the Laura and play with it and merge the weights and, and you know, people online. But that's part of the game, that's open source. The dynamic is just crazy.

Yep.

All right. So I think it's time for this week's buzz.

Welcome to this week's buzz, a coroner on Thursday I, where I talk about every update, the, and cool things that happen in wi Weights, & Biases from CoreWeave. My name's Alex Volkov. I'm an AI evangelist and this is my job to tell you about these things, but also it's not my job to be super excited about the cool things. And definitely we have some cool things, this week. So, as you guys know, we've talked about RL a lot lately. RL reinforcement learning is this new craze in fine tuning because, almost a year ago when, folks created, the test time compute paradigm with reinforcement learning, there's been a bunch of incredible, Innovations in this world. And then Deep Sea came out with GRPO and that exploded. And then many people are taking a regular Finetune model and then RL fine tuning this for specific purposes. and this seems to be a big thing. RL environments is a big thing. We just talked about multiple RL improvements, and then grok four, I believe half of the compute was spent on a L as well, which is like significantly more than anything else. and so everybody does it with Weights, & Biases because the Weights, & Biases model, because Weights, & Biases is the premier and the top, experimentation tracking software for machine learning engineers and you need to run a bunch of experiments on models, a bunch of runs and with basket models has been traditionally and still is the best place to do so. However, a rail is specific because as part of a rail, the so called rollouts, so you have to, let's say you train your agent to do specific email answers or you train your, I don't know, agent to do some deep research stuff like we saw just now with the Tencent stuff. as part of rl, you need to give your model rewards for good behavior. And as part of that, you have to like basically run inference on this model in the middle multiple times to have it do the thing that you want to check for, right? And that looks like regular inference. And because it looks like regular inference, people need to read this sometimes and see why in the middle of your RL release something drop down or reward, function, drop down, et cetera. basically it means that your agent or the model, your training, you need to run inference in the middle multiple times to see which one is the correct one and to say, this one's the correct one. Give that a reward. This is where Weave comes in. Usually when I talk about Weave from sbi, this is the tracing and evaluation platform. we have built a platform, for this purpose. And now the reveal folks. We have unified both the platforms now. We now have, we've built in into Weights, &, Biases, models. This is huge because Weights, & Biases so far is this incredible place where you can see charts and they're super configurable. You can see your trained reward loss functions. you can see all of the things that you want to see in a, machine learning Kind of process. But also now very easily you can report your RL rollouts into Weave. And then you have like a whole Weave panel here within right devices. So, why is this very useful? Is because you can then super quickly discover which of the steps that you were here. Let's say we're gonna take this train loss. There's a loss spike, and there's a loss dump here. So we're gonna filter this, and then we're gonna go into the, rollout of that specific Kind of agent thing. And we're gonna click, and with this click you'll see that, for this step, this is a, by the way, verifiers library from will Brown, from a prime intellect. and then, we have a first party integration with multiple libraries. At this point, you can see all of the things that happen. So here's, we use Open AI chat Complete to create, like some multi-step reasoning thing, please reason, step by step, put your final answer in the thing every, and we give it like a, like a question every hour past noon shadows from a building stretch X or five feet. You can see the conversation that your agent, during the training, during the OR l training has, has done. And you can see like all of them, you can like skip based on all of the conversations, just like one by one. You can see 36 conversations happening here, and you can see this for each step. So basically this combines the extreme power of Weave, which we've built specifically for this purpose for you guys. With the training. 'cause you can like select at which step of the training, which, rollout things you'd like. And I think for many folks who are using orl, this is gonna be super, super useful because basically we build, we've for this, but for AI engineers who work with just, you know, open the AI APIs and they build agents, et cetera, this is now coming to the folks who are training agents. And I think more and more folks are using RL to train their own agents. So if you're doing any RL at all and using Weights, & Biases for this, the way to get it is very simple. You just import Weave in Python and you just like wrap with decorators the functions that, that cause this. That's it. and then you have this incredible way to, to look into your data, which is very important. This is the update for its and biases, thing if you like, very easy to start. If you have any questions, feel free to reach out to me and we are moving forward because we are a lack of time.

All right, we're moving forward. I think in something like 20 minutes, we're gonna have, Jeremy, Berman here to chat with. And I think we have a bunch of stuff to talk about before and after. And let's dive into big companies and APIs. I wanna chat about GPT Five Codex, first of all. so let's do that. Who wants to start? Us Codex did a full circle. Started as a model, and that was a model. Again,

Full circle, started as Yeah.

you are absolutely right. Started as a model, but three years ago. Anyway, there are many things called Codex today. today we are talking about, codex that GPT five fine tuning version. Open AI had Finetune their, flagship model, GBT five to be better at code. You might ask yourself, okay, GPD five is pretty much the best already for coding, pretty much, or at least among the best. Why would you need a specialized version for agent decoding? basically the thing is that, what OpenAI is aiming for, you can see on, you can pretty much sense it, on the web app. of Codex that you would be able to just launch missions, launch Codex, launch a model, let it go on, do its thing, and, you review the PRS later and if you like them, you merge them. And that's pretty cool when you have the Chat GPT, app, infrastructure review because then you can launch tasks from your phone. Codex just works from the Chat GPT app, on your phone. So basically what we got, this week is an updated version for the model that is running this service that I just described. it was based before that on a specially trained version of oh three, and now it's been updated to a specially trained version of GPT five. And I must tell you it's really, really good and you do feel the difference. It can go on for long, solve really long tasks, and just come back to you after an hour or something when the test is extremely long and complex and it doesn't lose track, in my opinion. And that's Kind of already vibes and personnel opinion. I haven't seen other models not losing track.

Thank you so much for covering the, the, the beginning.

I wanna highlight this paragraph as relates to what you're saying. literally from the blog on OpenAI during testing, we've seen GPT five codex work independently for more than seven hours at a time on large complex task iterating on its implementation, fixing test failures, and ultimately delivering a successful implementation more than seven hours. this combined with the other ICPC news, just blows my mind how fast we're moving towards like super superhuman coding intelligence for agents that running synchronously. they say now the like, like 90, like 90% of open AI uses stuff and Codex is definitely some model that they use, based on open AI employee traffic. We see that the bottom 10% of users, turns sorted by model tokens. G PT five Codex uses 93% fewer tokens of GPT five 93% fewer tokens. So the, the thing is, not only does it upgrade, gimme example, it's

important.

that's the first time we see a Frontier Lab bragging that they are saving new tokens.

Yeah.

Like that's the first time that we see an, any, any of

them, any, any of the Frontier Labs. Just saying you can do the same with less tokens. We have seen other forms of efficiency, trying to help the user save, but this is the first time we've seen direct comparison less tokens you pay for specifically

because GP five, is very good.

It's been clear, but it thinks for a long time, especially in like large task Codex does better than GPT five. The regular one on, Swyx Bench verified gets 74% versus 72%. Not a big jumble, it's still a jump, but on code refactoring tasks and pr, it's like a significant jump on accuracy. 51% versus 33%, but it does so with significantly less tokens and also it thinks more on the higher, very high, difficult tasks. So they basically created this, I think this is the chart that I wanna zoom out so we can see the whole chart. This is the chart, like the most important chart here, on 90th percentile of, this is open end employee traffic. 90th percent of model generated tokens per response. GPT five Codex gets significantly longer on the difficult tasks and significantly smaller on the more like, like easier tasks compared to GBT five Medium. This chart is like incredible because they basically made it think for longer and the difficult task and just respond quicker on the smaller task without like in saving you tokens. I think this is like absolutely incredible. and yeah, this is a great model. It's now built into. The Codex Playground on the web that runs independently. It's built into the Codex CLI that they've improved and now looks super, super cool. And you can log in with your open AI account. So you just let go. And usually codex dash MI believe I have this, I have this here.

NPM, it's really, really simple.

Yeah.

After we have this with NPM, if you really like force it into GPT five Codex, you're like, do Codex Dash model dash MG PT five Codex, and just go like this. So basically

you can just do it through the menu, through settings.

and just select whatever model that you want.

and It's really dope.

So folks, definitely give it a try.

that thing can control your computer from there.

It's, it's, yeah, on the web, it does what it does on the web, but from the terminal, it can control your computer. That thing can run your computer for like, like seven hours without you on, on the keyboard. I mean, that, that's crazy. Seriously.

So just to clarify what Codex is generally, so now there's

a model called GBT five Codex. There's a Codex CLI, so command line interface that we just showed. There's the Codex, in extension for id, so vs code, et cetera. There's like a site panel. If you, if you're using Cursor for example and you want Codex in there, You can just like have cursor in here with this like OpenAI. This is like a Chrome extension basically. You can have the GPT five, codex here as well. And then there's also the Codex Playground on the web. So if you go to like, you know, codex that OpenAI, there's the Codex Playground that's like independently can do PRS for you, et cetera, for code review. And now Codex includes a code review component. so you can tag it on GitHub and it'll go and like look at your PR and it's like really good open. The, I says that the every PR now in open the I is getting reviewed by Codex. at open. The I Codex now reviews the vast majority of our prs catching hundreds of issues every day before we even look at them. And you can just like attack Codex, was super cool. Nisten is back just in time to say things about Codex. Super quick before we move on. GBT five Codex.

Oh no.

No.

Okay.

I can say something because the timing is amazing for op.

They are so lucky because now the controversy about, op Opus and, SONET being, yeah, like, and tropic the issues they had, that was a perfect time for OPIA to now have a coding model. Have the CLI improved and, yeah, they converted a lot of users to their ecosystem now.

Yep.

Yeah.

I should be good now. I still think, Opus are better. They're just not, it's just the tooling around them is pretty difficult to get right. at the end of the day, because I do use GPT five through AMP too, and I have found it to be the best at planning and the smartest at finding the bugs and analyzing them. But when I need something to look good on screen, there's still no match for it in my opinion. Like it just, or at least the way I'm used to writing stuff or the way I want it, I still find no match for Sonet for an Opus yet.

Interesting.

Yeah, that's great. And, we can rely on Ton to give us, a honest opinion always and not, you know, just cheer based on the timeline. So thank you Ton. folks, we have to move on, because we're gonna have a chat with Jeremy very, very soon. But before this, I do want to chat about Meta Connect. z and pretty much everybody from Meta took the stage yesterday for their annual Meta Connect. my glasses are talking to me. Hold on. I know the battery percentage now. basically, Zach took the stage yesterday, and, announced that not only that they're bringing, they didn't announce a new La Mow or anything like this, but they said not only they're bringing, new versions of the glasses, like a very similar like, Oakley the standard, standard Oakies,

You got the Ravens.

I mean, yeah, I, I have the, I have the AK glasses. That's fine. But I have the Ravens, by the way. Nice. So, yeah, you got the Oakleys, they have the Ravens. They're great, by the way.

So meta has been popularizing, the glasses,

the yamas wearing right now. the new ones that Kind of I'm wearing right now as well. they released a bunch of new ones and they're invested in extra Luxottica and Raybans and Olis, et cetera. However, a new category of thing was announced. I really wanna show this 'cause this is like super, super cool. I have a video to show you guys, video file. let me see if I can actually do this. 'cause I've been recording yesterday and I wanna show you this opening. Yeah. Okay. This is how the show opened. I am gonna narrate. So we basically sees Zach sitting on the couch from a, like a perspective reading a script, all right. And somebody's saying, we're ready for you. And we see like a video recording of him in, in the portrait mode. And then he reaches into the camera, picks it up and puts it on his face. Isn't that super cool? And then you can see a screen. So you can see WhatsApp in there. lemme pause and see. What else can we see there? You can see a screen. So basically he, they started recording on the camera and put it on the desk. And now he's like, okay, lemme put on this camera. You can see WhatsApp. You can see Instagram, you can see a camera, you can see a music tab, you can see calls and messenger captions And photos. So basically they announced a new pair of glasses and then he's gonna play, and they did this live demo where he walks out of the trailer, which Kind of looks nice, Zuck gets nice trailers, and then he kinda walks out of the trailer, walks into the hall while playing music. He sees the security guard. He walks forward, walks forward, all of this was streaming live. And then he gets a bunch of text messages on WhatsApp during this from the audience, And then Kind of goes into the hall, he sees, buzz Tank Buzz is the guy from Meta, vr and then he sees Diplo And then he's just like, goes to the audience. How cool, how fucking cool is that? This is one, one hell of a life demo for a product. And basically what he's showing is a new generation of meta glasses with a built-in screen that you cannot see from the side. Like if Yama is wearing those glasses right now, and he would have a screen, you would not see the screen, they have like less than 2% light leakage. So like a person that looks at you cannot see that you're looking at a screen, but they have a screen in the glasses. The form factor is not as sleek as the ones that young wears, but it's a little larger obviously. 'cause they have like to put a screen and more battery. This is a new category of device right now. This is like absolutely new category of device like Wolf. And we talked about this a little bit. There's a computer, there's a mobile phone, and now there's smart glasses with display.

Yeah, I actually have some at home already.

developer kit from the frame class. I had also a screen in it, but it was just a little one for a bit of text. So, what Meta is doing is much, much more advanced. Of course.

Yep.

they called it Meta RayBan displays and I wanna show you how they look because they look sleek as hell. and I wanna find a, a source that doesn't have Now the question would be like, why would you need those? And they have a high resolution, like very high resolution display. Zack said like 5,000 knits of brightness, which is insane, but like it makes sense 'cause those are sunglasses. You wanna wear them when you go outside. so definitely makes sense. 12 megapixel camera, all the right camera. and then you can privately view texts and things. You basically don't have to take out your phone to do most of the stuff. And this is like a whole category of device. Not only that, it has AI built in and I think that this is for our show. Maybe the most important thing, just before this though, I wanna say, this new input method, this is io this is output, right? you see the output, you need some input. You need to tell it. the glasses that I wear, the yam wears as well. the input is voice. you say the keyword that I don't wanna say right now 'cause it's gonna trigger, but like you say the keyword and then you start talking to it. And then you have AI built in. You can do a bunch of stuff. You can send text message, you can hear, the incoming thing. this is a new output, visual 'cause you can see the incoming message. You can see the photos, but also controlling that you need some signals. So what they did is genius and they did this for the past decade and a half. They've been building this technology. they purchased a company called My Labs a long time ago. this company used electromagnetic signals on your wrist. Send those Kind of understanding signals to some device. So now they've turned it into an actual technology you can buy in a week. These glasses with the display are gonna cost $799 and they're gonna come out in a week and you're able to buy them.

that is super affordable.

If you think about what you are getting there, that is amazing. So, wow.

Yes.

I'm struggling with a bunch of ads, but basically I don't like perplexity. they look very sleek. They Kind of look like this. Zach is wearing them. Zach is wearing them and he's wearing the neural band. Yeah. And, you can see the little screen here on the right and the neural band Kind of controls them. I find this incredible, absolutely incredible because, being able to silently control this without talking is one of the gripes that I have. This, definitely, with this device, The AI portion there is very, very important because what happens is meta is building super intelligence. Meta is paying, we saw between hundreds of millions to billions of dollars to folks to actually like deliver super intelligence. this is how they will deliver this to the physical world. You're gonna wear these glasses, you can talk to ai, you're gonna see some outputs because it's very much missing, in the form that me and Yam have currently on our faces. this is how they're gonna bring the AI to the physical world and controlling this with the wristband. They actually showed an example of a few AI things, wasn't mind blowing, but they had life translation. So you could see subtitles under the person you are talking to. You're gonna talk to a person. You can see subtitles below them. That's like super cool. And also they show the demo of a Gentech approach where you can ask the AI something and then it will send a notification when it's done. And Kind of, it went on a deep research task and then it came back. this is something that wasn't possible before, unless you pull out your phone and get a notification. I think this is an absolutely mind blowing thing I was blown away by the fact that Meta can do a presentation as exciting and as innovating as I've ever seen before. While Apple's presentation we're cool, but like, pretty much as Snoozefest comments folks on this, will you be getting the glasses for 800 bucks?

in Germany,

Look, I think, we need to mention that the demo

didn't go, as smooth as, planned. look, the glasses in terms of hardware, software and so on, the glasses are, amazing

Should we play the demo didn't go.

I have a recording of this.

Okay.

we can, I just wanna say that the ai, both in our glasses right now and also on method demo wasn't the best as the, you say the least throughout the day, but the glasses themselves are incredible, so I just hope that, meta is gonna catch up and, gonna change that.

Yeah.

So let's take a look at the demo. That didn't work. So we did see the demo that didn't work. Live demo of Z starting the show, walking out, so basically here's the demo of, let me actually go and find this video. I have this video of, basically him getting a phone call from Buzz and this phone call did not work for them. lemme see if I can have this on.

Yeah.

But even if the technology failed, I think the form factor is still the best one because the I is what you see. Here's what you hear can talk to you.

So here's the live demo that they tried.

he tried to get a phone call from Buzz

Paul, I think we should,

For folks who are viewing this.

Zach is standing near a table and he does something with the right wrist. What they have is this like, incredible thing they barely talked about. He is writing like as though he holds a pen, but he doesn't hold anything. He only has like the band he's writing on the table as like a person writing and they have like a writing detection. So he's basically answering, you can see like him write like scribbling and you can see on the right on the screen that says he, what do you, he says something and then in the writing it says yell. Let's do it. All right. and that is like super cool on its own. Literally a new input thing. You basically write on any surface and like detect it.

There we go.

Uh, oh. Well, let's see what happened there.

So basically he's getting a phone call, but he doesn't

get the UI to answer the phone call. Then he got the ui, but the call was already over and then Buzz was like calling him multiple times. The demo, the live demo was hard, but it worked like eventually and, and that was super cool. folks, 7 99 comes out in a week. Incredible new innovative frontier. I'll definitely be getting them. I will have to decide if I'm wearing those cool ones or them cool ones, but they have a fucking screen. I am very, very excited about this, and the screen is super bright. All right, folks, we're moving forward because we have a guest here. and, silently, while we're talking about like the cool meta stuff, we had a new person join us. Jeremy Berman, welcome to the show. This is your first time here. feel free to tell folks a little bit about you, what you do, where you are, and why you are here. We're gonna go into next.

Thanks for having me.

First time on the show.

Tell us a little bit about you, Jeremy.

what's your background? what are you doing, day-to-day life?

Sure.

So, I'm an AI researcher, currently researching, reasoning at Reflection ai, where we're building, foundation models. we're, pre-training and post-training foundation models. I got into RKGI, which I think is what we're gonna talk about. about nine months ago, I noticed that, the field of, language models was progressing very fast, but on a few benchmarks, notably RKGI, progress seemed to have been slowing. I found it really fascinating. about nine months ago, I decided to pursue, that as a research direction. Ended up getting, the top place on R-K-G-I-V one, and then have been mulling over ideas for how to improve it. And, just yesterday, the scores for R kgi IV two were released. I, updated my solution and I'm now first place again.

That's incredible.

So first of all, congratulations on this achievement. It's definitely a monumental achievement. RKGI has been around for a while. FF Francois, famously posted up like, 1 million something prize for beating. we've seen RKGI being Kind of populated. I think the last time that we actually talked about this on the show was in the context of, test time compute and reasoning models. And oh three got the Kind of the highest top score, but it costs like a thousand dollars per solution. So the, there's the famous RKGI thing that will put up on the screen right now that has, Kind of, like the x axis is cost per task. And the y axis is the score. And all three was absolutely to the bottom right. but also like it cost like more than a thousand dollars for, gimme a second. Lemme try to. Figure this out. Okay. Yeah. This is the, the RKGI leaderboard. And, and this is the last time we, we talked about this, and now you're also on this leaderboard. and you, you're not a AI model, you're not a foundational lab maker. You are a person that like scored high. So could you start with telling us what RKGI is? Like what is the actual test and why it's difficult for models?

Yeah, sure.

So RKGI is basically, IQ test for language models you can think of. So it's a series of grids of input output grids. And the idea is you're given a few examples of transformation rules to go from input to output, and then you basically just have to, solve the test input to output. Basically they're quite easy tasks actually. So, on A-K-G-I-V one, the average human mechanical chi gets around 77% on RKGI. But then at the time, the best language models were getting around 5%. And so it's interesting because. These, tasks, again, children can solve them. These are children's puzzles. so as you can see here, you get, three training examples, and then the fourth is what you have to actually solve for. everyone here could solve these, but language models can't. I thought that was just very fascinating, right? We're living in a world where language models are PhD level on math tests. They're now, top programmers in the world, but they can't solve these simple tasks that, children can.

Yep.

And, famously I think the whole point of this in the beginning was, hey, the second, you know, this is a GI test, if a model can solve this, it's gonna be generalized like a human. And then super quickly we saw that like models are starting to slowly creep up and we can actually see that The creep up is not that slow. we can see based on the models that, you know, the smaller models, like deep GR three, all very, very low percentage. And then suddenly, like the test time compute models, specifically the thinking models started like rising super, super quick. So you can see GPT five mini is Kind of low here in the low, 25 percentage, but like significantly. Higher than GPT 4.1, which is like less than 5% of completion. The more thinking there is, the better they, the performance. So like you can visualize the test time compute, like scaling Paradigm based on RKGI. The longer it thinks the, like the, the more it is able to solve. And then you can actually see the, the, like the very strong thinking models with a lot of rl, like ROCK four thinking is at the like 60 ish percentile or something. All three medium. famously got a very high score. And I think the highest score that we've seen from a model was O three preview back then, that cost a lot of money and put in a very high score of around 75%. And here you are, Jay Berman 2025 at the score of almost 80% okay. So you don't have a foundational lab under you with a hundred millions of dollars of training compute to give a lot of like reasoning things. And I think when it was posted on the Arc A GI as well, it says you also got it to get like $8 42 cents per task or something, which is significantly less than the thousand dollars that g GFI thinking cost. you wanna tell us how, is it a secret? Like what, what's going on?

have you done game development before?

So maybe I can start with my first solution.

nine months ago this was actually pre-thinking models. So, this was, you know, we were living in a world where you had to basically build your own test time compute, right? Because The models weren't inherently thinking, so you had to basically force them to think. So, the solution I came up with for the first one is to have, sonnet three five at the time generate python functions that would, basically code the transformation function. I would have sonnet three five, generate 200, transform functions in Python. And then I would execute each of those functions on the training examples. And then, if I ended up getting a hundred percent on all the training examples, I'm pretty sure that this Python function is correct. And then I'll run that Python, on the test example and then submit that. and then if it wasn't correct, I would basically take the top, let's say 10 or 20 Python programs, put them back into a prompt, describe how they got the training examples wrong, and then have sonnet update. its Python transformation. So basically generating a ton of Python programs, like thousands per example. Again, you Kind of had to do this because you had to make up for the fact that the models weren't thinking, they were just basically blurting out. I prompted them a chain of thought, but again, this was pre, RL chain of thought. the models were not really thinking for themselves. so that actually worked pretty well in ARC V one. The problem is ARC V two is much more complicated. Again, humans can solve it easily, but, it's very hard to describe ARC V two tasks with Python. how do you describe, oh, this model must be symmetrical, and if it has three holes in one and two holes in another, then the top right should be orange, right? Python starts to get quite brittle. And so this was the problem, going from RV one to RP two. And so what I discovered is, we can swap Python for natural language.

what I wanted to say.

I think it's gonna be, I think it's great to give some introduction to what is so hard exactly on these, questions. I was heavy on arc one, so I just know, intuitively why is it so hard and so easy to humans. But I think it's gonna be really interesting hearing from you because, yeah, just to give people perspective, like why is this so hard? It's just some, grid of colors, but it is so.

Yeah.

So first of all, I would say it, it literally isn't hard, right? for humans it is not hard. So what's weird about language models is when they're not trained on a certain distribution, they are not great at generalizing. And I think this is the core of Francois's. thesis, but it's also a thesis that I generally agree with, which is today when we train language models, they're very good on distributions they were trained in, but there aren't many, grids online and there aren't many examples of solving grids. And then finally, the test set is totally withdrawn from the internet, right? So, you can't train on the answers, for arc a GI. And so what you find is language models when you're asking questions outside of the distribution are very bad at giving coherent, reasoning traces. And this is actually the fundamental, blocker to general intelligence, which is when you ask models questions outside of the train distribution, it's almost like an adversarial example. They struggle to reason about it. And I think that's because. the reasoning that language models are trained to do is always reasoning in the domains that they were trained on. So, they can reason really well in math. They can reason really well in science. but it's like that reasoning is fused to that domain, which is unlike humans, right? We have this reasoning engine which generalizes, which means that, you know, you take Einstein, you give him a task he's never seen before, totally out of his distribution. He'll still reason about it properly and logically and coherently. That is not the case for language models today. And, so fundamentally, why is it hard for language models is because arc a GI grids are not in the, the trained distribution. And, again, fundamentally like what it means to have general intelligence is that you're able to adapt to situations and data that you've never seen before. and this, gets to the heart of a GI, and RKGI. And so, yeah, fundamentally I think that is why it's so hard for them. but one interesting thing is language models are climbing, right? They're getting better at this, but they're not, getting better at the same rate they are in, science, math, other domains. And so I think the next big frontier is how can we basically inject like reasoning that will actually generalize in these models and not just, be excellent in a specific domain.

So, RGGI V two was definitely more difficult.

And then I think you were starting saying you switched from pro programming Python into Kind of, like a language. But you also switched the models because the models grew since the last time you tried it, right? Like the models are, well, there were multimodel when you started this already, but there were no test time compute. But now the models are significantly better at like generalization. And so you chose grok for this one. How did you get to choosing gr? Did you try multiple ones? tell us about the process of getting to grok because, space Uncle, as we like to call him here on the show, really liked the fact that you chose Grok and it was like, yeah, grok five is testing, blah, blah, blah. So he really leaned into the fact that like grok, he likes when grok wins versus other labs. And in this case it did win significantly. how do you choose this? What other models did you try with? I would love to hear, meanwhile, folks who are just listening, I'm trying to solve example 18 of the thousand, easy set and it's very easy, just like you need to figure out the template for humans, very easy. I can realize how for models this is confusing as hell. So please, please come up.

So the first thing I'll say is this is not a multimodal challenge.

The way I did it, I actually started by giving base 16 images of grids as well as their ASCI representations to the model that did not help performance. So I actually, don't give images of grids, anymore. So, so my solution just gi they just see the ASCI grids, like two dimensional in plain text grids. I tested every single model, every thinking model, GPT five, high opus four one, thinking everything it happens to be that GR four has really robust. long form reasoning coherence on tasks. It wasn't trained on, that I, you know, I'm actually not exactly sure why there must be something in their training distribution, that has allowed them to do this. Or maybe that they've scaled RL in such a way that, they've, they've been able to Kind of, like inject this, these reasoning circuits in, in various places in the weights. But yeah, so really just, empirically, GR four is basically double the performance on thinking about and reasoning about grid transformations. So it's, and you can see that actually in the raw scores of GR four versus GPT five high. It's about double, I think on RP two.

It's stunning.

I remember when they posted, graph four Kind of training stuff. You guys remember this? They showed how much RL was like, went into graph three or like three point whatever, and then they showed graph four. RL was like half of the whole GPU time that spent on rack four and looks like it resulted in, you know, graph four is also some folks code with it. It's great for code, some folks like chat with it, et cetera. But on the actual tasks, like I think folks are still getting to maybe sonet, It is, but mostly it's between sonet, like open AI now analogize with Codex. but it, it does look like they have cracked some stuff there, towards like visual and intelligence. But you're saying, just to make sure that I understand you. You used to send multimodal images and grids so the model could see them and try to figure them out. Now you're sending A-J-S-A-N representation of what needs to be done, and then the output you get is also like programmatic. And then you just convert it into, like how does the conversion to text happen, where it happens and how the conversion back happens when you submit.

Yeah.

So you get the input as JSON grid, right? Mm-hmm. this is represented as numbers, you know, JSON 2D array, yeah, 2D inte array. And that's how I give it to the model. it basically performance, increases a little bit when you give it the grid, the actual base 64 image as well. Mm-hmm. But the problem is it's much more expensive and this is, partially a game of compute efficiency. it just was not worth, sending the image in the end. fundamentally the bottleneck is not that they don't, see the grid well, it's that they are not able to reason well over the grid. And, you know, you can see, when you look at the traces, they're weirdly logically incoherent. It's really like an adversarial example, because it's outta their training distribution and they're like, oh, I don't know how to reason about this.

Because it says like $8 per task, but I'm assuming like at least a

hundred tasks or something like this.

So yeah, there are a hundred tasks and it was

around $8 per task on RV one.

All righty.

and so we're seeing on the screen Kind of, the, the revision prompt example. This is Kind of, what you submitted, what you arrived at. could you tell us what specific prompt engineering you got, iterating through this? What's things you noticed that work for Dale? What don't work for other things? I'm sure people would like to learn from your experience, getting the model to do this for you.

Yeah.

I find just being really clear and concise with the instructions are best. And then, you know, you'll try out a few, you'll realize, oh, this line needs to be updated a bit. It was really a lot of it, iteration, right? Like, started with a very small prompt and then just gradually added lines and lines until the, performance was starting to ascent tote. And then at that point, did you write

Did you write the prompt yourself or did you

have AI help you improve it

No.

DSPY, I started writing it myself and I'm sure maybe it's like, 75%, written by ai just because I'm iterating, with, GPT five open next to me. correct. But yeah, I mean, it's more or less written by me

GBT five, prompt iteration.

And so we called out on the show before and definitely folks should know. GBT five is an incredible prompt engineer, incredible for itself and other models as well, especially if you give it examples of how you want things to look at. Jeremy, I wanted to ask you, maybe you saw this, maybe you don't. Christian sei, do you know the guy Christian sei? Do you know who this is? he used to work at, I know, yes. XAI. he had, and I keep busting about this 'cause I just absolutely love this. He basically had almost a competition with Francois Sole about how we'll see these models go forward. And, recently I think they released Mathematica and I think we covered it on the show. I'm gonna try to pull up the tweet. He had, a public bet with Francois, I don't know if they actually got to a bet, but he had a public statement that by the end of 2025, we'll see the current set of RKGI, fully saturated. And it does seem like he's on track with the, with Kind of, the releases that you had. because, you are, you're getting at 80% with the current iteration of grok. I dunno if GR five is gonna drop by the end of 2025, but like I know that they keep training this, I know that like other labs are definitely fighting for, for this. Do you think that his prediction is likely to to be true? Do you think that like we'll get to a full saturation of at least the, the first, the first situation

of ARC V one?

I think that's likely, yeah. And by full saturation, do we mean like, 85%

said, said, 95%.

This is 95%.

I wouldn't be surprised on ARC V one.

in ARC V one, the hardest challenge is just not that hard. And, so the search space for finding the right transformation function is small enough that I think yeah, with a few percentage point, differences, you know, updates to grok or GPT five thinking we should be able to get there. It's also worth noting I left a decent amount of compute on the table. You're technically allowed to spend $10,000 of compute. I guess I spent 800, right? So, I probably would've inked out a few more percentage points if I maxed out the compute. But for, a bunch of reasons, actually, mostly the X-I-A-P-I stability, I didn't do that.

Wow.

So if, Elon Musk is listening to this, give Jeremy some more extra credits, we wanna see that higher score getting there.

To your point about DSPI, and the prompt engineering,

these tasks are quite expensive to run. I think DSPI would work well on, less expensive problems. you could easily, if you're running in like an automated loop, you could be spending thousands and thousands of dollars just to optimize a few lines. for this task in particular, you really have to Kind of. it's almost like an art, right? you only get so many, runs before you run outta money.

so that's why to sponsor Go ahead.

Ni

I hope you recorded the data in and out, because then

you can try and train a small model. Like I think the challenge will end up going to like, what's the phone or the one B model that can actually solve it next 80% is pretty much the benchmark defining a benchmark. But it would be interesting to see how much you can make, like what's the smallest model you can make the scores above 50%.

Jeremy, just before you Kind of answer this, just to give you

more context, at the beginning of the show we talked about Isaac, which is a 2 billion parameter model that does like incredible things compared to like GPT. Five, I wanna say, in pointing, and we also talked about a Finetune deep research, like a 3 billion active parameter model that does deep research at the level of like GG GPT five as well. We're absolutely in the realm of, you know, smaller models getting incredible on specific tasks with ael. And so like, I think this is the context of instance question. do you, predict that something like the task that you have been doing with grok, given that it's very specialized and there is a reward function actually, that you can run, you can literally test whatever the outs and see if it makes sense? do you predict that smaller models will be able to do this, in a very specific way?

Yeah.

So, first of all, I can say for testing every single, input output, when I'm testing it, is saved to my Postgres database. So I have, you know, like a hundred, almost maybe a million traces at this point, for this exact idea. I personally, when I've trained the small models have found that they're just not good enough at all, which is interesting. but I plan on, doing this as a more serious project at some point in the future. so I think I actually am not so optimistic about solving arc. I think you probably need 70 B model or maybe some, Moes, with, fewer active parameters. I think they have the capacity to do that maybe by the end of this year to get, you know, frontier scores. I think there's a lot of. the fundamentally the tasks are not hard and you just need, deductive coherence over, a paragraph of text. So for sure, two B models are physically capable of doing it. whether or not we have the right training techniques is another question. So I would say I'm, pessimistic for the two B model to be able to beat Arc V one this year, but, I could absolutely see in the future how that's possible. it would also certainly be overfit.

do they need to have very, very long reasoning traces?

I think you mentioned that was the like long reasoning coherency. That's probably why the small mos don't have enough meat in them to just carry that on or

Right.

I think they don't have enough. their logic circuits are just not, baked.

Awesome.

Well, Jeremy, we're coming up on two hours on the show. We wanted to first of all congratulate you on this incredible results. Second of all, thank you for coming and telling us how it's actually done. I think a lot of folks really wanted to know you. You had a great blog post. We'll add this to the show notes. The folks can let go and dive in. We have this button here with applause for folks who open source data, open source models, If you do choose so, and you wanna come back and say, Hey, you know, the data is not open source. You guys can train and do RL, definitely, the button is warm and ready for some applause. thank you so much for coming and telling us about how this is done. It's very, very interesting. we heard about RKGI, multiple companies are using this to show the out of distribution thinking for a context, for their models. And it's great to see how the sausage is made. I appreciate your time here. Thank you for coming up. We will add your blog post and, anything you want to, anyone you wanna shout out, anything you wanna say where folks can find you, et cetera, feel free to kinda like,

No, thank you.

appreciate it. And also the code is open source that I used, so you can find it on GitHub. if you're interested in post-training, pre-training, frontier Language models, definitely check out reflection ai, DM me or, apply on our website.

Amazing.

Awesome. Jeremy. Thank you so much. And folks, we have just a little bit over, we're a little bit over two hours, but now we have a few more things to talk about, including examples. But, I will just point fingers at some stuff that you guys definitely need to check out. nv uh, er and OpenAI together released this like incredible paper, like super in-depth paper, of how people use Charge GPT. It's, it's timed at July, like they issued it at September, 2025, but it's timed at like, from November 22 to July, 2025. So almost like two and a half years about early adopters. And they have a bunch of cool insights here. Like early adopters were disproportionately male, but the gender gap has never dramatically like a bunch of stuff like this. And I wanna just shout out a few things. There's this graph that shows the distribution of tasks. Nisten, we talked about this graph, publicly somewhere. this is 700 million weekly users back in July, and we're extrapolating, we're saying like, OpenAIs is about probably to hit like a billion users, weekly users of Chat GPT by the end of this year that the projections are there as well. And this graph, like the width of the graph is as how many proportionate segments there is. So like 28% is practical guidance. 6% is multimedia or something. 7.5% technical help tasks. And then each Kind of column is broken down to like category. So, tutoring or teaching is like 10% of our all uses of Chat GPT. So that's like 70 million people weekly using Chat GPT for tutoring. How to advise is 8.5%. there's, create an images 4.2%, like I'm assuming that will grow as well. games and role play is only 0.4%, so not many people are playing games on role play with ai. but relationships and personal reflection is 1.9% isn't. You talked about this and it seems low for Chat GPT, but calculating this, it's around 70,000, 70 million people weekly having relationships and personal discussions, with ChatGPT. So that's, that's, that's quite a lot. Anything that you guys remember standing out when people talked about, the use of ChatGPT?

That's pretty crazy.

The entire thing. I mean, I think, it pretty much shows that we are in a bubble, you know, us using AI and so on, and you actually see, these over the distribution of, I don't know, the entire population is pretty much, the case 700, weekly active users is, a lot of people and things are surprising. I mean, that's pretty surprising. I wouldn't imagine that, these are the usages that on the first, the first thing that people do, like near to what, what was it, that 28%? what was the guidance or,

Yeah.

That's surprising.

Yeah.

I mean,

I like comments that, that, well, that shows that, that people

are using what, what the models are good for, and I just wanted to say, I was trolling about telling people that they're talking to the lop because I, I'm even making companion apps for seniors with dementia and, trying to. Figure out how to keep them in the conversation. so, yeah, my surprise here is that they've been trained to be companions, but not a lot of people use them that way. So I just think that is interesting, as, nobody's really getting up and, just telling it, hi, how's your day? Get my plans done. This is how everyone imagined they would work, including myself, but that's not how we're using it. We're just Kind of using them as smart search engines or task delegation, type of tools and problem solvers. So, yeah, it's pretty, these things are hard. It's hard to predict how users are gonna use it, but overall I am happy that people are using 'em for problem solving. That that's what they're best at.

It's hard to predict how they're gonna use them, but

trajectory wise, it's easy to predict. So here's the, here's the one line that, you know, the, the summary gave me. JGBT traffic is on pace to rival Google search in daily message volume as early as next year. seven 50 million weekly active users in July. And like, if you remember, July is before return to school. So now everybody returns to school. They need to do homework. they go back to these AI more often than they want. message volume hit 2.6 billion per day as of June six times bigger than the year before. It's absolutely insane. and, you know, the usage is doubling every six to eight months, since launch.

Absolutely insane.

Yeah. I'm surprised not to see larger percentages, people talking about personal stuff with chat g pity. Yeah. we hear about it a lot and pretty much anyone you talk to, feel that, you know, they ask Chat GPT or any other AI personal stuff, and we don't see this, too much in the data. what I really like the most, the vast majority of people use it for things like, how do I do like Kind of? I need to know how to do something. I took a picture of a sink and asked tragic, how do I fix this? And so on, because I just, yeah, I just don't know. And I mean, it's not the thing I do, but I managed to fix it, to fix it because I don't know, all three just told me how to do it and that's great. moreover, search absolutely amazing to see how huge percentage of used usage is just search or how do I do the thing, which is basically also search and editing. Yeah, I think. Pretty much half of all of ChatGPT usage is just me asking it to fix my broken English. that's a lot. That's a huge percentage. I hope you duplicate that.

ChatGPT, when they added memory features, I asked

Chat GPT, what are the most embarrassing things I asked you? And he told me, bro, you, you, you could boil an egg. You don't have to. Tap into the intelligence, you know, of the world to like learn the boiling egg is like 10 minutes in boiling water. All right, folks, we have to move on because, we are already past time, but it's just like I, it's impossible not to tell you about some of this stuff, right? So we'll just run through super quick, about incredible breakthroughs, We mentioned this multiple times already, but just for folks who are still with us, OpenAI, aced the 2025 ICPC World finals. This is the, reasoning, algorithmic competition. This is like super, super hard. they have five hour, constraints to build the solutions. And, basically we've got to point where like GPT five on its own is super, super human level at the best. Coding people in the world compete at this competition and they are not always able to solve all questions within the five hour limit. and they, this year they had like an AI separate track. So virtual first place, not actual first place. GPT five, was together paid with the experimental reasoning model, which we're probably gonna get released at some soon. And GBT five on its own, the regular GBT five, the one that we have access to it, solved 11 out of 12 of the strongest, hardest reasoning problems. This is like. Olympic level competition for algorithmic code. It's incredible. Like everybody who from OpenAI who talks about how hard this is and everybody not from OpenAI like, like, you know, math olympiads and people like this, they're all, balking and they're like, this is just unbelievably hard and it's incredible achievement that we got there. so we got two superhuman coding levels with the levels. The same model that Yam uses to fix his correct grammar. Now wins 11, or 12 is the hardest, hardware, software, and algorithmic, questions. It's quite incredible. super speed updates on ChatGPT as well. they have released, some conversation about, teenage use as well as they're thinking about getting to Google level. And, you know, like we know that like Zach and his billions of users also had teenage problems. Wolf mentioned this in the beginning. We'll add this to the show notes if you're interested in how OpenAIs is going to treat, usage while giving you as an adult more power to use, like, you know, treat adults as adults and not just refuse anything. they're also thinking about how the teenage, use in the apps Kind of goes against their, principles of,

he also said something very important, I think,

because he compared conversations with AI to conversations with, medical or legal, specialist. So that is an important concept. I hope that is more. getting more traction because people are opening up to AI so much and having all these discussions. It, needs specific, special protection.

Yep.

And then the last thing, update on open air. Thank you all from, by the way, and, open air adds thinking budgets to ChatGPT app. So now when you use ChatGPT five thinking, you can decide if you want like standard or extended. If you are pro tier, you also get light and heavy thinking modes, basically, fairly resembling the API, although API had three and now they have, four, four versions. because they heard complaints that people say that GPT five are thinking for too long. And it does sometimes. And you want the best intelligence. You don't want the router, auto, you want it to think, but you don't want it to think for like five minutes for. How long to boil an egg, et cetera. So updates to GBT five app folks. the last, I will skip Huo, just because we don't have time for this. so Huo is this human-centric multimodal video, example from, Biden's. It's incredible because it's still very small. it's a small model and it only generates like four seconds of video, but it's very general in the sense where like you can provide sound and you can provide text, and I think multimodel video models, human centric video models so far, you couldn't provide text as well. So you will provide the frame of the first image and they will take this image and animate this image as though from that frame. This model does it differently and it's very interesting to know how it understands the subject reference from this image, but doesn't necessarily treat this as the first frame. So you can take a picture of me and put it in a scene somewhere or replace somebody else, but it doesn't treat me as the first frame. So it doesn't only animate this, it's like a unique approach. And then you can also change it with text, You cannot give it like me speaking and have it animate. Like we have, you know, the animated, faces, so like head dry and, and, others. So this is Kind of, a combination of all of this, but it only works on like very short segments so far. But the direction is super cool and I think we'll see, incredible stuff from that space. so this is, Huo from By and it's open source and they gave the recipe to folks. The last thing that we have to talk about is, world Labs. Presenting Marble. Marble is just absolutely incredible. We talked about World Labs before we showed their examples. But the fidelity of this, the recent release is just absolutely incredible. Absolutely incredible. You can see from one image or maybe a 360 image, a full on world generated. we talked about Gen three before, so this is. Lower fidelity than Genie four. Genie four is like VO three level a world that you can control and play, that was built on simulating Kind of game. So that feels like a game. You can jump, you have a character, et cetera. this is one picture to generate a world and the consistency in this is just absolutely remarkable. and in order to show this, I actually got access and I'm very, very happy that I did, and I took Wolfram's background. Wolfram's sitting in this, very cool Kind of, like spaceshipy type thing. So I asked him in the beginning of the show to send me an image and behind the scenes I ran this with Ma Marble all one mini, the lower version of the world. So let's take a look at the world. Okay, so basically I provided one image, and you can see this image of a spaceship. You can see the lower quality because I use the lower quality model.

That's 3D this is 3D model.

I see.

no, no, it is.

See you can walk around objects. You can see it's not a 3D model. No, no, no, no. Yeah, it's a, it's a, it's a generative model that that generates gian splats. So go gian splat is Kind of like how to create like a 3D scene from a thing. So you can see if we like leave the perspective that I provided it, you can see that like we can see a bunch of artifacts. So we can see like what's behind there. I do believe that this thing, well from this thing behind was fully generated from this thing upfront. Yeah, exactly right. So like, this is the perspective of, of where you sit. and then this smaller turn around, right? This is, this look like still really cool, really cool. And I think the, the persistence of the fact that like the, the chairs are, I'm sorry for going all, all the place. The chairs are the same place where the chairs are after you like go around. So this is like, it generates a 3D scene based on one image and you can provide like 3D images. The thing that I wanted to show, I wanna see if I have it. folks are using this to create very cool Kind of 3D consistent scenes. Because now if you don't move too much, you can have like a very consistent like character scenes and you can provide, like different angles to those scenes and then provide this back to like sea dream or something. very, very cool. so this is world Blobs Marble and we've talked about VE and we've talked about Rio. I think we've covered pretty much everything folks. if you missed any part of the show, just a reminder, we're available as a podcast on Thursday, I news on Substack and everywhere you get your podcast, so you can really listen to a conversation if you missed it with Jeremy or our analysis of things, or the things that we did. the recording is also up on YouTube as well. We're here every Thursday. We're gonna be here next week to talk about whatever crazy things that happen. Huge thanks. Goes out to with biases, from CoreWeave, the sponsor of this podcast for the past two, almost two years. And, huge thanks goes to Wolfram, Yam, Peleg, and Nisten for, being here from week to week and telling you guys all about the Kind of, the world of ai as we follow it. So, with this, I will just say, as we were talking, Wolfram, here's your 3D world and we're gonna just like leave it at that. And it's, it's really cool and it's really, really cool. 'cause it, they like, it generated all the, all the images. All righty folks, this has been, Thursday. I, and we'll see you here next week. Bye-bye.