ThursdAI - Apr 17 - OpenAI o3 is SOTA llm, o4-mini, 4.1, mini, nano, G. Flash 2.5, Kling 2.0 and 🐬 Gemma? Huge AI week + A2A protocol interview

Welcome everyone to Thursday.

I for April 17th and this has been a week folks. This has been a week. I've been telling folks that this is our third live stream. And, I don't remember another week. There was live stream for three times for, two hours. So a lot to talk about. Today's show is a very interesting one because also we have an interview with someone you don't want to miss. You absolutely don't want to miss this. We have an interview with Todd Segal. Todd Segal is a, software engineer extraordinaire at Google. I believe his staff. that, He works on the A2A protocol. We've talked about the A2A protocol a bit before, and, Now it's time to us to actually interview somebody that works on the protocol, which is, I think is, is great. So if you heard us talk about MCP, we're probably going to mention MCP in this show this week as well, because that's on everybody's mind. A2A protocol from Google is agent to agent protocol. we briefly covered this, last week, I believe. But we definitely want to go deeper. And as we go deeper, we'll realize a few very interesting things about this, specifically that, they can coexist with MCP, but also. This is the protocol on top of which agent communication is probably going to be built and Google launched it with 50 partners. Waits and Biosys is one of those partners as well. And I'm very excited about this interview. I think that you guys will get a front row seat to a conversation about a protocol that's as likely and likely as likely to blow up as much as MCP did. Which is, I think, great. welcoming also to the stage, our friend Yann Peleg. Yann, welcome. How are you, man? How was your, third, third live stream this week, man? Third live stream. We've been, we've been very online.

What a week.

What a week. Crazy.

folks, just before we begin with the TLDR, just as kind of

folks are coming in, I want to reflect on the first, what, three and a half months, almost four months of this year. Do you feel it's crazier regular pace as expected? I want to hear from you guys, because I feel

Absolutely crazy.

Absolutely. It's much faster. There is no, there is no question. It's crazy. Seriously. Like we haven't seen a model after model after model from like the same players every month. like I think we really got to the point where they are all, training, continuously training the models, making them stronger and like release checkpoints, over time, just. Every couple of weeks and you just see it like there are new Gemini models. Already in testing and like we got Gemini, I think two weeks ago, it's just crazy, absolutely crazy. And the thing is that everything is like, things are novel. We didn't have this image generation before. It's not just a little bit better. it's, it's, we didn't have this before. and I didn't even start talking about those three, which is. I don't know, I'm, I have one day of experience with it and it's absolutely crazy, in my opinion.

I'm in love.

I, I, I've used, the thing, Yam, to reflect on what you're saying. We do this every week, and we've done this for the past two years, and I remember weeks, GPT 4 was launched, and then, 06. 13, if you guys remember, GPT 4 would, the first one with function calling was a big one. Mistral launched somewhere during, the, the, the, the fall, and then, a bunch of, follow up, fine tunes, whatever. There has been, like, a lot, a lot of stuff, from the community. I don't remember such a insane stream of updates with the big guys just like fighting each other on the arena of attention and this is how it feels like this is the arena of attention you know it's really funny because of the transformers thing but it's it's our attention not the model's attention that they're fighting for and it really does feel like there's a reason why metal released llama for on a Saturday. That reason is likely because then, Google had their own thing in Google Next. And then OpenAI always fights it out with Oh, they're about to release something. And they're fighting for our attention. And we are in the business of we, me, Wolfram, Yam, Nisten, like all of us, We're in the business of delivering this news to you guys And we also need to know about them to deliver this news to you guys And it's been harder and harder consistently. I'm very happy that we're able to do this as well Like I said last week. I always my Lady of the heart. I always like not complain but like after I summarize everything I was like hey I have no idea how we're gonna cover everything this week somehow somehow we get to and we we're able to but it's been consistently more and more and more. After our filter, we also filter stuff. Folks, you should absolutely know nobody can deliver everything to you. Like we don't cover startups. For example, we don't cover valuations barely, unless it's like Mira Moratti's 2 billion supported seed. That's like the biggest in the world. We barely cover like sales. We barely cover, we're probably going to talk about opening and windsurf, but we are not focusing on that part of AI at all. We also don't tell you about tools we tell you about specifically. Kind of foundational labs and releases, the things you can do right now, but there's so many tools out there that are also like getting launched every week. Like I'm using Aqua voice, for example, and shout out to Aqua voice became my new text to speech thing. So there's like just so much that we are aren't able to cover. And there's so just the stream is crazy. With that said, follow us on Twitter, follow us on XA hundred percent. With that said, we cover even more following Wolf from following me. Yam, we, we do talk about those in the ether. It just, this, my point is it doesn't make the cut to the show because otherwise it would be a five hour show and if, if Yam, if we're summarizing the, the live streams of this week, we had the one point. Have one and a half hour on Monday. One and a half hour yesterday. That's three hours. We're going to do two more today. That's five hours of us talking to you folks. So I really, really appreciate everybody who tunes in to this. but also I do feel, especially with the old three releases, that. It's just like breakthrough after breakthrough after breakthrough. And it's been incredible to see.

So you can feel the acceleration, but at the same

time, we are getting used to it. That is also an effect. So we are getting these great models and okay, our mind is blown, but we are used to it by now in some ways. I would be even more out of our minds. I became a

goldfish for sure.

Like I, I do forget the excitement of before I just posted the thing. Do you guys know the Indian meme where somebody crossed somebody's name. It was like, this is new, my best friend now. So I did this with all three as well, because it's just a week ago. We got excited by, I don't know, 4. 5, a couple of weeks ago, right? 4. 5, the biggest train of the biggest model that OpenAI ever did. And now it's deprecated our IP for 4. 5, at least in the API. but, the thing that I also wanted to say is that we're also accelerating ourselves and our capabilities with these models. Like yesterday, and I will tell you straight up, I used the opportunity to prepare for the show with O3 and O3 has the capability of reasoning over the URLs and reasoning over like searches and I literally, there's a standard and you, you guys already see the standard in kind of the notes that I prepare, all three helped me through a lot of it and it was incredible. we're also catching up because we now have access to these tools and I think it's very important for the audience to also like. Realize we're not the same as we were two years ago as well. Our capacity to process information also increases, which is great. I don't have yet. And I think in 2025 in April, I should have is agents running all the time and doing things for me. I would like to invite the folks in the audience to tell me if you have agents running all the time. I have agents when I need them. and like other places, I really need. like ephemeral continuous background agents to pop up news for me. I do have them in the form of the community folks like Lincoln, for example, shout out to Lincoln again, Colleen and like other folks, Ryan, that I see in the space and, and folks are agendic on their own because they're people and then they post and link and tag and I can follow them, et cetera. but I need like specifically trained AI agents to go and do some summaries for me so that on Thursday I can just show up and just talk about this. I think with this very nice intro and excitement about kind of the speed of improvements, I think we have enough of folks to start with the TLDR. What do you guys think? Let's go. Let's go. All right, so we have, let's go to the TLDR. Basically, TLDR for folks who are new to ThursdayEye, is everything we're going to talk about, or at least that we know of before breaking news, which we love that folks in the audience will just tell us about breaking news. here's everything that we're going to talk about on ThursdayEye. That we know of.

Alrighty, folks. Welcome. you already know this. This is my notes for Thursday. I, April 17th, as a reminder, the show, if you missed it. Any part of the show. You can listen briefly to this 10 minute segment, hopefully in the minutes. And then remember, Oh, I wanted to hear about cling and then come back to the show. The show is also posted on every news podcast platform like Apple and Spotify. And also the newsletter is posted on Substack, Substack, Substack, Substack, Substack, Substack. For X specifically this was for X and then you can go on Substack and all the links will be there as well So you don't have to miss anything. You don't have to write anything down. I got you Just just subscribe to the newsletter along with I think we're crossed 6, 000 subscribers soon. I think we're very close to 6, 000 subscribers, which is crazy. and then, very important. Do not miss today at 10 a. m. Pacific time, which would be in an hour and 15 minutes or so. we'll have a chat about A2A protocol with Google's Todd Segal. you don't want to miss this. I really am excited about this, because A2A is about to be like the biggest, news very, very soon. And you heard it here, hopefully first, or maybe one of the first, definitely one of the first interviews about A2A. today on the show, the hosts are Wolfram and Yam. We may have other hosts joining down the line, distinguished, co hosts of Thursday Eye for the past two years. it's been great to have you guys as well. we will of course, start. to cover with the major news of this week, maybe this month, maybe the past couple of months, even maybe since December when it was announced, feels like it feels like it, all three From OpenAI launches, O3 is the flagship reasoner, the biggest reasoner model, test time compute and everything, that was announced in the 12 days of Christmas. And, O3 is now available to all of us, all of us, I think. and also O4 mini is available in charge of the NAPI. We're, we're going to cover, We covered this for a year, an hour and a half yesterday, but then we played with it since then. Pretty much didn't go to sleep, I think. and we will definitely talk about O3. We'll, we'll cover some stuff. I already have some breaking news through Yam. Yam, thank you. About the long context performance, which is insane. Absolutely insane. You don't want to miss our general session about O3. Remember before three, like months before three, essentially two days before three, OpenAI also launched GPT 4. 1 and 4. 1 Mini and 4. 1 Nano. You remember that? That was this week. That was this week on Monday. It's crazy. We also had an hour and a half live stream and coverage, and we must tell you about 4. Mini. I have some evals. Wolfram, you have some evals. That model is crazy good, crazy good. Compared to the fact that it's not thinking. We shouldn't let the thinkers and reasoners take the wind out of it. There's 4. 1 mini and the prompting sandwich stuff that we like covered. All of those like things we absolutely will get super excited about. I'm already getting excited. I should move on because it's just the TLDR. Mistral releases something called the classifiers factory. It's pretty cool for developers. We're going to cover this maybe in just a little bit. and tropic Claude adds research, not deep research, but they do do deep research, only for the Max Pay tier for now. So I actually, folks who just paid for on Tropic didn't get this. But I also have a workspace. Google Workspace integration and Tropic Cloud can now through the UI reason about all your emails or your docs and, and do a, like a research thing. They didn't call this deep research, but hey. It's effective what it is. and Cohere released a new embed model. That's like state of the art multi modal embeddings. We talked about multi modal embeddings before with the folks from Gina, which released it in open source. Cohere releases embed for some multi modal as well. Multi modal embeddings are great because we are getting multi modal across the board. As we move to open source LLMs, which was less of an exciting week. Sometimes it's very exciting. This category in this category, I think the two more exciting things that are going to come very soon. not this week though, is open the eye when they finally open source something. and they had an open source event with developers in San Francisco. They're, they're about to drop something. That's going to blow us away and we're going to talk only about this, but also DeepSeq at some point. Where's DeepSeq folks? we haven't heard from Alibaba Coen in a while. Like this category is going to get exciting very, very soon. Not this week though. This week is all about the Big Labs O3 itself. However, there's some stuff. So OpenAI actually open sourced, OpenAI actually open sourced some stuff this week, not on models, but But they open sourced a new benchmark called MRCR for long context. And they did definitely do breakthroughs with, with long context, which we're going to cover. Microsoft released BitNet 1. 5, which is like a small thing we should, we should mention as well. Prime Intellect trained a 32 billion parameter globally distributed reinforcement learning experiment. We talked with Prime Intellect, before, so this is Intellect 2, and this is A, RL, and they're like looking at reasoning there, distributed. We should mention also the chat GLM, which we've covered. The GLM models is rebranded to z. ai, z. ai depends on if you're British or not. I don't know how to pronounce this. Let's say z, z. ai. and they also have a open sourced, GLM for, 0. 4. 14, which is like literally from April and open source family. In this week's buzz, which I, I now add MCP and now A2A as well to this week's buzz, because this is what I'm leading in, in, in points and biases. we want to mention, first of all, we're going to have a chat with Todd Segal about the A2A protocol. And also I want to mention the, the Weave Playground support for the latest GPT models is there, and I'll show you how cool it is to use the Playground, to compare between those models, to maybe even Grab one, trace from production and then retry this with the new models to see how well it's going to work. I'm going to actually show you this. I've been, I recorded the video about this, but it's, it's really, really cool and useful. Vision and video category is also heating up this week. VO2 from Google, VO2. We've talked about VO2. We showed you videos of VO2. has hit ga, which means that you now can use this, it's free in the, Gemini app. It's now free. Google just kind of like gives things for free because they have a lot of TPUs. so VO two is now free in the AI studio as well. You can just go there, you can upload the images, so it's, it's, text to video and image the image to video as well. And combine text to text and image to video as well. so it's now in GA and it's. It's one of the best still video, video generation models, together with them for some reason, I stayed up very late one night and then cling the folks, the, the, the released, multiple video models at this point have released cling 2. 0. And they also released, an image generation model that they claim state of the art on top of other models. and basically a whole creative suite of things in cling. I would love to show you. It's super, super cool. They're trying to be the holistic place where you go and do all of your AI creation needs from, from the art perspective and also. We should chat about by dance, announced. I don't think they released it yet, but because of the seven B, I assume that they're going to release this by dance, publicize seaweed, seven B a video generation foundational model. There's also one other thing here in video that I didn't add to notes yet that one X, the Alibaba video model has released a last frame, first frame model as well. these models are very useful because you can add the first frame, last frame, and then you can take another video, take the first time of it and continue changing them together. In voice and audio, in the most incredible news that I ever heard, maybe, Google released Dolphin Jemma. And we have to, we have to talk about Dolphin Jemma, at least a little bit. Because, they didn't release this on April 1st, they released it a little bit after, on actual, World Dolphin Day or something. Google tackles dolphin communication. And, I find it incredible that if this age of intelligence will enable us to talk to other intelligences like dolphins. I, I chatted with Wolfram that we wanted this whole segment to just make dolphin noises for one minute and then see what's up. so Dolphin Gemma, we're going to chat about this as well. In AI art and diffusion. Biden's took over the category in artificial analysis, at least, of the image generation models, beating even the Sora generation, from now on, by the way, I def, I refer to chat GPT's native image capabilities as Sora image. because you can also access it on Sora. And also, I don't want to keep saying chatty between native image generation. That's not Dali. there's no, they didn't brand this. So for me, it's already images. and then, they're incredible still, but according to the, few of the artificial analysis leaderboards, see dream three from, by dance, bilingual image diffusion is now the top. AI art and diffusion model around the world, which is crazy. in the tools section, OpenAI together with everything debut debuted Codex Codex is ACL I. That's very similar to cloud code that we mentioned before on the show and we've been using it. Codex previously used to be a model from Open AI that does code Since then, obviously those capabilities were rolled into the main models. so they brought back and I'm very happy about this, the Codex brand name and now it's a CLI that I've used this week. It's pretty cool. And we're going to talk about this. the only thing I have here is codex is like a lot of code, but actually it's not. And also, the use of all three, I believe all four mini and for one and Windsurf, is free for the next week. I definitely want to shout out Windsurf. not to mention the, the, the Varun, I believe his name is from Windsurf showed up as well. Add the 4. 1 event that we've seen. And I was like, Hmm, that's very interesting. They used to be friendly with Kirsten and now the Windsor family. And then there's rumors about, open AI is rumored to buy Windsor at around 3 billion valuation, which is insane. so Codium AKA Windsor, there's rumors about this as well. Obviously to remind we had Kevin. from Windsurf on the show and which added about why Windsurf is so dope. And since then, I've been using predominantly Windsurf. Even though, it's a mess, it pays for my cursor. I went and paid for Windsurf and I've been very much enjoying this as well. I believe that this is everything that we're going to cover, hopefully, because also we have an interview for at least like 20 minutes with Todd Segal about A2A. Folks, anything else I missed? Folks in the, in the host panel, welcome this and by the way, folks who are listening to us, anything else huge from this week that I missed that we must absolutely cover? And if not, we can get started.

There are so many things we can, we can talk about every topic

here is, I don't know, the model you talked about agents running full time. You talked about cursor versus Windsor. Every single one of these can be like an hour. It's absolutely crazy. The pace. it's great, but it's absolutely crazy.

So speaking of pace, let's keep this show on pace as well.

And let's get started with the big companies, APIs for which I don't have a transition, unfortunately. I wish I had no transition for a big company's APIs, but we will just do this. AI breaking news coming at you only on Thursday night. And the reason I use the breaking news button is not because we have breaking news from literally right now is because it's been breaking this whole week. It's been breaking. We've been following the breaking news and we've jumped on live streams that we didn't know about before. and this thing is looking tired because we didn't sleep also. yes, we have followed the releases from open AI and we'll just cover, I think let's start with the smaller one and then cover the bigger one that that's how it feels to me like it's the fair way of doing this. so we briefly mentioned kind of the smaller potatoes in the big companies, cloud ads research. So basically now the search cloud historically was very slow to adopt like search to its, its UI and then tools, et cetera. the artifacts was great. And they, they, they kinda, went first on that. but the cloud AI, if you pay for that service now does research. like deep research. This didn't call it deep research. It does reasoning across multiple links, and it looks pretty cool. And not only that, they, together with this, they added the workspace integration for, specifically for Google workspace. So it can reason through. your documents, your links, et cetera. and they're very happy about this. very interestingly, they released it together with the week where OpenAI also released an eval for deep research. And I haven't seen the comparison yet. And I also don't have access yet. This is not also an API. This is only UI feature, but it does go and reason about stuff, including your documents, which is pretty cool. If you use cloud, max, which is their 200 a month, tier.

I see the connected app stuff, but, and the Gmail search and calendar

search, I don't see the research. So I got to ask this half the features. So I guess.

Yes.

I've looked at this and research is now available early beta for max and team and enterprise plans in the U S Japan and Brazil. Not only you're not on max, I'm assuming maybe you are, but I'm assuming you're, you're a plus subscriber. You're also Canadian, which they didn't release it to because. I don't know, Tarek, I don't know. so anyway, Anthropic has always

had, like limited DevOps.

It's

always, it's always been a limited release.

Yeah, they're struggling. let's mention this briefly, Mistral Classifiers Factory. Mistral released a cool thing, which I had the link for sure. let me see, Mistral Classifiers, Classifiers. Mistral risk classifiers factory, which is basically, you can build your own classifiers, which are small models that run on their service, and you can build classifier for moderation, intent detection, sentiment analysis, data clustering. Those are not LLMs. Those are very, very small models that Mistral will host for you, but you can upload. Very limited amounts of, I don't know, data in CSV format. And we'll just they will train and build this for you, like fraud detection, for example, in financial sector. But if you're a company that absolutely needs a super quick and fast classifiers, Michelle classifiers is for you. It's very interesting step. That's not LME, into, into kind of the world of, of supporting and guardrails. it's pretty cool. I think it also supports within biases to some extent. So if you are a company that needs this service, Michelle classifiers is for you. coherent bed. We, we briefly chatted about, let me just show you what they, they mean. Bed four is a multimodal search for businesses. so if you want classifiers and then also multimodal embedding for your rag cohere. Four is sota, multimodality and bed. Four, uniquely capable, accurately and quickly searching multifaceted documents. PDFs, et cetera. previously we chatted with Zach Nusbaum, if you guys remember from, Nomic AI and that talked to us about why multimodal embeddings are important is because basically when you embed text separately and images and PDF separately, then you have to do a bunch of searches to combine them together. multimodal embeddings bring this into one, embedding space, and so you can pull relevant chunks that have, some PDF stuff. Some PDFs are images only. So that's great. And shout out to, to Cohere for embed4. With all of this out of the way, the most important news of this week was absolutely GPT mini and 4. 1 nano in the API only. So this was a developer release on Monday and then all three and all four mini everywhere. Let's start with this news, GPT 4. 1 mini. I want to ask this question to the panel here of folks, folks, why is it the GPT 4. 1 is better than GPT 4. 5? Did somebody answer this question to me?

They trained it longer, it's just, it's just

probably what happened like GPD 4. 5. Look, I tried to ask it its size and like to estimate for it to guess what its size is. But it's fair to say it's probably the largest in the world. Probably that's the whole thing. And. It just takes more to train and every single step takes way more. It's more expensive. So I assume they didn't train that long. And obviously, I don't know, 4. 1. Sounds pretty close to GPT 4, like probably it's the same model train much longer on much longer context to on on much more, focused data sets because currently we just see we just see by the behavior of the models that the models just the data sets improve. But the data sets, the evaluations, the things they're doing behind the scenes, you just see from the behavior of the models that they sit with people and the models become more and more like sharp on the things that you need. So I think that's what is going on. It's just better data, and flops. A lot, a lot of flops. A lot of flops,

yeah.

Wolfram, you have a

chance to There is

a magic, there is some sort of magic to GPD 4.

5 which I'm not sure is only the science. There is some sort of magic it gets things that no other model can. so I don't know, but, I don't know, maybe scale is all you need. I don't know. Yeah,

scale is definitely, at least part of what you need.

I will say, to, to your point, GPT 4. 5 has, what. We got to use to name a big model smell. That's, that's something that magic is the undescribable thing. Like we'd have been talking with models for a while. some of us are able to already get it and we know it's a big model and we know because, open, I essentially deprecated the API and said the GPT 4. 5 is coming. to its end on the API, which is sad, but it's cost them like so much to run because it's a huge model and they haven't distilled it yet. Wolfram, let's do a brief summary of what 4. 5, 4. 1 is, why they released it. And then, let's talk about some evaluations for 4. 1, 4. 1 nano and 4. 1

Yeah.

So 4. 1, and mini and nano, I tested all three of them. And the interesting thing is that nano, it wasn't that good, but still better than a lot of models. But the mini and 4. 1, they are so close. That is, if you are budget conscious, then you will use a mini a lot, I think, because it is so much cheaper and it's very nearby, but if you need the best quality there is, then you will go for one or maybe all three, if you want reasoning. Or, all three mini, Oh, four mini. yeah, so many models you really have to think about which one you want to use. I generally go for the best one. If I'm using the interface, it doesn't cost extra. But when using the API, you really have to think about what do you need? And then you'll need evaluations for your specific use cases. So you can know if the model is good enough for what you want and you don't overpay.

I will say the naming of OpenAI is obviously ridiculous,

especially the drop down the model, especially for folks who don't follow as closely as we do, that's why I opened this with a joke of like, why a joke, but I took it seriously and explained it very, very seriously, Why 4. 5 is like worse than 4. 1. Shouldn't models naming be like 4. 6, et cetera. Yes. Yum. You're correct. It's because of the model sizes. Like we get it, we get like what 0. 5 release means it's a 10 X jump. And then a full release is a, is a hundred X jump. That's, that's traditionally what used to be, what used to be named. but 4. 1 is not a 0. 1 jump. but we do feel this, I will say this thing in. In defense of the naming, let's say I got confused myself by talking to you guys and generally thinking about this of, all four and four. Oh, Oh, four is for always confusing the hell out of me. The, Oh, the, the models that start with, Oh, like Oh one Oh three and Oh four mini right now, they are, they're reasoning models. The four, Oh, came from four zero like that. We just call it, Oh, in English, but like four, Oh, it was four, zero, basically like GPT 3. four. and then for, sorry, no, no, no, no. I'm. Completely confusing right now as well for Omni. Yes, see, it's very confusing. What I will say though, the thing that really confused me in the API as well is that we now have 0x04 mini and then 0x40 mini. 0x40 mini still exists in the API. However, it's getting deprecated towards, 0x4. 1. so we can no longer, we no longer have to say 4. 0 or Omni, et cetera, because essentially what they just released this week is a drop in replacement for many folks for the previous 4. 0 and 4. 0 mini models. So in the API. We

can.

three came after all one because there's no, two, it's, it's, it's so confusing.

yeah.

Oh, two, Sam did refer this to as Oh two at some point, but then because of the auto arena in London and they didn't want to get into the legal fight. yeah. Somebody should buy all 17 and a trade market right now, right now before everybody else. All right, folks. So I just want to run through the actual kind of launch, of the 4. 1 because we do have to talk about those as well. they launched three. Main kind of like things. GPT 4. 1 is the flagship model that replaces GPT 4. 0, essentially in the API. They're testing it right now. At some point, they will probably deprecate 4. 0 in towards this, 4. 1 mini is 80 percent of the cost with like lower latency. Those models are fast. We tested them in windsurf. They're just flying. And 4. 1 nano, which is ultra fast, but honestly stupid. I've tried nano. Nano is not up to par with like essentially anything. However. In tasks that we, we're testing very serious tasks because we're testing the bigger models. Nano is absolutely great in translation. So if your translation use case is translation, nano is probably the model for you, and it's going to cost you one cent for a million tokens, which is nothing you can transcribe all of your things. It's probably good transcription as well. and by the way,

that's a great example for, there is no chart where you can just look

your use case and which model to use. So you have to experiment, find it out yourself. So yeah, you can't even ask JGBT which model to use. They didn't even train it. The models

themselves are notoriously bad at telling you

about the models themselves. They don't have three. Can

you search now and find it out?

I really had to convince 4.1 that it had 1 million context.

It really argued with me that it doesn't, and, and then like I, I like told it to, to the IQs, perplexity, MCP, just to research how much context it has and only then. It just gave away. All right. But it was like, yeah, they don't know. But it knew, it knew that it's 4. 1. And that was cool. it knew the name of the model, but it didn't know the cool, the feature of itself. Yeah,

System prompts are getting so long.

Maybe put that in as well. It doesn't matter anymore with the long context.

Yeah, you are senior model at OpenAI.

You have long contacts.

coming back, and the thing that you just added, the absolute banger

thing that we have to talk about, all these three models, is OpenAI's step in to the one million context window length. OpenAI traditionally gave us, a hundred K now, all the, all three models are 200 K. this is a 1 million context model with low latency. And so absolutely like the nano for the use cases of nano, for example, there, there, there is a lot to do with low latency, 1 million context. Absolutely. we, we definitely should shout out that like even 4. 1 nano gets 1 million context at 10, just 1 cent. This is oh, sorry. This is 10 cents in and 40 cents out, which is just an absolutely crazy, crazy price. By the way. You

have batch processing now for 4.

1. So you can do data sets with just 4 per million tokens. And, I, I generated some, some medical data at two different doctors look at it, and this is by far the best. Best model for, like doing medical soap notes, but it's just like the styling and the way it describes everything so nicely. it's just, yeah, I don't know. 4. 1 has the best. Style and responses right now, because it's just long enough as well. You could say the sonnet does give you more data and the is, is actually more detailed, especially if you're on conversations, but in terms of giving you a report, this is by far the nicest and it's been perfect. Okay. I did it in 36 languages. I've checked. Quite a few of them, even random ones. And it's been perfect in every language too. yeah, it's, it's, it's good. I don't think it's that good at code, but for, for other stuff, it's, by far the best styling on the, Like the, the data that they had on 4. 1, I think, I think that's, it's the best one.

So let's talk about, coding focus for this model.

They absolutely focused these models and said, Hey, these are for coding. they showed us a sweet bench, the jump from 33 percent to 55 percent from 4. 0, and nearly close to all three mini. Also, he's a reasoning model. This is not a reasoning model. This is like Also faster than the reasoning model, doubles the diff generation accuracy in either polyglot. So they used either polyglot, the kind of the benchmarks that we, the, the, we used to look at, for coding and, very, very significant boost. Nistan, you don't think it's that good. I honestly will say that I got very excited on stream. You guys remember with the front end abilities that they mentioned that they've trained more front end UI, et cetera, on, on these models. I got. Maybe perhaps overexcited, since then I've tried it in windsurf because it's free to try and windsurf. and the website, it gave me, I had an idea about the website. it gave me, I will share this website with you very soon once it's live, I think it's pretty cool. But, the, the, the UI 4. 1 gave me was, let's just move on from, from, from that point. It wasn't, it wasn't a great, it wasn't that great. I also posted a, a, Paul on my ex after our live stream, about two additions that I asked cloud 3. 7 to do on the website. And then 1 actually did a good job there. We all looked at it. It did a decent job. Folks did prefer cloud 3. 7, but I think cloud 3. 7, is a unique, very eager person that goes and does the things that you don't ask it to do. instruction following upgrade. They, they, Also had a, like a, instruction, internal hard mode instruction evals that they didn't release yet for OpenAI. Decides 30 point jump in instruction following versus 4. 0, and multi tone consistency as well. Rivals in Tropic Cloud, it does listen to instructions better. And here, we should talk about the sandwich. Which I found absolutely ridiculous. And I actually tested this. I actually want to show you. This is not the This Week's Buzz sponsor segment at all. This is literally just me. I, I, Wolfram, can you describe the sandwich while I pull up my emails? Because those are crazy. The sandwich? The prawn sandwich. Where you have

something at the top and then you have the context.

Ah, yeah. OpenAI gave, was it OpenAI or Entropiq? Now I'm getting confused again. It's different. Yeah. OpenAI is that you should have something at the top, then the context, then the query at the bottom. while at Entropiq they were saying have the context first and then the instructions at the bottom. by, with OpenAI you sandwich whatever. input you have for your query, so you can give it, you could even repeat the instruction at the top and at the bottom to improve it. And, as Alex has, evaluated and found out, that improves scores tremendously, right? It's a very noticeable difference. That was the crazy part for me.

so 4.

1, so again, like Wolfram said, the prompt sandwich technique is you have your instructions, then you have your long context, whatever, and then you have your instructions again, the same exact ones, and that improved performance. OpenAI released a prompting guide and somebody from OpenAI also shared their like, techniques for 4. 1. I did not expect this to work at all, but I did not expect this to work. this much. So I'll show you what I mean by this much. So you can see for those who are looking for those who are not looking, this is my standard eval for the 20 hard to reason about questions. the reasoning models absolutely tear down this list benchmark, but still they don't get all 20 of them. this is the question for like the ice, the ice, ice cube question that we ask. There's a bunch of those here as well. So there's 20 questions around them all three times. usually non reasoning models are very bad at this. This benchmark, and nano is absolutely better. This benchmark. on the, on the right here, the three models here is a 4. 1, 4. 1 mini and 4. 1 nano nano is getting 1. 8 percent of the questions for, mini is getting 31 percent and then, 4. 1 is getting. 48%. I know it's confusing just by listening to this. I, I, I will share this in the newsletter as well. What you can see on the left, those are the, the exact same models just with prompt sandwiching. All I did in this new eval, and this is clarified here by this little sandwich emoji that I have, is I just added the, prompted the end, and you can see that for. 4. 1, this did nothing. It got exactly the same 48 percent of questions, exactly the same, but for, 4. 1 mini, the mini model, the one that costs half and runs like much faster, it jumped from 38 percent to more to 49 percent of the questions. This is crazy. The jump is crazy. There's not a reasoning model. They shouldn't be this good at these questions at all. And even for the nano, it improved a little bit, but a few questions as the biggest jump by far. Was for for, for the, the, the mini, one mini matches almost beats the regular model with this prompt technique. This is crazy folks. I don't know if it generalizes to other attempts, but this is like absolutely crazy. and then you can see the total tokens that they output is also a little bit, a little bit bigger. So they actually like output more tokens once you ask them to do the same thing, which is crazy. I just wanted to show this, what you're seeing right now is WNB weave, the product that we have for evaluations. If you don't evaluate your models. If you don't evaluate your models, how do you know how they will perform production? This is how I do it. all right, folks, what else about this model that we want to talk about? So definitely worth checking out the 1 million contacts window is crazy. I want to chat about the needle in the haystack thing that they posted, and we definitely should move on to all three very soon. Needle in the haystack is this Very incredible, but also very annoying test that Greg Kamradt came up with, Greg Kamradt, now the president of the RKGI, foundation, that just gives the model in a very long context, a bunch of random, or one random piece of information, and then ask the model to retrieve it from the context, and then test whether or not in long contexts like this works. Most models are now good at this. However, we know that reasoning around long context is not necessarily, as correlated as need only haystack, green fields. However, they did show us that all three models here in the need only haystack are absolutely like 100 percent across the board. They even showed an example. Listen, do you remember the example? They had a bunch of logs and then they shoved like a line that's not a log and the model was able to pull this.

Dude, I just opened it, on my non paid account.

I opened ChatGPT. And when you hit switch models, it says, GPT 4. 0, great for most tasks. O4 mini, fastest and advanced reasoning. GPT 4. 0 mini. Faster for everyday tasks and try again, GPT 4. 0. How is anybody going to be able to tell?

Here's here's my guide.

Okay. Here's my guy. Here's the hire me to

name them.

Yes.

Here's the third official guy. First of all, we're all are available as consultants. You want to reach out to all of us. We'll consult you for, if you're listening to Thursday, I use the Thursday promo for 20 percent off for our consultants, that's first of all. Second of all, I'm, only partly kidding because really People need support with this. I will just say for coding and building with APIs, 4. 1 series of models is now probably the, the one that you should be evaluating, trying on 1 million context window at those speeds is crazy. If you're just chatting, oh, three, if you're just chatting and you're okay with waiting or free is by far the best one, if you're building with intelligence, then, then all four mini, those APIs are insane, but we didn't get to offer many. So for all intents and purposes, if you're just in the UI, there's no reason for you to, to, to use the previous models. Just use the top one and wait, the faster for everyday tasks. That's some bullshit. That's just You don't necessarily need to. so if you're building something as an engineer, evaluate. And if you're building something as an engineer, if you don't evaluate on your tasks, then what are you doing? Talk to us at Ways to Buy. So we're going to help you because, as, as Wolfram's title is AI evaluator, and at our mind, if you're not evaluating your models on production, you may be hurting performance. You may even don't know that like your tasks are not getting done. So back to Nero and the Haystack, OpenAI also showed that they've, evaluated the long context performance on the million tokens on the new MRCR benchmark. And they also open sourced the MRCR benchmark, which I believe was Entropiq's first or somebody else's first, and then they kind of like improved on it. It's on Hug and Face right now. It's open source. on the same day, and, they show great retrieval across the full one million window for all the three models, which is, which is great, even nano. So that's like very good from opening it also release some benchmarks because, evaluation on long context is difficult. multi modal games, they claimed state of the art on video MME, which is a long context, understanding of video, by pasting in frames by frames, and long context QA on video. They, they claimed state of the art on this with 72 percent for, 4. 1 mini. Mini is a great model. Do not sleep on mini. I'm, I'm, I think that mini is better than 4. 1 in many, many cases. They're also fine tunable from day one, which is great for folks because you can upload some stuff and then get your model specifically for your use cases. And then I think that that's basically it that I want to mention. Windsurf has it for free for the next week. I think I know why now, but also I think Cursor followed suit and Cursor also has it for free for the next week or something. it's very interesting as a, let's get users in the door, the new model releases, they're like, here, use it for free for a week. They also want to evaluate this as well, right? They want folks to tell them feedback.

By the way, if you were a Windsor user in the beginning and you

stopped using it, like I did, you, you will still, at least yesterday, you could still get the, early adopter pricing, which was 10 instead of 15 per month, if you lock it in now.

Absolutely.

I will just shout out this prompting guide from Noah, Noah McCallum, that he's been working with GPD 4. 1, for agentic workflows and chain of thought. you can prompt these models for chain of thought. Don't forget, the new O series models and the Thinking models, they have chain of thought built in, but the previous trick still works for these models. You're going to ask them to think. And when they think they get better. And when you, and when you ask it in the problem centers, they get better even still. so definitely, you can ask the model to spend more time understanding user intent that helps, start with simple instructions. And, at the end of your prompt, as the model is already quite good at thinking through problems. Long context, the suggestion. So basically we'll add this prompt guide as well. they actually. This is his thread, but they actually added prompt guides in the docs as well. All right, folks, moving on to the main course of this week, this month, maybe this quarter. OpenAI yesterday, I really want to hit this button again. I'll just do this. OpenAI yesterday released finally to all of us the flagship prisoner model, probably the best model right now in the world across pretty much everything that LLMs do. 03 and 04 mini. Let's go. And we were here right here watching the live stream. And we were very, very excited to try these models. Folks, what do we think? Jan, what do we think about O3? It's been less than

24 hours.

It is really, really good. No, seriously, I don't know what they trended on. I really don't know. I have a guess, but I really don't know what they trended on. And I don't know where to even start. I have so many examples. Every tiny thing you do, it just knows stuff like I don't know, it just knows so many tiny things without even go before even talking about going online and searching them, which is also crazy like you can do. You can direct it to go to URLs, specific sites, and, and it's just, it's just a crazy release. Seriously, crazy release and I can't wait for O3, really, it's It's a crazy model.

It's a crazy model across pretty much everything.

I agree. I've used it exclusively. here's a few of the things in the announcement. Like it's going to be hard to cover everything, but here's a few of the things in the announcement. This model, I think is unique. by reasoning across tool use. It uses tools while reasoning to come out with answers. So one examples that we've tried yesterday on the show is, the, the, the, the cowboy example of generate an image of a cowboy that on his head is the, the five last digits of the hexadecimal code of the, MMMU score of Gemini 2. 5. I think this was the full prompt. Now, what I said is a mouthful, but basically there's like a four point tasks in there, including tool use. It needs to go and figure out the MMMU score for Gemini 2. 5. And then it needs to use the coding tool to convert this. It needed to figure

out some model.

We didn't even tell it which model it was.

Yeah, Oh, yeah, The latest model for Gemini series right there.

So it needed to figure out this. So it actually went and searched on this instance, Oh, no, I actually need to go and search this. Then it got the actual score. Then it used the tooling code to convert the score into hexa. Then it used the image generation feature tool, et cetera, to generate this image. It was crazy because it did seven steps and four searches. And then it like thought about the searches. And then it's Hmm, let me reason about this thing. It was. crazy. And this is not even the thing that Yam is talking about that he's excited about because the knowledge that this model already has is great is what Yam is referred to. This is in addition to the knowledge, in addition to the training, in addition to everything else it does, it knows it can also do stuff like go and look up things live when it needs to. To reason about them, go and look again. I will just say our excitement of deep research, deep research has been out for a while, and we've been using this and then, you have to pay 200 bucks a month for it. And then it came to, to the users. Deep research was powered by all three. Deep research is incredible, but I don't want deep research for everything that I do. I want the model to react faster. And this, this is what all three kind of unlocked for me. All three unlocked for me, the ability to do. Go do mini deep research, not shallow deep research. Let's say on a very specific topic, reason about this, see if you have enough of the understanding and then bring it back to me. And that's been the biggest unlock for me. It reasons across tools. It uses tools during its reasoning process to then give you an answer. That's much, much, much, much better than anything out there, which was absolutely incredible.

It's my new default model and we are just starting to get to

know it, but, I don't know if it's controversial or not, if I'm saying this, I want to hear your opinions as well. But I think the jump from what we had before to three, all three now is bigger than like 3. 5, GPT 3. 5 to GPT 4. I think the capabilities it unlocked. I could imagine that this is even a bigger step forward and we will notice over the coming weeks.

I, I'm with it.

I don't think it's controversial. Jan, go ahead, react, but I'm, I'm with this.

sure.

I think, I think 3. 524 was insane. And, and this is a huge jump at the moment. No, no, no, no question about it. think g PT for 3.5, think, think about it, to the end like, come on.

It was smarter, but it didn't do much more.

I think the, the capabilities on lock is massive here.

and I the fact that

on this one.

No, I'm just saying I'm with the, of this one, the four jump was insane. That was on another, You know what,

here's how I'll like, I'll tie break this like debate

that we're having, which is great, which I love, but we don't often get debates here on Thursday. 3. 5 to four didn't massively have a capability jump. it was an intelligence jump, but not capability jump, for didn't come up with new tools that we got to play with immediately for did come up with image and understanding that we waited for four months until we got five months. I remember like Greg showed it and we didn't get it for a long time. this. Plugged in into the ecosystem of tools and the things that we're used to and does them on the next level So reasoning across tools the tools we already have is happening on the next level I think for normies for people who are just using stuff This will be a significantly more of an unlock while we did get super excited about gpt4 Back then because it was like significantly better than 3. 5 in everything that we had access to so I think overall It will feel like it should have felt for folks back then, but it took a while for folks to get into. Let's talk about some of the actual stuff, in the all three announcement, state of the art across everything, across everything. AIME called forces, we bench with tools. So we bench gets 80, sorry, 64%. Like we looked, sorry, 69 percent on sweet bench. The previous state of the art models with tools, with multiple spans, one of them for bits and biases. Another one is from, I'm like, they, they beat us, but like one point, this, this is nine points above or sorry, five points above all of all of the, Oh, one base tools. It's crazy. Good at coding. It's crazy. Good. The coding is crazy. Good at reasoning about coding sweet Lancer, which is a eval from way from open AI that uses, Upwork based tasks and measures them based on how much money is a model can make the money. O3 high, which is O3 with a high reasoning effort, which you can provide reasoning efforts to both these models, O4 mini and O3, scores 65 earned on top of O1 got 28, 000, O3 got 65, 000. So double. More than double on this performance on real world upward tasks that users do and developers do either polyglot is like just basically It's I think it's not worth repeating even it's state of the art across every evaluation that they tried It's state of the art every evaluation that we saw one interesting thing. Yum. I wanted to bring this up I saw this on your ex first, so I wanted to shout this out. it's Ridiculous at long context performance. Did you guys see this? I don't know if you and, and Wolf, you had the chance to check, check this out. Let me, let me try to pull this up here. it's ridiculous at long context. There's this, fiction life bench for long context, deep comprehension. So we know that we just talked about like the 4.1 models that are great at the internal benchmarks. M-C-R-C-M-M-C-R-C benchmark, this fiction life band shows. just performance across like long context, all the way up to 120 K. And if I zoom in here, and if I actually show this to you first, and then zoom in here, you can see that the previous biggest winner of this very difficult table to parse at 120k, the previous winner was Gemini 2. 5 Pro experimental, the model that Google just released. It started with 100 percent comprehension, dropped to 97 and 4k, and then goes and at 120k, it's still at 90 percent ish. Among other models, you can see the degrade in performance. Even in 4. 0 latest, it drops down to 65. 4. 0, 4. 1 mini drops to 62, despite the things. O3 is a hundred percent across all this, all this long context across 4k, 8k, 16, and 16 gets 88. I don't know why, but then at 120k context, it's still at a hundred percent. This model absolutely slaps at long context, which is also crazy. And this is like additional one of the things for O3. Yeah, I have a question.

Can you actually use that on the app?

can I just dump in 900, 000 tokens like I can in Gemini and will it respond?

That's something that you should test right now and let us know.

Let's test.

Let's find out.

What I love about ThursDI is not only that we have the answers

for you, that we also have the questions and then we are trying to do some stuff. but yeah, we should absolutely test this. The thing that only has

200k context maximum.

Yes. All three only

has a 200k context.

It doesn't have a hundred, 1 million. the thing that I got excited about the most. is the thing that I got excited about the most is and the thing they showed it is this model because it has access to tools and reasoning. One of the cool things it can do is zoom in on images to read more. Did you guys see this? You guys catch this? That was crazy. That was crazy. You can upload an image, a very long one and say, Hey, zoom in and read this. And this feels to me like sci fi, like nothing else feels like sci fi because the model can like zoom in and read and understand the stuff. Because we are used to visual language models, VLMs, whatever, just receiving a, let's say, multi resolution image. It used to be that you have to provide it the specific ratio. Now they have patching, they all break down, etc. They apply the vision headers, but all consume the image as an image. And if you want the image, one part of it, you crop it before you upload this model, can like zoom in and crop for its own and extract like different pieces, which is absolutely crazy. I

told him to go and get.

GPU prices over time per, per device per type and like just gather the data from somewhere on the internet and just make a plot of each GPU over time. And it just did that. it's just crazy, like the thing that like when they have like in the UI, this tiny Python thing that can just run stuff, do things on the fly. It's just crazy. it's just crazy to see just how much a tiny tool can add to whatever model that is using it. One thing to note. I am running a pretty sophisticated agents locally and the jump between GPT 4. 0 and 4. 1 and not even speaking about O3, just in terms of the willingness, the willingness to use tools is crazy, like the amount of agentic capabilities between the models, the jump is insane. And it's not just. every model could, I could understand the output of a tool. some models can also pick the right tool to use, but in a loop, like over and over again, you need to pick a tool, look at the output and infer what you're going to do next. Like in a loop, like errors accumulate really fast, really, really, really fast. If the model is not trained to do that, like before, 4. 1, I think the only model that could sustain, Repeated agent over time was Claude 3. 7, but now all the new models for opening. I absolutely can do it. And it was and it's and it's a huge, huge jump in performance. And so what I think is going to happen is what you see now in the U. I. S. Just the start. Like they're going to be MCPs in there. They're going to be like crazy things embedded in the UI. Today, power users, us do it locally with, with our own setups, MCPs and so on, like everyone, everyone has their own. I think the way it goes now is that people have local setups. And people just go to extreme length. on YouTube people building extreme things locally, like thousands of MCPs gathering them on. It doesn't, but it's it's a small subset of people who are willing to do this. We are going to see things like this. Automatically in the UI services that automatically just, you know, based on the context at the moment, automatically, like inject MCP throughout the conversation. So we won't need to think there's so many things that we're going to see. They're going to be absolutely crazy and what we see now is just the start. So I think people don't realize that that's just the beginning. Like you have a model that can serve online and search the Internet. you know that all of them are going to be able to do this, in, really short. I

I will say that for all three, the one thing that we

should mention that the tool use that we currently see in the UI is not available in the API yet. They're still working on this and hopefully that will also come with MCPs, but the tool running API is not available just yet. but it's coming very, very soon for all three for a 4. 1 models. Absolutely. they can use your tools that you provide them. The, the, the highlight of this model to me is that like reasoning across tool use and then coming back with a response. Additionally, it's also. State of the art and multimodal. And one of the reasons why state of the art and multimodal is because it can manipulate images and do things and rotate them and zoom in and do a bunch of bunch of stuff. So math Vista and MMU and like all of these things, even without tool use it beats, I will say this when it does use the Python tool, the AIME stuff, like the, the competition math is solved at a hundred percent for these models. It's just it crushes like all of these things. If it uses Python, like the, it's not just reasoning. It just like uses, uses them across the board.

I just want to say I just want to say about resolution images.

one thing I noticed is that I run, I run local agents and, one of the things I do is just give them screenshots of my computer because, I want them to see what I see. And there is the code in there and so on. What I saw is, until recently, only Claude could see and actually read what is on my monitor because of the resolution. it's not just being the state of the art, and I'm just throwing it out there in case, I don't know, I don't know if 4. 1 is already, capable of doing this, but I'm just saying that people to the people behind the scene make sure that the resolution is enough because there's some people like myself are going to throw in a pretty large resolution and are willing to pay for the tokens. If the model can absolutely read the code and, be, be like, a side, sidewinder throughout the day and, and help me out just by looking at the screen. So just make sure that the resolution is also, is also something that people need to take into consideration. And, but yeah, everything I just said might be nonsense because it might just be the state of the art with, high resolution. I just didn't test it yet, but it is a point. Like it is something that people need to take into consideration.

one thing that Wolfram, you mentioned this at the live stream as well.

The, the cost efficiency for reasoning is also improved. It's not only that these models like replace 01, which is, 01 is deprecated folks, RIP 01 models, Greg Buckman said that there will be deprecated. but also, they're cost efficient at reasoning as well on AIME. They actually showed us like estimated inference costs and these models perform better while also being cheaper. So we're getting advancements in intelligence, but also getting advances for cheaper as well. So all three is cheaper than, one, was, we also told, got told that all three pro is coming. moving on a little bit. The continuation of the test time compute scaling is just beautiful to see. The more compute is spent on training the models, the more, the better it gets at AIME which we saturated already. These AIMEs are getting to 90 percent solving rate without tools, without tools, without the math, without the Python, just by reasoning through them. we're getting to saturated AIME, we need harder benchmarks. Humanities last exam, for example, was foolishly named as humanities last exam. And I, foolishly because, scale, scale AI leaderboard showed, that O3 just let me show this as well. The O three, just like absolutely crushed it on humanities last exam, with 20% when Humanity last exam just launched. They're like, I would be surprised if you models passed 5% and O three high is at 20% on humanities last exam. I think by end of this year, last exam will also be like saturated. what else? What else do we, what else can we say about O three that we haven't said already? Niton, I haven't heard from you about O three at all.

I, I haven't tested it that much other than from

single shot data gen stuff. So that's why I don't. And in that case, I did find sometimes it made stuff up. And, so in generally if it's, if it's a one off, I, I preferred the answer from 4. 1 like I. But if you have something that's a lot more complex and that's multi step and it has a lot more, I don't know, like the technical stuff or calculations to do, it's gonna be, it's definitely gonna be better. Yeah, I don't know, I prefer the answer style of 4. 1 even over O3 and I did quite a bit of A B testing. on that myself, it's not like they were far off, but, yeah, let's leave it at that.

Yeah, that makes sense.

I think I've covered most of the folks. Let me run through the notes and see if there's anything detail wise, and then we can move on to open source. because like I said, if you want our full two hours, we did this yesterday. and then let's talk about Codex in the context of this as well, even though it's in tools. OpenAI also released. So codex is a CLI tool from open AI that, basically in command line, you can ask this model to go and do stuff for you and has access to the repository. It's also has, sandboxing access to the repository, to the folders. You can actually run this with auto accept. And not be scared that it, it doesn't, it won't auto accept anything that like has to do with going to the internet. It won't auto accept anything in other directories. So I actually use it like sandboxing very well, to, to, to let you be very okay. Like I know some of us were listening, YOLO moding on, on like windsurf whatever. And just have the model do whatever you want. don't do this with maybe DeepSeek. Maybe do this with the, the open AI models, before you go to deep seek and YOLO mode, but OpenAI released a way for us to trust what happens with YOLO mode by just sandboxing this to the folder, which is pretty great. I think he uses some macOS APIs for this, on macOS as well. Codex, is also open source, which is every time open source from OpenAI, they get applause. We're waiting for the open source models. But meanwhile, we got a bunch of actual open source from OpenAI, folks, they gave us like the eval, evals, like the MCRCA, and they gave us another evals, they also open sourced codecs fully, and, Oh, okay. We're having, we're getting a question from, for somebody like, can we explain what a CLI, how the CLI works? I think maybe we'll just show an example. I think this is, this is the best way to do

it.

If you really, really want, there, there is a lot to talk about this. There is a

lot.

I'll set up my thing and we'll actually show

just goes like this.

for the people that don't know, a month ago, a job, we released Cloud Code and it's just insane. Seriously, Cloud Code is insane. But we're going to talk about Cloud Code in a moment. Okay. Basically, you ask things on command line, like you just go Cloud and just ask whatever you want, as if you're talking to, to it through the UI. Okay. And it has, it has permissions to run stuff and do things on your computer. It basically has your computer, like it can run tools, it can edit files, it can do what it wants, and it just runs in a loop and try to do what you told it to do. The thing about cloud code is that it is really good. So Absolutely great, like absolutely great release by OpenAI. so thank you very much, seriously. Of course,

also Cloud Code doesn't use O3 and this, has built in native use of

O3, not without the tools used yet, but it does think about different things. So for 39 seconds, I thought about this repo that I don't remember that I installed, user asked about You can see the thinking here as well. Short this repository fully featured AI agent called Winston. It lets you chat with a large language model assistant. It understood all of this from the repo and the code and the readme, and it showed you like how it works. basically an AI coding agent to together with you in, in the, in the CLI, you can install via NPM. And then, it's open source so communic can contribute. I already saw that folks are starting to add MCP support. With, by the way, with Codex itself, Codex adds Codex support to MCP, shout out to Jason who's been on the show, and supposedly this will land, as well. super, super cool. let's, Nissan, one last thing that you wanted to add about this and let's move on to open source.

Yeah, just please people when you run these kind of command

line tools, please just please stop making you account on your Mac. Don't just do it. don't just put it on your own thing because it look, it can access your signal chats if it's on your own Mac. And you can actually see your attachments. And I was going to vibe code something to show that. So making you account. Make a non admin user use that codec scan. Codecs cannot listen. It can go in the files. I'm gonna, that's a whole other conversation, but I'm going to show it. So there's, we can save that for some other time. According to

what OpenAI promised you, at least the auto, auto

accept mode like happens. It's pretty

good.

But, but look, you're leaving that, that vector open. So please just make a non admin account. And then that non admin account does not have access to your main account on, on Linux or on Mac. I don't know. I would listen

about this.

We as a responsible, news outlet need to warn you about the fact that these models running on your thing can be, especially if you use MCPs and MCPs can be like, injected with some prompts. they definitely should, you should be aware of what, what can happen. I will also probably say that most of the YOLO people who listen to us will not do this. they're not going to do it. Yeah. we also need, we absolutely need to tell you, folks, I think that I think we're out of time for the big companies and APIs. I think we covered everything, very, very well. Folks who are listening to us, if you have any questions about all three that we haven't covered yet, or you have use cases or prompt techniques or anything else, definitely, definitely let us know in comments. We would love to hear about this as well. I will shout out. I see Sam listening to us, Sam Peach, who's now in the Twitter, the space has the EQ bench. Eval and then show that all three is also one of the best at creative writing as well. That's also crazy folks. You should also know about EQ bench. We'll talk about and creative writing. All three is also greater creative writing. I don't see a things that all three is not better than other models, maybe speed and all four mini, which we barely covered, but may in fact be. The hidden gem of this release, O4mini, is also crazy in a lot of things and like much faster and cheaper as well. and the, sorry,

who is it that wrote the EQ bench?

Sam

Beach.

I will add his, Yeah, I just, yeah, I want to look at that. There

are some opinions online that the model was rushed and because there

are some crazy hallucinations that people saw that didn't happen on other models. there are some vibes on some of the answers are questionable, you can say, but, it's a huge release. I just want to, a little bit more balance just to give to give a stage to some other opinions. So if you care about, just go, go, go check it out because. Probably, there is a price to pay for, cramming these hard. So many, so many tokens in training, people are investigating that as well. One last thing about the command line tools. I just want to say to everyone, it's not just about coding. for me, I can. Okay. I went a little bit far with this, but for me, it really changed the entire way I use with the models in the last month. I use nearly all models through the command line tools. Just because think about it, they can now access perplexity and then go online and then check my computer. Like I can ask much more general things that no other agent or, or, or client can do, especially not something that is running on your computer, for, for mundane things like, did I like check my calendar? Like what meetings I have today? Yeah, like How can I organize my desktop? I can't have too many files like them. Mundane things and it's like really go and look into that even if you're not a developer. Like those things are crazy powerful, seriously.

All right, folks, we're moving on to open source because

I think we covered everything. This has been a big week for big companies and APIs in in the AI world. I think it's time for us to chat a little bit about open source. And then we're going to have a chat with Todd. Very, very soon. And I know ton you need to back out earlier this, this one. So cheers and we'll see next one. No,

I, I actually had a call at this stuff, but yeah.

Oh, you already had one?

Yeah. and I

found, I found, oh, I, I realized that I know Sam from, from Twitter.

Okay. It's eq bench.com and it has a lot of the, open source models too. It's actually a pretty good, pretty good site. Yeah. yeah. We need more models up in there though. I know the main, the main ones are, but yeah.

All right, folks, we're starting with open source.

Open source AI, let's get it started. All righty, we're gonna get started with open source. there's not a ton of updates here, so we're not gonna, spend as much time as we usually spend on open source. Although we do love open source here at ThirstyEye. let me just share my other screen. let's chat about the stuff that we have in open source. first of all, we chatted a bit about this, but, OpenAI did open source something. the open source is multiple things. Codex is open source and, we heard from, from, Sean that it already got, a bunch of PRs in less than 24 hours, which open source rules. 60 PRs merged in less than 24 hours, so this is absolutely, absolutely crazy. but also, we, we saw from OpenAI an open source of MRCR, Long Context Benchmarks. And we're gonna actually take a look at how this looks and it's pretty, pretty cool. this dataset tests for long context benchmarks and then, desired index, total messages, blah, blah, blah, blah. It has 2. 4 thousand rows that they've selected from the MRCR. I think, oh yeah, yeah. I mistaken before I thought there was like, on topic. No, the MRCR first introduced by Gemini. And then OpenAI expanded the task difficulty and provides open source data for producing results. So this task that used to be just a, a. a paper now has arrived as, as a open data set from OpenAI. and it looks like this user assistant, user assistant, user assistant. And then, they have conversations for two needles in Haystacks, whatever. And then they ask very difficult questions like, which was the second one? Which was the second story that the user talked about? Like rocks, for example, or tapirs. They love the peers. For some reason, the peers, the animals, they love them. They mentioned them on the live stream as well. So basically the benchmark works by, asking the model like difficult questions. They're not only need only haystack questions. It's contextual questions within multiple conversations as well. so that's very, very, very cool. so if you want to build a leaderboard of, MRCR, you're more than welcome to, because this eval is now built in. If you want to use bits and bytes as we for it, you can absolutely do so as well. Also in open source, we have Microsoft releasing BitNet v1. 5, or, assumably quietly a little bit. I, yeah, can I tap on you to, to explain bitnet just a little bit? And like I said, on this 10, one of you wants to take this one.

I, I, I have to go in three minutes, so I'll

just, I'll just make it quick. The thing about bitnet that gets people very excited is that on a chip, you can just, when you have like the 1. 5 bit, when you have ternary operators, so either. Not zero and one, but three, three ones. You can, you can just use addition instead of multiplication to do the matrix multiply. So in theory, you should have some, some great improvements when chips are made. I just want to say about this release first, I really appreciated that they finally put the effort to train it all like that. But I do want to address some misconceptions that people have about what happens on a bitnet. If you look at the gguf file that Microsoft released on their hugging face, you can see that it's a 2. billion parameter model. But the gguf file ends up being 1. 8 gigs. And this has been the major issue with bitnets is because The last weight, the big one, the output weight, is kept in F16, otherwise the model deteriorates. And the token embedding weight is kept in F16, so while the rest of the model is in ternary, the last weight is in float 16, and the last weight ends up being bigger than the whole model itself. So when you actually calculate how many bits per weight it has, It's more like seven and that doesn't make it as impressive for the benchmarks in the end. Now, if they move to a 4 bit output weight or maybe a 2 or 3 bit output weight and then have the rest of the model in ternary, then that would be closer to a true bitnet. But there is no real bitnet yet that's all ternary that just does not exist. This is a big misconception that people have about, about bitnets in general. Thank

you, Nistan.

and

so much for, for dropping some knowledge about BitNet, folks.

very quick to open source updates here. Prime Intellect trained a 32 billion parameter model that is with RL, with reinforcement learning, globally distributed, completely globally distributed, which is super, super cool. and we've talked about Prime Intellect. We also talked about News Research's attempt in a similar thing as well. Intellect 2 is launching the first global distributed, 30 millimeter model with RL and reasoning as well. it's called Intellect 2. I don't, I don't know if we can actually use this, but they broke down the whole like system and I think it's very cool. That we are now at the edge of distributed training that people from home can join as well. So just wanted to shout out Palm Intellect super quick. And I think the last one here is ZAI, Z A I, Z A I. previously chat GLM, released GLM 4, and, really good. I, I haven't seen like very state of the art, like updates from this. However, they claim performance on like the 32 billion parameter performance is claimed to be very close to Quent and they, they have special RLs for math and code and long form analysis. they did release this in MIT license, which is shout out to ZAI for, for this release, and they released like a bunch of checkpoints from 9 billion all the way to 32 billion parameter. GL is great for code generation, functional calling, et cetera. and so shout out to the rebrand and, the parity of a much larger 72 billion parameter model on MMLU and human email. All right, folks, we have a bunch of other things to cover. Specifically at the end of the show, we'll cover the video model super quick, but for now we're moving on this week's buzz and this week's buzz. We'll, we'll have a interview with Todd Segal as well. So let me just say hi to Todd super quick. Hey, Todd, welcome to the show. The first time here. And how

are you?

Thanks for having me.

Thank you for coming up.

and, we will start with the segment that I call this week's buzz that we talk about everything in with embassies and this news comes, and you come into the show as well from that perspective, because with, with embassies is a launch partner for the A2A protocol, which I'm very, very excited to chat about with you as well. So we're going to switch over to this buzz, cover something super quick, and then we're going to have a conversation with Todd Segal from, Google that works on the A2A protocol. Let's go.

Hello, folks. Welcome to this week's buzzer category. In Thursday, I talk about everything that happens and new and super cool with the show sponsor Weights Biases, right? And as you all know, I'm an AI Avengers there. This means that I talk about models. I host the show, but also talk about the cool features that we have. And, Weights Biases, if you're a frequent listener of the show, you know that we have a product called Weave, which is LLM observability and evaluation, which is very important. When weeks like this happen, when weeks like today happens, when every model releases and you're like, Oh no, what model can I use? we are now supporting the, the newer models in the playground of Western Biosphere. Playground is a place where you can compare based on the production traces that happened on your, on your traces, on production, in your app, in your agent, you can go in there and say, Hey, I want the same call, the same agent call, et cetera. I want to try this with a different model, including the whole state that the agent came to, and you just switch the model and compare between them and say, Hey, what would have happened if I had this agent run on O3 or Gemini, et cetera. And, with embassies, Playground is absolutely not supporting all of those. I will show you this later on because I don't want to delay. The other thing that happens with weights and biases is obviously also that we have launch time support of the A2A protocol. A2A is a protocol released from Google and a bunch of supporting partners. We're just one of I think 50 or so supporting partners, launch partners for the A2A protocol, from Google that is all about agent to agent communication. And we mentioned this last week when the protocol kind of came out, and we launched support and I wanted to go deep into this because I think that. It's very important and I really got to reading across the protocol, started to think about like the internet of agents that's coming up to us and how this will affect this. Obviously we're all into MCP as well. And so in order to understand the differences, in order to chat about deeply about the A2A protocol, I have here Todd Segal from Google. Todd, welcome to the show. This is your first time here. so for folks who don't know who you are, would you do a brief introduction please? And then we can chat about the stuff that you're working on.

Yeah, sure.

Thanks for having me. I've been listening to your show a bit. I was unaware it was on video until just now. this is a nice surprise for me. My name is Todd Siegel. I work at Google. I work on, AI protocols. So I'm working on MCP and A to A and connecting enterprise information, and personal information, et cetera, with agents. So the systems around agents.

So you, you mentioned the The aforementioned elephant

in the room, let's say, because many people reacted, with kind of bewilderment about MCP specifically. first of all, we should mention, Google has, came out with, we're supporting MCP protocol, publicly, I think, both Sundar and, and, Where am I blinking? Demis Hassabis, both said like MCP, we support MCP. there's a bunch of examples, cookbooks, et cetera. there's going to be more support, I'm assuming. And you mentioned that you're working on both of them. And then when A2A came out and I saw an immediate reaction of folks who did not go to the spec website, did not read the section that says, here's how A2A is complemented with MCP, and just reacted like, why do we need another protocol? MCP is all over our timeline. So maybe let's start there. Maybe that's a good point for us to start. What is the A2A protocol and how is it different from the MCV protocol that, people are getting very, very excited about?

Yeah.

So in a short summary, we see MCP and A to A is complimentary. We see MCP is for connecting agents with tools and A to A is for connecting agents with agents. tools are, are structured and they're for very specific purposes. and you have very precise control over them. They do a very specific thing and they do it very well and they often do it deterministically. Thank you. agents, are much more unpredictable and they can do things you didn't know about. And also your interaction with an agent can change over time.

Interaction with the agent can change over time, which is a very

interesting attack vector that I saw for the past week that, people are very worried that the, MCP tool announcement when you just installed the MCP tool is one, and then there's like this stuffing attack that like people can change the definition of the tools, but you already installed the MCP. so the follow up question here, would be. Do you guys not foresee, or maybe you foresee it differently, while thinking about the A2A, that agentic tools will also expose themselves via MCP? Because technically I could do this, not only this, and I don't want to push on this point like too hard MCP has A2A, I want to talk about A2A, I just like want to want to understand like deeply. MCP also has this summary survey protocol that you can ask the, the The LLM. Chatbot etc to complete stuff for you. So you MCP does technically expose a little bit of agentic capabilities Although I haven't seen those implemented yet sampling. I think it's called Do you not foresee that MCP will like also host agentic tools in the future? Specifically, and is is this your attempt to say and draw the line in the sentence and say, Hey folks, for tools, MCP should be like stateful tools, et cetera, maybe API calls, but agents are like a different category of a beast and it should be treated differently as well.

Yeah, I think there's a few different concerns and, different

actors are, are interacting with them. on the, on the agent to agent side, I think we have it structured more like services. agent a talks to agent B and says. I need help booking a trip and that can go a lot of different directions. there can be questions coming back and forth from the agent. there can be asked for more information. There can be asked to, Oh, show me a picture of somewhere you'd like to go. Give me your time constraints. Give me your budget. and then all sorts of unstructured information passed back and forth. That's extremely hard to model specifically in a, in a tool. so we, we really do see it as a application. And a remote application, delegation protocol compared to MCP. You want the model to have exact knowledge over the various specific capabilities and have the model be able to reason and call, tools with very specific parameters. So we see it really as a model level protocol on the MCP side. We think it's named very perfectly. And then an application level protocol on the A to A side. They should level protocol. I love that.

Okay.

One of the things that got me excited. I know for a fact that, when I provide my agents with tools, my chatbot with tools via MCP, I do the discovery me as a developer. I kind of like, I know what it needs to, or I think I know what it needs to. And then we recently saw a few attempts of the MCP for MCPs, for example, but that kind of scares me because I don't think that MCP was built for this. I even remember, a chat on latent space. With David's or, and like some of the folks who work on MCP, they also think of tools, like very specific thing. They don't recommend the tool that does all of it. They recommend to scope it down. So your, your context will have limited, additions. One of the things that I got very, very excited about A2A is the registry concept. Could you talk about this a little bit?

Yeah.

Yeah. one of the reasons we, we built it is we see people building a lot of really specialized agents and going really deep and building these specialized agents. So domain specific agents that have all sorts of custom expertise built into them that use custom reg sources and custom tools and private APIs and private data that isn't exposed. Publicly outside of the domain specialist. So with that, having some sort of centralized repository that you can register all the agents, and, or that, a web crawler could, could access all the agents. And then having some sort of service where you can say, Hey, who is the best expert to solve this sort of task? and to interact with me, it's a much more similar to, trying to to hire a human to do a specific job, trying to hire a specific contractor, trying to hire a specialist in a specific category versus getting access to all of the specialist tools like I'm not good at, building things with my hands and installing them. I'm gonna hire a person for that. So giving me the tools that the specialist use is not gonna be useful to me.

could you let's get a little bit technical here.

Could you talk about how that communication happens? How does my agent talk to this registry? What like this schema looks like under the hood? What do you use? Like, how does effectively, how do you envision this world where my agent Go and ask a marketplace of agents, ideas, et cetera. First of all, is it a centralized registry? Will there be multiple registers that you consider? Will you guys provide one? And then actually how the conversation happens. We'd love to hear those details.

Yeah, so the agent registry is something that's on our

road map, so it's definitely enabled by the protocol right now, but it's not something we've released yet. So right now, with an agent publishes a card, think of it like a business card or a resume or a capabilities card, where the agent can describe the sorts of things that it can do examples of how to interact with the types of modalities that speaks. For example, you can send me images, you can send me documents, I need to reply in an iframe. I need to reply with a video frame, things like that. The idea is something that's coming on our roadmap is there's going to be many registries. I think public registries are interesting. registry like robust dot text. Where web crawlers can, can find agents or your browser can find agents that are available on your website, as well. And then also having things like app stores or agent stores or agent gardens where, private agents are curated agents curated by yourself or by your organization or your team, et cetera. Those are going to pop up within companies as well. And so that's something that we're actively looking at. that registry service, where you have multiple agents, we are looking at. How in the agent card can we interrogate each agent and say, Hey, who is best equipped to handle this specific task, especially when you have agents that can perform overlapping tasks or things that sound similar, even though they may be dealing with different context.

So one example would be an enterprise who uses GitHub and

GitLab interchangeably and like they have agents for both, for example. But GitHub and GitLab could be examples of tools, right? So agents are more things that can go and perform multiple tasks and then come back. I want to talk about the task protocol as well. there's also the async and sync example, right? Like agents can go off on a long tangent. how, how does the protocol handle the tasking and async and come back to me with an answer, especially in the era of we're looking at, Gemini is a reasoning model. The 2. 5, 03 is a reasoning model. Those reasoning models take longer. They do agentic things as well. once agents are built on top of these, so it just takes longer to execute tasks. How does the agent, sorry, the A2A protocol, handle asynchronicity and tasks, the handling and offloading.

Yeah.

it was originally designed for these really complex, long running tasks. So things like, hey, I want to place an order with the store. The store wants to ask you questions. you ask questions back and say, oh, can I get a discount on this? The store then says, I have to check with the manager and that may take a day or they have to contact the part supplier, or they have to contact someone else to figure out what to do. And this could take place over, Not just seconds, but it could take minutes or hours or days or weeks, something like, repairing a car or booking a very complicated trip that has many vendors involved in it. and so with respect to that, yeah, there's a central abstraction called a task. So it's a stateful entity. It represents a goal that the agents on either side or that the client agent and the server agent are collaborating on. the client sends contextual information and, and descriptions of what needs to happen to the server agent. The searcher. The server agent responds with status, what it's working on, and also if it needs any more information. as it's generating responses, it can stream those back as well. and then if the remote side is blocked on needing more information, it can signal back to the. To the client that, I need you to give me this structured information, or I need you to give me more documents, or I need you to give me more information. some emerging usage that we're seeing that wasn't really anticipated is people are also using ADA as a app to agent protocol, where I have some app that is not an agent, but I want it to speak with an agent. I want it to be able to communicate easily with the agent. and so that's where we're looking at adding more support for sort of this. more quick moving conversation access.

So app to agents, is different than agent to agent.

I got it. I want to chat with you about, so we're talking about like capability discovery. You briefly mentioned this, but I think that's very important as well. that like the conversation is no longer just textual. Agents are not only performing textual tasks. I'm just thinking about, the stuff we talk about here on Thursday is like, there's video AI generation, a bunch, and sometimes those take a long time. And maybe I wanted to go and create seven videos, review them, et cetera. I want like video generating agents. Maybe I want to connect it to a lip syncing, whatever agent, et cetera. but then I want to see this and there's the capability discovery, but there's also the, the, the kind of what language are we speaking discovery as well. And you briefly mentioned like the types of stuff. Could you speak about that? That approach is, I think is very interesting. It's not only textual. You mentioned iframe, I think you mentioned the video frame as well.

Yeah.

Yeah. so Texas definitely the dominant modality today, but we do see a lot of models that not just models. We see agents that work in different modalities and even have custom branded experiences with them. So we, we definitely want the agents to communicate in their natural modality. So there, there could be scenarios where, Agents are audio only or video only. They should be able to negotiate that through the protocol. There are times where they want to, start textual and then move to audio or move to video. You could think of a scenario where, a client wants to ask a, a work helper agent, help me reset my password and says, Oh, we need to do an identity verification. We needed to start a video stream with you and an identity provider. and so you work with that identity provider with a, with video stream. There are also cases where. one of the agents really needs to have their branded experience, in front of their user for, business reasons or trust reasons or relationship reasons. and so that agent may say, Hey, I, I need to render all my content. You need to give me a HTML, iframe that, that I can render in and I'll do that, or I'll provide you a frame and you can embed that there. So it's really up to both agents to determine what the right modality is for, the given task.

And I love the fact that it's like, it's built in.

And, I think we'll see a lot of innovation in there, especially in the iframe area, but like also in the video and modalities area, I think we'll see like a lot of innovation, very exciting to see this, but you did mention a password changing agent, and I think this moves us to the next, conversation about just protocols in general and, security and authentication. And I think that that's one very important piece because we did mention Registries. Let's say we did mention enterprise use cases internal to the enterprise. We're like, I, as an it provider for the enterprise, we'll expose multiple agents to the whole organization for them to use those that will supposedly be confined within the off scope of the enterprise as well. could you talk about, How you guys approach, security enough as a concept in the protocol, what's treated, are there primitives for this and then how identity and scopes, like all these things work.

Yeah, that's a that's a great question.

So one of the core design principles is that we really, especially as it comes to enterprise readiness, authentication, tracing, et cetera. We do not want to invent anything new. We want to make sure that it works with everything that exists in the world because of that. We designed it to be a point to point. Protocol between essentially two services. instead of having multiple identities, instead of having to fan out or go through many proxies, we want the client and the server to be treated as applications as normal applications or services within the enterprise. And because of that, you carry your application identity and your user identity and standard OAuth, standard IDP, standard internal enterprise off, is intended to just work.

And so

the idea is identity, identity information is not carried

in the payload of the protocol. It's carried in HTTP, so it's carried in HTTP headers.

I see.

So that, just to try to wrap my head around this, me as a user, not a developer, me as a user of an agent, That I have access to, I don't know, our Jira and my Gmail specific with scope and a specific Google group. I would still have those accesses. Will I be able to delegate this to my agent to then delegate this to the other agent via the protocol? so that the server agent will be able to carry tasks in my name, in my scope?

Yeah, so what will happen is you'll be using some agent

and when that agent wants to work with the JIRA agent, the JIRA agent will reply within the protocol. Hey, here's the identity information I need from your user. your application will then say, hey, you need to log in with JIRA and get a token, assuming that it uses OAuth. supposedly that would

be a standard OAuth process that I, the user, am very familiar with.

Yes, and then you'll go ahead and you'll do the standard OAuth flow

with JIRA and you'll, authorize your agent to speak to JIRA on your behalf. And then those will be passed through HTTP headers that here's your user ID, here's your bearer token, et cetera, that are in the HTTP headers of the A2A request to actually make the changes.

So you'll do

the OAuth handshake out of band, you'll do whatever IAM, access

control, identity management, work with the IDP, whatever that service provider requires, you'll do that outside of the protocol, and then you'll carry the credential material and the HTTP

headers.

You mentioned out of band, could you elaborate what that means?

Yeah, so you don't do the OAuth handshake, for example,

in A2A, there's not an A2A Method for, for getting credentials. Instead, a to a server will say, Hey, here's where you need to go get credentials from support, whatever authentication scheme, or protocols is required. so if that's OIDC, if that's OAuth, if that's, some SSL, something proprietary, you can do that as well. you work directly with that identity provider to grab the materials and then you can include them in subsequent A to A requests. Here

we go.

That's, that's great. moving forward a little bit, moving forward a little bit, I want to talk about governance neutrality and like all these things for many folks who like think about adopting a protocol. Vendor neutrality is like very important, and, but also there's like standards bodies as well. How are you guys thinking about governance and how do you think about standards bodies? How are you thinking about kind of proposals, suggestions, to the protocol from the different companies? we'd love to hear your kind of approach and thinking currently about this.

Yeah.

So right now it is in a Google GitHub repo. when things stabilize, we do plan on moving it to a community owned repo, maybe something like the Linux foundation. We're not exactly sure yet. We'll have a, we plan on having a completely open governance model. This is not something that we want to gatekeep. This is really in the best interest of the, of the industry and of. Consumers, to have all of their agents actually be able to work together and not have any sort of, protocol or competitive advantage, for company A over company B because of special features. We really want this to be a standard and become a standard quickly and have open governance.

I think Google has a long history of, of those things happening

as well, which I probe, speaking of like protocol submissions as well, could you speak to how this is used within the scope of what you can talk about, how this is used within Google, whether or not this is used already within Google, and then also other companies that are already like using this protocol for their like agents or building against this protocol, could you speak to like actual use cases that you've seen, and come up and like some examples.

Yeah.

So I'm Google next last week. I believe we announced a product called agent space. that is now generally available, which is a, internal search engine for your company. And it also has agents in it. And then, it has a no code agent builder and the ability to host third party agents in there as well. So agent space is starting to adopt a to a at some point agent space. All of the agentic interactions will be a to a, agent space. Also is working with a lot of partners. So you can build agents on different platforms. You can build them in whatever framework you want. You can build them in link chain. You can build them in crew AI. You can build them from other low code, no code vendors like Salesforce, and have all of those work in, in agent space. and the way it works is all of them will speak a to a, so there's not a lot of happy, heavy lifting that, agent space actually has to do. There's other products in Google that are, that are starting to adopt it as well. I don't want to. Say their names right now, because I don't know if they will like when they're going to announce or what they're going to announce, but there's a lot of momentum inside Google.

Awesome.

we are excited to see, and also obviously we're like a supporting partner at this as well. this, this thing, I would love to chat with you about, how are you thinking from a protocol level about transparency to the developer and maybe not the user, the developer of the client agent, let's say. because at least part of the concern that we have, and we have created an RFC to this, to the MCP spec is that, the more, Things that are happening outside of the client that I built as a developer my agent that I built as developer The less I have of the real team to like what's happening generally the less I have the ability to debug to evaluate To see to see how my users interact with this, right? So we open this kind of like a hotel and MCP support as well Have you guys given any thought into that kind of category of problems for developers? and if If yes, what are your current thoughts about like how developing and developer experience with A2A could happen?

Yeah, absolutely.

That's one of the enterprise concerns that we have in the operability concerns. You can't have a mesh of services if you can't observe them. And I think the key word there is service. We're treating agents like standard services, that you would have either micro services or cross organization services. And because of that, hotel, So instrument your HTTP clients, instrument your, your SDK. I think as we're building out the SDK, we're absolutely going to have hotel support in there from the beginning as well, so that it can be customized in each installation to record wherever you need it to record. But the, the key feature of simplicity that we hope to design into it is. really just model it like a normal client server application and use everything that already exists. Do not invent anything new.

Okay.

I'll tell for folks who aren't listening who are not as familiar as another protocol, open protocol called open telemetry, which specifically specifies how, in the open way companies like Ways and Biases, Weave, and like other, can, can collect this telemetry from multiple servers and, and, and compile them into something that the developer can see. I have a question that came from Yam here on the panel as well. How do you define like agent failed cases? not, we know in the world of LLMs, like a bunch of hallucinations happen. Sometimes failures happens, maybe like errors happening. what is defined the agent failure kind of use case? Does the client or server agent like communicate back? Hey, I wasn't able to do this. and also how does it communicate over? Over time. Is there like progress updates about I'm about to do this and then oh I failed etc. Like what? What definition do you have for the agent did exactly what I wanted or and how does it update you on the progress?

Yeah.

So that's core to the task abstraction that we have. That's the main object during communication, between the two agents. So a task carries with it a state, the state is set by the server agent. and, the server agent can respond with all sorts of updates and they're generally categorized into two things. We call them, messages or artifacts. Artifacts are official generated results for the task that's happening. And there can be multiple artifacts associated with one request, like I want seven images. Each image would be its own artifact. messages are for everything else. It could be thoughts, it could be status, it could be requests for more information. some of the states officially are like, Hey, I've failed, or hey, I'm done, or hey, I need more information. It doesn't say, Hey, and I'm just realizing, but it says, status is failed. Status is complete. even when the status is complete, the, the client can evaluate the results of the artifacts, the result of what the agent generated and decided that decide that it's not good enough and say, like an example that we have his, client says the agent, create an image of a rabbit for me. And the, the agent generates a rabbit and streams it back and sends it to the client and says, I'm complete in the task within that task, the client can then send, no, no, no, I want it to be read. And that should reactivate the task on the other side. The task would then move from completed to working things would stream back and then it would be completed again. And you'd get a new artifact, which is the red image of the. of the Revit. So the, the state is carried in task and then both sides are able to send each other messages.

So we covered, I think, mo most of the primitives, Then we

have the secure collaboration, which like authentication is a part of it. And you, you mentioned standard auth is supported in like regular enterprise of things. You don't want to reinvent the wheel within the protocol. You want to just have it ride, on top of the protocol to expose my. Credentials to the. Let's say through my client server, a client agent, I need to get used to saying this client agent, and then several agents as well, tasks and state management, which we covered as well, like the continuous asynchronous tasks, happening. so that, things can happen in the background and then, notifications are happening back backwards as well, right? So if my agent or my client application, let's say, has the capability to notify me via an iOS app or notification, supposedly then I can get, I can, be kept apprised about like the happenings of the server agent, right? This is the notification thing.

Yeah, there's another feature that that you just mentioned

the push notification feature. yeah, so you can create a task. client can create a task and the server can stream back results and give status updates. But for extremely long running tasks, like especially ones that could take hours or days, we don't anticipate the client and server being connected the whole time. So instead, the client can provide a push notification service that the server can send updates to. That has another set of a thing. Authentication authorization that's required because then the server becomes the client so we have details in our spec for that. But the idea is that, as the server agent is changing states or generating artifacts, even if it's disconnected from the client, if the client has provided, Some sort of notification callback, that the server can post results there.

And that, notification system, that's built into the protocol?

Or is that like writing? Yes. I don't know, the iOS and Android ones, or the web one, web worker one. That's, that's built into

the

protocol.

Okay. so basically, in the protocol, if I build a client agent and I wanted to talk to agents that take three days to do something while waiting on humans, let's say to fix my car and send me a notification, your car is fixed, come pick it up. the live connection via. I don't know, web sockets, or we didn't actually talk about like the life connections happening on the protocol, but that doesn't need to be maintained for three days straight. Very, very difficult for companies to build those long connections as well. One of the issues with MCP was also stateless versus stateful and how, how to, how long to carry this connection as well. so you have a notification service built in with the, to a the thing that you quickly mentioned, I want to like, just make sure that we clarify the client becomes the, the server and the server becomes the client. That only refers to the push notification communication, right? Because like in this case, me as a client, I like open up the, here's where to send notifications. And then the server agents now subscribe to that and send notifications back. So like only in that level, right?

Yes.

Yes. So only, only in the case of notifications and it's more from the authentication authorization perspective. It's just really, it's really risky to just open up a port and say, send me updates here. And here's a cookie. Instead, we highly, highly, that is possible, but we highly, highly recommend that, the notification authentication. Service, which may not be the client itself. It may be something internally that posts things to a pub sub queue or something, fully authenticates and authorizes the, the server agent.

All right.

Todd, I know that I promised you like to be very mindful of your time in the last, I know we're like over for a few minutes, just like one last question. Is there anything important that we didn't cover that I didn't ask you about the protocol? and maybe where can folks go and find out more and how they can actually start interacting with it?

Yeah.

So go to the GitHub project. so it's, I believe it's github. com slash Google slash eight a, there's a lot of PRS, there's a lot of open issues. We're having lots of discussions in the open. We're going to continue to evolve things in the open as we work towards, official SDK and official kind of production release. And as more Google products come out there, and work with more industry partners to get more examples and have more products there. So try it out, give us feedback. Send us PRs. We love code.

That's incredible.

Todd, thank you so much for coming. Shout out to you and the team who worked on this in, in, in Google. All right, folks I hope this was helpful to folks in the audience as well as with protocols Like yam said we can't use this now like this decays on its way But the same happened with the mcp protocol. We talked about this at Thanksgiving. I remember like last week, last year, we announced this and we're like, we didn't have a lot to say. There was not a lot of implementation. And then recently, MCP just like absolutely blew up. Everybody's supporting this Google, Microsoft, GitHub, Anthropic OpenAI, like all of the big players are now supporting MCP as well. It takes time for a protocol adoption. it needs critical mass of users, developers, and companies. It looks like Google is doing the right thing. And, hopefully this gave you a little bit of an insight about what's coming. So if you want to be an expert, In what's coming to enterprises in six to eight to 12 months, now is the time to start. And as a listener of Thursday, I, you got the alpha, hopefully. moving on super quick. I know we're like past 10 minutes, but I cannot skip without telling you like the things that happened in the world of video. we won't have that much time to actually show anything. I just want to like, announce a few things. So VO two, ve O2 from Google is the video generation model is now in, in, general. availability in the Gemini app in AI Dev as well. So if you, if you use Gemini, I believe Gemini, will I have access to it? let's see. help your guy. No, no, I don't have access to it. I, I don't think, but if you go to AI. dev, which is the shortcut URL that I love for AI studio, you'll see the new, I have video journalists right here and you have VO2 video generation. You can do two results and you can ask for stuff. You can even upload images, super quick, and then you can just generate a bunch of stuff, with video in VO2, VO2 is still considered one of the best video generation models out there. Very, very strongly realism. very, very good. It's also fast. So other generating images, they generate up to eight seconds and then, other generating video images generally look longer by other. other ones that just came out called cling. I really want to show you cling cling so far. Wasn't useful to you on the U S side because cling did require a Chinese phone number. Cling just released 2. 0. And I said, throughout a very late night to just try to summarize all of them. Let me actually open this in another thing. Hold on one second. So Kling now released, not only a state of the art video, they claim state of the video model they also released a full suite of tools to create. as clean guy. com that's now available to the Western kind of, folks with non Chinese phone numbers, which is, I think is very, very important. You could use cling via foul and you guys know, we'd love fell. They either great friends of the pod and you could use clean their, their partners, but also in the experience of clean itself. there is two things, the model clean point to the image model colors 2. 0 also. So they have a, Not as quite as Sora images, but like very, very close image generation as well. and, oh yeah. for me thinking that the phone was out for a while, Yeah, so maybe, maybe I'm overstating how much just was released, folks, they're listening, Wolfram, let me know in the chat that like the, the Google account experience and phone, the phone number has been gone for a while. I only got access to it recently. The thing that I wanted to highlight specifically is, there's a few things that I've been doing, playing around with some stuff. this very interesting new multi elements, editor that they have. The Dave kind of outdated thing, 1. 6, not the, not the new one, to be able to reason about, an image and then have text images all together in this one interface. I've never seen anything quite like this before. So you can like upload, I don't know, you can like paste in video. let me see if I can, what was this? Let's say like this, this is my video. and then you can reason about this video. You can tag the video and say, Hey, replace this, this thing in this video with this thing from this image. I haven't quite seen the conversation, multimodal conversation like this anywhere else. I just really want to highlight this. They built a new version of cling 1. 6 that I think they call. I have a note about this MVL prompt language, concept, MVL prompt language, MVL stands for multimodal visual language, that inlines images inside the text prompt for precise control. This I think is the thing that I wanted to highlight the most. You can say, I didn't upload my image, but you can say oh, they have an example here. They have an example. So following the structure of swap content from at image from content from at video and a couple of multiple images, we can say in this video swap is I don't know, hoodie to this type of hoodie. And you can add an image of the hoodie and say, in, in the UI, this allows for extreme precise control in the video generation. The, the kind that like AI artists like have been craving for, for a while. they also have effects. Effects are pretty cool. Lip sync and I think the least but not very also interesting, very interesting is sound generation as well. You can ask for like thunder or car horn, which I hear people. walking in the car horn generation, but car horn as well. so very cool. The thing that I love the most about the, the kind of the new update is that it's a holistic editing experience. Now you can start with image generation right there in there. Not only that they integrate deep seek. Within the thing. So if you don't know your prompts, you can ask DeepSync that's built in to help you with prompts. You go to image, to video generation, you go to from video to extend, and then to lip sync and sound generation. They try to build this holistic tool because some of us who does AI art for fun, they know that we have to go to 17, 000 like different, servers, like here's one for prompts. Here's one for the video. Here's one for images and combine them. And they're really going after this one holistic platform and is really cool. And, you can now subscribe with, paying with dollars as well. That's, I think is it folks. There's a few more releases on the open source side of the video stuff by dense released seaweed, and then, one X released like a new model as well. We didn't get to talk about dolphin Gemma. Womp, but super quickly, folks at Google collaborated with like researchers to try to build a language model that understands dolphins and dolphin sounds, which is super, super cool. We didn't get to. do dolphin sounds ourselves, which we wanted to. But I think other than that, we covered pretty much everything, in a little bit over two hours and 15 minutes on Thursday night. So folks, Thank you so much for joining. Yum. Thank you. Nistan was here before Wolfram. Thank you so much for joining and trying to cover this insane week of news, including an in depth interview about a protocol as well. Not an easy two hours as well. we celebrated the launch of O3 and O4 mini and then 4. 1, 4. 1 nano and 4. 1 mini as well. We chat about, voice and audio stuff, which is talking about VO2 and Kling. we did the in depth interview with Todd. Seagal from Google that works on the A2A protocol and why A2A matters in the world where MCP is taking place as well. We chat about BitNet and Intellect too, and just like a crazy week of updates. I really appreciate everybody's time here, especially folks who tune in and listen from week to week. As a reminder, if you missed any part of the show. You're welcome to subscribe to thursday. news. You're welcome to subscribe to our YouTube channel, as well, thursday. news. yt. We crossed a thousand followers and we're raising fast in rankings as well, last week. And, if, and if you liked, and the show was helpful to you, please share with a friend of yours. That's the best way to support the show, to share it with friends. With that, I think that the next week is going to be insane as well. And so I think some breaking news is going to come even today. I cannot say which. But keep, keep, Wolfram said, follow us on X because we will be on top of it, even if we're not in the show format. I think I'm done with live streaming for this week, but we'll see, maybe tomorrow something else will happen, but hopefully I'm done with live streaming this week. And with that said, I'm going to go and start editing the show. Thank you so much, everybody, for joining and we'll see you here next week. Bye. Bye bye.