ThursdAI - Mar 20 - OpenAIs new voices, Mistral Small, NVIDIA GTC recap & Nemotron, new SOTA vision from Roboflow & more AI news

Hello and welcome everyone to Thursday.

I, my name is Alex Volkov. I'm an AI Avengers with weights and biases. And you're on Thursday for March 20th today. And with me, my co hosts here, Wolfram and Nistan. Welcome folks. Hey Wolfram, how are you doing?

I don't know.

I'm doing great. Great week. Great day. I'm excited. There's

a bunch of open source stuff to talk about.

welcome everybody who listens and tunes into the show. Thursday. ai is your latest and greatest news about what happened in the AI last week. And, we're here to talk about multiple things today. There's a bunch of open source. there is an open AI live stream that's happening at 10 a. m pacific, which is in an hour and a half. we are obviously going to restream this I know what it's about, and I'm going to tell you, I got a little bit of a sneak peek interest in it and it's so dope. besides this though, we have other breaking news here, soon to come as well. I'm not going to tell you what those are about. You have to stick around with us, I just want to mention for folks who are joining us for the first time that, last week we celebrated two years of the live stream that was awesome. I really enjoyed it. I like the output of love that people said, and like, congratulations that people sent me. It was awesome to see. it's just getting started. the speed with which the AI models are releasing is ever increasing. We have a bunch of guests as well. So I'm very, very excited to tell you about all these things. And, I think it's time for like a quick TLDR of everything we're going to talk about. because I guess that's how we started the show. So let me, let me just share this with folks who are watching, but folks who are not watching, you can just tune in. basically we started the show with a TLDR. I had some feedback from folks who were like, Hey, don't tell us about what you're going to talk about. Just talk about this. And I disagree. I disagree because we are here for two hours. Not many people have two hours to sit on a Thursday. Many people just want to know what's going on. And then if interest peaks and they want to just stay with us. So I disagree. But I appreciate the feedback. so let's do the open source TLDR section. So in the open source we have, a great release from Mistral. Folks, Mistral is back with open source. Let's go. Applause. Yeah, let's go Mistral. we're very much welcoming. Back to the stage Mistral's models with an Apache 2 license their multimodal Mistral small 3. 1 24 billion parameters that fits on a single GPU. I guess was released Wolfram you play with this I'm assuming we're gonna chat about Mistral coming back to the land of open source. Ah a very interesting release from the fridge maker LG that makes great fridges By the way, my screen here. My monitor here is also an LG monitor. They have an AI side, apparently, I didn't know about this and they released a new LLM called X01. It's a thinker also. So X01 deep, a 32 billion parameters thinking model. I actually had a chance to test it. ExaOne was released, from LG, from all places, so we have like a new, new entry to the stage of LLMs. It's very interesting. ByteDance released something called DAPO, D A P O, which apparently is better than GRPO that DeepSeq released, which is a RL training method, really reinforcement learning training method. that's going to be very interesting to look through, And then, folks, finally, Finally, we got some open source from NVIDIA for the second time. NVIDIA drops, because GTC happened this week. NVIDIA finally drops Llama Nematron. they dropped Nematron Super and Nematron Nano. and, we've been talking about these models from the last time that Jensen was on stage, and now, Jensen was on stage again, and now they actually dropped the models, the models look pretty good, so we're gonna chat about them as well. I invited NVIDIA friends, but a formal request to the NVIDIA PR department. We love NVIDIA! We absolutely love Nvidia, all of us. Please let your folks come to the show. Okay. That's all I say. but I did request some folks to come to the show and hopefully the PR department will, will understand that promoting their stuff on Thursday is a good idea. in the big companies, it seemed quiet this week. It looks like nobody wanted to step on the toes of Jensen's performance. But, Google released a few things that are interesting in the Gemini app. We released deep research for free. Now everybody can access deep research, which is great. they also released Canvas, which is super cool. you can now build apps within Gemini and actually see them. so we're going to chat about this and also they have a bunch of other stuff, to mention. Not to mention that last week we told you about the native image output of Gemini. And it's been blowing up all over my timeline. It's incredible if you haven't tested it out. OpenAI just yesterday decided to say, Hey, remember that pro tier level of a model that we have with like extra, extra compute? we're going to make this available in the API for oligarchs only because it costs a ridiculous amount of money. OpenAI makes O1 Pro, which is not a model, which is Something else we try to understand what exactly this is, but not a different model. It's 01, but probably a lot more compute and resources. They make this available in the API, and, and it costs 600 per 1 million tokens, 600 for outputs for 1 million tokens outputs. If you code with this, if you vibe code with this, you better already have VCs backing your startup because it's ridiculously, ridiculously expensive, but hey. Maybe some people want it. Definitely some people ask for it. It's way easier to use repo prompt and just go to OpenAI Pro for 200 bucks a month. we have a quick, NVIDIA GTC recap that I hope we will test out. We're moving on to vision and video step fun, a company that already gave us like a great model for video. They released their image to video model, upgraded from their text to video model. looks pretty good. They claim state of the art, but it looks pretty good. just the breaking news announcement. AI breaking news coming at you only on Thursday night. So the breaking news that we have. Just literally just before starting the show is that open the eyes live stream is going to go live at 10 a. m. And the only thing that we can talk about publicly right now is they released a. Tweet and in that tweet there's a video in that video. There's a voice that says live stream at 10 a. m. Very Empathetic. I don't know. So there's gonna be a live stream at 10 a. m We're gonna restream this folks so you don't have to miss anything You can be on Thursday. I and also enjoy open the eyes releases, though. I will say They tend to go long. So sometimes they have to announce like one thing and then they're like, oh, let's talk about this for 15 minutes. We don't have that much time on the show to give them 15 minutes. We'll see. We'll see how that goes. But, I will tell you though, I will tell you, given that I had a little bit of a Preview of what's going on in this live stream. I already have queued up a guest who's going to come and talk to us about the implications of what they're going to release. So that's a little sneak peek for you. I'm very excited to be able to have this, because, the more time we're here on the show, the more kind of, we get, to see a little bit before, and I love this. And also for everybody who's listening, we respect embargo with a strict respect. And so if you were to give us some news, we will only talk about this when it's public. Speaking of public, in the vision and video category, we will have another breaking news with another guest, very, very interesting guests, this week. Also in voice and audio and I think Wolfram you had a chance to test this out some a company called Canopy Labs I've never talked about on the show before I never heard about on the show before released Orpheus 3b It's a natural sounding speech language model. It's not just a DTS. It's an actual Model that's built, I think on llama 3. 1 or 3. 3 and, they trained like a TTS into that thing. So you can actually ask it for directions. You can ask it for a bunch of stuff. It's 3 billion parameters and they're going to release the tiniest version soon as well. it has zero shot voice cloning, which I tried to run and, CUDA ran out of memory for me. But this is so cool. This is like a state of the art, TTS. Language model, basically, that you can prompt to talk to you and multiple things. And just before the show, NVIDIA open sourced Canary, which is a 1 billion parameter and 180 million parameter speech recognition and translation, so basically like whisper competitor that takes like number two on the speech arena based on vibe of a friend from hug and face. So a lot of news in the voice and audio, a lot of news, including some upcoming news very soon. I want to talk about this, this segment. We don't usually cover AIR diffusion in 3d, but. Folks, this week we're going to cover this. Tencent released Hunyuan 3D. Do you guys remember Tencent? Of course you remember Tencent. One of the best performing video models right now is Hunyuan Video. And they have been just on a tear of releases lately. And this is also, I believe, open source. And Hunyuan 3D Version two is an absolute state of the art 3d model right now. Not only that, They released a 3d, multi view, which basically, if you imagine, we're going to cover this, but basically a state of the art 3d model, let's, let's keep over the details in the TLDR is, coming at us from Houdini on 3d. NVIDIA also released something called Cosmos Transfer. they dropped a conditional world generation with adaptive multimodal control. It's a mouthful and we'll see if it's going to be interesting to talk about. And I think we're coming up on the end of our TLDR and in the tool section, we have, our friends from RC, A R C E E, our friends from RC did like a bunch of fine tools, did merge kit, did like a bunch of great stuff, our friends from over there, and we'll have Lucas on, hopefully, to talk about their Conductor, Conductor is a new release from RC that is a model router, routing your requests to different places, so that's going to be awesome. And then also in tools, Cursor ship something called Cloud 3. 7 Max that many people were concerned is like a new model, but it's basically just like unlimited Cursor 3. 7 with a bunch of reasoning tokens. Notebook LM is about to release a new feature called Mind Maps, which I find super, super cool. You just upload a bunch of stuff and you'll just create Mind Maps for you. You'll be able to create your own now. And. The one of the funnest things that I've seen this week is using a Gemini, code drawing, which is, which is basically using the Gemini image output native features and you can just draw with it together and we're going to play around with this. It's super cool. I think that's it for the TLDR folks. There's a busy week for AI this week. Busy week. There's a bunch of stuff, not to mention that nobody wanted to go live because GTC, nobody wanted to outsend Jensen. I will just ask super quick. Folks, how are you feeling about this week in the eye? Let's get started or am I, am I missing anything? This thing Wolfram and also folks who are listening to us, please feel free to leave me a comment about, what, what we just released and yeah, we have some comments already, but if you have anything that we haven't mentioned in the TLDR, please, please add a comment. We'll absolutely get to it as well.

It's really exciting now that we have

a lot of cool local models. I have had so much difficulty choosing which one to run. We had Centra on the show, that was actually breaking news when she, introduced us to, the new Cohere command R model. So that is excellent. And Gemma3, we had Gemma3 and now we have Mistra, which we will cover soon. it's very hard to choose and they are all great.

Command a, since Sandra was on the show and we announced it as

breaking news, almost entirely on the show, command, they went on LM, LM arena and got like a very decent, decent score. So everything we talked about was great. and, yeah, I think it's about just anything I missed or we're good.

I got retweeted by Aiden, the CEO

of Cohere.

it was about what we talked about last week, because, you do

want to use the cohere models to do, especially for data gen work. If you want to get some other distribution, that doesn't sound like Sonnet or GPT 4, if you're doing data work, you want to get some better distribution and cohere is pretty amazing for that because they do all their own data and stuff. Like they don't actually use. other models. but, Yeah, there's, a lot of, things to play around with.

So folks, I think we're going to start with our

favorite corner that we always love. It's going to be open source AI. Let's go.

Open source AI. Let's get it started. I feel like I need to upgrade that with a little bit more excitement. It felt a little bit like, hmm, let's get it started. And it needs to be, let's get it started. All right, folks, so we're jumping right in. And the first thing that we have to talk about is Mistral's new small I gotta, I gotta be very like the Frank, the definition of a small to me is like very interesting, but mistral small is a 21, 24 billion parameter, model mistral small 3. We already had mistral small, and this one has a bunch of cool things about it. First of all, they love this triangle on the left, for GPK diamond, for example, they're showing, In the realm of the similar model sizes, they're showing an absolutely great performance compared to Gemma 27, Gemma 3 that we talked about. it's a little bit larger than Gemma 3, but still, same ish area. They also throw GPD 4. 0 mini and cloud 3. 5 haiku to there. Mistral's most 24 billion parameters and gets, very high results on, on. Very interesting benchmarks like GPT, GPQA Diamond, GPQA Main, and they're beating, Cohere's Aya Vision and, Gemma 3, and GPT 40 Mini. Wolfram, you had a chance to play with, with Mistral a little bit?

Yeah, I even did some MMU Pro CS tests, I regularly do.

And yeah, it beats Gemma 3, which is a bit bigger because Gemma 3 is a 27B and this is 24B. that is better. Also notice that in German, it's a little better. In general. I think I like it a little better, but it's very close. And Gemma three was really a great release. So that's the AI, scene we are in right now, where a week later, you already have some new toys to play with. And it's so far back. they are both great. And I personally liked the Mr. Small a little more and it's Apache two licensed and a multilingual multimodal, which is actually a bit of a drawback compared to Gemma because the Google people made sure that, the. Visual stuff already works in Lama CPP or at least in, Onama, which is not the case here. I could not test the visual stuff with this model yet, but the text generation, excellent.

Text generation is excellent, on the multimodal instruct

benchmarks, they're getting on MathVista, they're beating Gemma a little bit. I think they're coming very close on MMMU and MMMU Pro, it looks like they're very, very close to Gemma. Every other model is basically that they chose to compare to, right? obviously, we know and love Mistral, but also the usual meme that we have for Mistral folks is like, where's, where's CoNVL, where's this, where's that, where's this, where's that? This is the usual meme for Mistral, but hey, whatever you guys chose to do, if you choose to give us Apache two models that are multimodal, we're not going to say, no, we're going to be celebrating this. Absolutely. So shout out to Mistral folks.

a small model and you can run it on 48 gigabyte of VRAM, but

I think you can run it quite fast, on 24 and it beats the Mistral large model. when you compare the quants that fit in the VRAM. So for me, I would rather run in my systems, the small model in less quantized than the highly quantized. large model. So it's really, really good.

Yeah.

Long context as well. How much context do we have here? They say 128k. 128k tokens. we're just there folks. We're just there. We're in the realm of like, this is most of what you need probably for most day to day unless you go like crazy amounts, 128k tokens. I still, I still like, yeah, like last week we reminisced about jumping from 4k to 8k and celebrating this. And I still remember this. It was like, we celebrated Lama, was it Lama, Lama two? I think it was right from 4k to 8k jump with rope. We were like, yeah, 8k, there's so much context folks. 128k is pretty much all you need for regular use. I don't think you need like way, way more unless you have, it's pretty much hard to, to, to scrape all of this. Nissan, how much was the, the, the Jeffrey, the JFK release? I think you posted like, it's like 2 million tokens.

Yeah, 1.

8 million tokens. So it actually fits into a Gemini Pro. All of it. You can just dump it in there and then you can, that's insane.

years of records and they fit into like a context window of Gemini Pro.

if you take segments of them, you can run them locally with. Mistral, this is me trying to bring back the conversation. but basically, unless you have 50 years of JFK records of 2 million tokens, this model is pretty much all you need with 128. it's, it's, it's insane.

So I think most people running it locally will go

to a 32 K for example, because the quality also degrades if you go. beyond that. But it's already great that you can go that far and you have the option to go even further when you need it. I think usually,

usually Mistral is very good at longer context answers as well.

I've noticed this to be quite consistent. With them, since the first, Mistral large one that I, one of the very first models, they've always been. Now I haven't tested this one, but I've tested every other Mistral model and they've been very good at stuff, like rewrite this whole code base of 50, 000 tokens, but remove these comments or just add more comments. It will actually do that. It's quite a workhorse of a model, for that. So I, I, yeah, I would advise to be actually tested if it works for them. Because, the models do have very particular abilities in different jobs. So if you're going to use it where you want something consistent, where you have, I don't know, pieces of 50, 000 tokens or 100, 000 tokens of code or whatever it may be, and you want that rewritten in a certain way, and you want it to do it reliably, the Mistral models do tend to be That's at that job I found out, amongst, amongst other things. So yeah, you should test that for yourself.

One thing we should absolutely discuss is that Mistral

is releasing the base model as well for the fine tuning community. The base model is very important because they want to do their own stuff compared to their instructional DPO, whatever. our folks, our friends from all around the fine tuning community are very, very happy about this. Folks Apache two with base. Mistral is back. Basically. Let's call it. Mistral is back beating Gemma from Google just a week after Gemma released. This is an incredible release from Mistral folks. Again, applause to our friends on Mistral. And

it's less censored actually.

I tested both with the same prompts and Mistral is more compliant.

That's awesome.

applause to Mistral. As we move forward, you can try this model on their le platforme. I don't know how to pronounce this. Sorry, I don't, I don't speak French. but you can try this on their, the platform called le platforme. the model is also on Google Vertex and, soon we are available via NVIDIA NIM, which is their like, NVIDIA inference, modules. I believe that's the thing. Maybe not. and everywhere, basically everywhere. they mentioned, I want to shout out this. They mentioned our friends from new research, news research. they mentioned DeepRMS 24b, because that was the base of Mistral small three. and I'm pretty much assuming that like the folks at news are already cooking, like with this model, because it's significantly better. and, this has basically inspired them to release the base and instruct versions. So folks. From the community, thank you for this research to inspire Mistral to release and thank you Mistral for releasing the base model. I think that's pretty much all, folks. We are moving forward because we have a busy show. Let's talk about, let's talk about the, the, the refrigerator in the room. I think I have endless puns on this topic, folks. Cause, I think I have an LG, like, I think I actually have an LG fridge. LG's AI, LG research. Which is a, an arm LG is a huge company. Of course, I'm, I'm, I'm making jokes. They don't want to make refrigerators also washing machines. Also, they have released, they have a research, LG research. ai. They have publications here. There are a bunch of stuff. Folks is like a serious, serious folks, working in there. It's a huge company. so please, if you're listening, tuning into us from LG, please forgive our jokes. we're just having fun and they released. Deep model like an LLM that's reasoning. I don't actually believe that it's a fine. Don't know anything I haven't seen it could be but as far as I saw on Hugging Face and Hugging Face does a real good job of showing you where it's like fine tuned from the base model is This model. So I haven't seen any evidence of this being like a fine tune of anything else. It feels like they train the foundational model, which is very welcome and new. so LG released XR one deep 32 billion parameters, XR one deep exhibits superior capabilities, various reasoning tasks, that's. They all say this, this is literally like the coffee pasta for, everything, but basically it's super cool. It's, they also have small ones. Excellent deep 2. 4 B and seven, 7. 8 billion parameters. We can look at the, let's start to look at the.

That's great for speculative decoding because

they can generate so many talks. If you have a smaller one for speculative decoding, you get a lot of processing speed.

Yes.

And I have Ellum studio to pull up because that's exactly what I did. And for folks who are interested in my full review, blind review with X oh one, you can see it on YouTube. I'll add this to the show notes. I'm not gonna repeat this, but basically, basically I tried to get X oh one deep with speculative decoding. I loaded the X oh one model here. so that, wolf from, you're absolutely right. It's great For folks who don't follow us as closely, speculative recording is a way to use a smaller model of the same. Bigger model to speculate about where the bigger model is going to go next. And if the speculation is correct, then the bigger model doesn't have to generate those tokens, significantly speeding up the inference sometimes up to like, I don't know, like, like almost like 20, 30 percent and even more depending on the temperature. And, this actually, and LM studio supports this shout out the Gill and LM studio. LM studio is absolutely the best way to test out local models. Shout out a lot more friends as well, and obviously GGML, but like Elm Studio visually for folks is the easiest way to download and play with these models. And I've used this model, and I tested it on the Beth Tries Ice Cream challenge or question that we have. And I just want to show you that I gave up. I give, let me, I don't know how to show only the chat. Let me just remove this. So basically this is my question. Beth places, whole ice cubes and frying pan. You guys, those of you who follow the Thursday, I know this question. It's a question made to trick models to start doing math when in fact they don't need to, they just need to reason about some stuff, reasoning models tend to go haywire about this question, try to rethink and overthink, et cetera, because they, they're trained with like. Oh, wait. Oh, whoa. Let me see. Like all these like things in the middle. They trained on chain of thought. This model went deep when they, when they mean excellent deep, it went deep. It started going off the rails. And I don't know if that's an artifact of quantization because some tokens here look weird. Honestly, they look weird. I, I chose the default quantized version. So I, I don't remember exactly which one this was, but I believe like Q4 maybe. Yeah. Q4 cam. And then it didn't see like too much contest. I don't know, but maybe efforts going today, maybe templates, but if not, this model just wouldn't stop. It would just like kept thinking, thinking, thinking, and would go, back and forth, back and forth until such point where the tokens that it thunk did not make sense to me. So while I was doing this, and you can see all of this on YouTube, this is my test for X1. I also fired up, our friends, Quinn QWQ, which is a 30. And that one also went hard, but it solved the problem, literally solved the problem. It took a while. It took like 5, 000 tokens, but it solved the problem. It ended. DeepSeeker 1. Why am I mentioning both these models in comparison to X01? Because when X01, released their stuff, they basically compared themselves to DeepSeek R1, and, let me see, actually, she, yeah, they, they compared themselves to DeepSeek R1 and, and QWQ, and they said they're doing better on multiple things like AIME and AIME 2025. competition math. Now, Could be the results of quantization. This could be absolutely like folks, when you download quantized models, you don't actually get the same performance as the companies that release the evals. But Nissen, I also saw you saying that this could potentially be like a leaderboard chasing a little bit, and that's how it felt to me. If quantization had that much, didn't have that much effect. what do you guys think about these models? Did you have a chance to take a look at it? The vibes on Local Llama anywhere? This one I

didn't really check out yet, but that

could be a prompt format thing. Sometimes the tokenizer is not the right one. I had that with the Another model on a studio, which by the way, I'm a convert after last show, I got into LM studio and I'm using it now too.

Let's go.

I will say everything that I said right now is my personal experience. This model could be dope and this could be a quantization version. So until we get folks who are like fully testing this on GPUs and soon, hopefully. I'll also have access to a bunch of GPUs to test a bunch of stuff. We'll see, we'll see how that unfolds. But until such a point, this model runs on local machine quantized. So everything I said should be taken with a grain of salt. folks from LG shout out to you releasing the model. The, the license I believe is a specific license. by the way, they provided the pre quantized versions by themselves with AWQ and GGUF, so I downloaded like the official ones. But they have like a usage guidelines with reasoning. So it could be just like a token thing. So if your guild from LM studio is listening to us, we should have LM studio update with like the, the, the proper things.

I thought that's a great point because no matter how good

the benchmarks look, you always have to test the model, the way you run it. On your own systems, because it doesn't help if it's in theory perfect, but on your own, that quant you are using, if it's not working, doesn't help you have to test it yourself in the end.

Yep.

And so additionally, the model is licensed under XO1 model license agreement, because it's a huge company. It's a non commercial license. It's NC. So you can play with it, you can research, but you cannot use it in production. I think we are moving on folks. Let's move on. we are moving on because a busy, busy show. Where's my. Where's my notes? Where's my notes? There we go. okay, let's talk about Neumatron. Nissan, we talked about Neumatron multiple times before. At some points, we was like, hey, was that just an announcement or the model dropped? I think last time Jensen was on stage with his, like, leather jacket. I, I believe this was Maybe January, that we talked about this, that he showed up somewhere and Nemotron did not release. Actually, in our green room chat, I asked like, Folks, was there a release that I missed? And no, there was just an announcement. But finally, NVIDIA drops Llama Nemotron. And they drop a nano version, which is 8 billion parameter, a super version, 49 billion parameter, and Ultra is coming soon. 253 billion parameter model that's distilled from 405b. So these Nemotron models are essentially distillations and pruned versions. If I remember correctly, they also talked about pruning in their kind of papers as well. essentially those are like distilled and pruned versions of Lama, I believe 3.3 I I'm, I would assume, yeah.

If you look in the config that do Jason of the model, you notice something.

Remember how they acquired Des CLM? Yep. Which, some of us were upset about because they were, they were doing some pretty cutting edge research as a, as a company earlier on. So now you see is, is that, is, is, is that CLM for casual lm So it was that team that made this one. so this is a, from a different team from NVIDIA, not the first one. And, you can see they stacked a whole bunch of, of tricks. So what I think they did is first they did the distillation to the 70 B I'm just guessing here as to how I would do it. And then, and then they went. to they did the pruning to 49b and then I think they either kept distilling or kept training. I haven't read the actual report. I'm just, that looks like cloud diagrams. It looks nice. yeah. So yeah, that's why this one's probably worth looking at, especially if it's the Desi team that, that made it at, And, and video like this much.

So they have a bunch of details in this blog and shout out to

Chris Alex, you are friend, for authoring and working on this blog as well. shout out Chris. there's a bunch of details here. So nano 8 billion parameters distilled from 3. 1. So Lama 3. 1 super is distilled from Lama 3. 3. So very interesting. super is 49 billion parameters coming from a 70 billion parameter model. and it gets. Very good results as well. and this model also supports test time scaling. So that's super, super cool. These models are reasoners folks. It sounds like we need to make sure to like mention this all the time. but these models are also reasoners. So LLAMA is not a reasoner model. We're waiting for LLAMA4 to drop. And I. I, I'd assume that llama team will not skip the test time compute paradigm. I, I would assume I have no knowledge about this, but, they not only fine tuned and distilled this model, they added reasoning to this model. they distilled it through their NAS architecture, not neural architecture search and knowledge distillation. So they have like a, a paper about this as well. They did supervised fine tuning with 60 billion, 60 billion tokens. And the way to enable this. let me see if I can find this, the way to enable this reasoning is super cool. It's very similar to how, news research did this. when you go to the model in the system prompt, let me find the right thing. yeah, in the system prompt, you just say in the system prompt, you're like detailed thinking on and detailed thinking off this in the system prompt makes the model to reason or not reason. And folks, we've been talking about this for a while. And OpenAI actually confirmed this as well for them. The paradigm is conversions towards models that can think when you need them to, and don't think when you don't need them to. So this is a great example of where things are going. Wolfram, any, any thoughts on, on Oh yeah. On this model,

I, I did not do the benchmark, but I did, wipe check it.

So I can say it's a nitron if, you know the nitron models, they are very different from the normal chat models because they, they feel likes, like a call menu where they give you options and you can choose one, even if you just chat with it, it gives you options. But you could say and stuff like that. that is very different from the normal chat models. And if you use LM studio, there has been a ginger template error in the model, which is in the community notes for the, GGOF version, not this one, because that is the official one, but the, the one other one, they will need to fix it. Otherwise it keeps going and never stopped thinking. This is, what I had.

Oh, so maybe that's what happened to me.

Yeah, it could be that it is the same there,

but let's check that and fix it. Shout out

to Bartowski, Colin Kielty for quantizing, I just want

to shout out Colin for a second. Can we do that? it's incredible that if you go to LM studio, LM studio community and Bartowski, there's always a quantized version for you to run on your local machines. because, folks like Grotowski, other folks for MLX, like Prince Konuma, just doing incredible community work, and we have to celebrate them as well. So shout out to those folks. Let me actually do a little bit of a we really appreciate all of us, because then when we test, it's easier for us to test. And, I don't got time for this. He basically took over from the blog. Yep. Yep. We remember the blog from a year ago that disappeared and then Colin absolutely took over. So shout out. We really appreciate your, your support to the community.

Yeah, there's a lot of back and forth, like direct back

and forth that goes on in the community to get the quants right. So the CEO of, the CEO of LM studio will be there. I've talked to Colin several times. If you use the KL, like the, the large quants, those leave the beginning of the end of the model at eight bit. And that often actually. Corrects some big, big missteps. So I always recommend to use the Q4K L the, the large ones with, the quants from Colin, and I think he's the only one that, that does that, that way, him and me. so yeah, there's, but an insane amount of, of work from, from him. And the other person whom we don't know is Mr. Adenbacher. who's also posting a lot and, yeah, if it wasn't for those two and hugging faces, AWS bill, it would be, we would not be having as much fun with these models.

So the, the, the only one last thing that I want,

I want to go back a little bit. So yeah, absolutely. Shout out to those folks. I want to go back to the, to, to the Nemotron models super quick. it, it fits on a single H 200. And they focused on this and I hear, from a friend, Chris, that, FBA is coming very soon and it says on the model card as well. And once FBA comes, this single model will fit on the 49 B will fit on a single H 100. Which is so dope folks. This is like a big model. This is the still of, of, Lama 3. 3. And this will just like show up and like be able to run on a single H1. So that's absolutely super cool.

Also 66 GPQA on with reasoning on for a model that I'm interested

to actually test GPQA quantized, on this one, but 66 is pretty good and 50 to start off as stock is, is actually pretty good considering even the best models, like the 72 B's from last year only got up to like 45, 46. So considering, yeah, yeah, this is pretty good. So shout out

to NVIDIA, we waited for these models and we are very, very happy

that you guys finally released them. So folks who are interested, there's a blog post that we'll add to the show notes about the technical, how they did this, accuracy, above LLAMA, including, this is a distilled version that gets like higher accuracy than LLAMA. so super, super cool. Shout out to folks. They also, one last thing that I'll say here, they also raised the dataset, which is not common folks, not common. And we need to celebrate this every time this happens. There is LLAMA Nematron post training dataset V1 with. this is only the first five gigs. this is, I don't even see how much, Oh, 4 million rows of data. That's crazy. Wait, no, no, sorry. 50 million rows. 4 million is just the first five gigabytes. so there is this, data set and shout out to them about this whole work. and we really, really appreciate the Nemotron team and the, the Desi LM team. I'm assuming Nistan, I didn't know about this. So thanks for the tidbit. So the desi team that Nvidia acquired for these releases, let's go Nvidia. Let's go, let's go. We're also going to talk about NVIDIA GTC in a second. last thing that I wanted to chat about is this DAPO release from ByteDance. super quick because well, we were missing our friend Yam. Who is back on X, by the way, finally. but he's not with us on the show. He may join at some point. and Yama is great to cover and like, make these like models and, and papers for us. Very approachable. but basically, if you guys remember R one when it was released, one of the cool things about the the release was a paper. In the paper. They discussed all different ways to achieve that supremacy with R one. And one of the ways was this formula for. for reinforcement learning called G, GRPO, group rating, policy optimization, something like this. I don't exactly remember, GRPO, we should go and check. and then as far as my timeline is concerned, as far as the vibes are concerned, ByDance, yes, TikTok ByDance, cause they do incredible stuff in, in, we know, we talked about ByDance models before. They released a new, open source reinforcement learning system called DAPO and outperforms, previous state of the art on AIME. With 50 percent accuracy using 50 percent less training steps, 50 percent less training steps, which is, which is crazy. the folks from the open source Chinese community just keep giving us more and more. And I'm pretty sure that like the major labs in like, open the eye and topic, et cetera, they have a bunch of like optimization techniques in there for RL. I'm pretty sure that that's what's happening, but they're not releasing any. So the only ones that we get is coming from the open source, any stuff. Um, Also one release 50 percent in my AIME 2024, they trained quen 2. 5, 32 billion parameters to surpass DeepSeq R1 with 50 percent fewer steps of training. I think it's super cool. so there's a bunch of reasons, dynamic sampling and token level loss, whatever, if that's of interest to you. Make sure that don't miss the APO, because a lot of people are now doing training with RL. if any of this did not make sense to folks in the audience, because we, I know for a fact we have some machine learning engineers like crazy folks doing crazy RL. And so if this did not make sense to y'all, please correct me and add some more details or come to the show to talk about this. so this is a big invite to, to, to our folks.

No, this

is

pretty, I, I can't actually go in, go in a little bit.

So it looks like the, the by dance method is still pretty close to a group relative policy optimization. And, and yeah, and they just added, they just added some more stuff, but it's still roughly the same thing. And for people in the audience, this is. You do reinforcement learning where if you have some kind of data set where you know what the answer is in the end. So it could be a math problem. it could be like a medical problem, which is what I'm doing right now with Dr. Johnson Thomas. And I was also able to show like on a 3d model, it greatly improves performance there. And, if you know what the right answer is going to be, you end up generating eight different samples. And then you, instead of only reinforcing. What the model says on the right answer, you do some fancy trickery. I need to wrap it up, but you use all of the answers and then you do like a group relative like distribution of tokens. There's some stuff in there in the code that even I don't fully understand and then, and then you train that back. Into the model versus doing only rewarding the model on the right answer. And what this does is the steers, the model to just think more and, and, and ramble more and because. You're not necessarily rewarding on it, just getting it like a lottery system of it. Just getting it. Perfect. Randomly, you're actually rewarding on it, exploring that blatant space. So the model actually exploring many different ideas and the way this training works is you tell it to just do put some reasoning tags or put some thinking tags and then come up with the answer. And then you just check if the answer is right in the end. So you can only do it for stuff where you do know the answer. So it could be code math problems. and medical answers, actually, which is, it's pretty interesting. And it resulted in drastic improvements in the end. And this is how DeepSeq R1 was, was, was trained. And yeah, this is like playing around, playing around that, that idea as, as well. but I have to say that I don't see the code for the ByteDance one yet. I see the paper right here and there's a rebel

paper is here, but yeah, there's no,

there's the code's not out where is the, the deep sea

code that that was allowed people.

Yeah.

So I'm pretty sure that the GitHub is here. They're planning to release it. they also have data sets here, and model weights coming soon as well. So shout out to Biden's folks for this work and, Tsinghua university as well. Right folks, we're moving on to big companies because we have a lot to discuss here and let's move on. Anything else in open source that we missed, please give us in the comments links. We'll definitely cover like the most important stuff. There may have been three releases of models just as we talked about these ones, right? please, please help us stay up to date folks while you listen to the show, scroll Twitter, and there's always up to date, news. by the way, here is a good opportunity to shout out a Great friend of the pod, a community member. That's a very, very much updating everyone else as well. Lincoln, president Lynn on X, give him a, give him a follow for sure. I'll add his things to the show notes. He always has the releases as well. I also have the releases. I don't know. I want to show you guys my, my new thing. actually didn't just, just added this. I found a telegram bot that is called Jenny. I monitor. And what it does is, it just gives me GitHub releases and Hackenfest releases on anything that happens and all big studios as well. It's so cool. So for example, I can see that today OpenR1 was released from Olympic Coder, blah, blah, blah. PyTorch released something. There's like, this GNI monitor thing is super cool. Not a lot of people. I think they did a great job connecting multiple things. this is how my new thing, to, to help follow the latest news as well. So I'm going to be checking this for you. You don't have to go and check this unless you want to. It's, it's a lot of notifications. I added this yesterday. I already have 136 notifications from yesterday. So you will still need to filter out the bubble. However, the cool thing about this, this sometimes shows you the hug and face space before they like. this, the first thing to open the space, then they need to upload the files. They need to make sure it runs. Then they ask for zero GPUs. So you can actually know about some stuff before they release and it will Twitter thread about this. So it was pretty cool. That's what we want. just skimming through this. Nothing is very interesting there. Announcements from GDC, blah, blah, blah, blah. Okay. Back to the show folks. Big companies, APIs for which we do not have a transition, but we have like a sound, I think. No, not this one. We have a sound for this.

All right. In the big companies, APIs and LLMs, the place where we talk about, Multiple approaches of how to build your RLMs without open models. we have not much news this week, some news, but not much news. Specifically, we have Google, who made deep research for free. Folks, you now can go and, we, we actually looked at it. if you go to gemini. google. com or the Gemini app, you can now do a dropdown and go to deep research and you don't have to pay for Gemini Pro to do this. And this is Pretty damn cool. Not only that, we talked about the deep research before. the UI is a little bit different as well. So you can like actually see sources. This is pretty cool. What is the latest episode of Thursday? I. Let's say like this, I'll just test it live super quick. this is not the AI mode that we've talked about with Robbie Stein from Google's AI search, right? So that's, this is a significantly deeper search. it shows you the plan. It lets you edit the plan as well.

Yeah.

If it has a lot of YouTube sources, I think it actually watches the videos and it takes a long time because of that. I didn't, that, that's what's slowing it down too. So I was gonna, I was gonna make a joke that hopefully it's done by the show. Okay. So we'll start with

this and hopefully you'll watch the two hour videos.

While we wait for this, I want to say a huge welcome back to our dear friend of the show. Yam Peleg. What's up, man? What? And he's gone. And he's gone. And he's back. Short reunion. Let's do this again.

Okay.

It's going to work this time. It's going to be back in a year. Yeah.

Let's go.

Yeah. Welcome back. Welcome back, man. How are you?

Fine.

Fine. Thank you. Thank you. Welcome back. A lot, a lot. So much went on.

Yeah.

Folks who tune in and are new, yam is a dear, friend of the pod and the co host for a while. And, we're very, very happy that you're back with us to cover the latest and greatest news. welcome back. we will. Not bug you about why the fuck did you disappear? I will say only the one thing, many people in the community directly reached out to me and said, Hey, is he okay? I was like, I'm, I'm hoping. So send us, send us an update. People love you and wanna know that you are safe and okay, and everything's fine. so we'll . Awesome. So we will move. Oh, so look, the deep search did like, get a lot of. sources from Thursday news. It's free, on that topic as well. Let me open like another thing. Grok launched deeper research, which I think is just like really funny. if you go to grok. com, I think, if it's on X already or not. Come on, come on, come on. no, I'm going to open this in here. Once again, if you go to grok. com. And now there's a drop deep research. There used to be a drop down. I don't know, but there used to be a drop down and now there's a deeper research. It does even more search for you. Grox one. Did they release it for free folks? I think it is. It's just, yeah, it's just

different, different rate limits if you're on premium

plus, but you still get it. Yeah.

whoops.

yeah, so definitely, we are in the realm of like, if you need to do very much, like a lot of search, there's now no excuse to not do deep research from Google from Gemini is now available for everybody for free. let's move to. the next thing OpenAI, not the news going to come very soon. So in 30 minutes, we're going to tune into the OpenAI live stream to talk about their breaking news, but OpenAI also had something of a, of a release yesterday and they have made O1 Pro available in the API and. For a ridiculous price. So for folks who are not following as close as we follow, OpenAI basically has, released O1 as a reasoning model in the APIs, but in the, for the pro tier, for the 200 a month tier that many people, because of that, jumped on it. They had a specific version of O1 called O1 Pro we got confirmations from the O1 team, just publicly, not like directly to us, but publicly from Michelle Porcras and some other folks. The O1 Pro is not just a different model. It's not only thinking for more. There's like some system in there. There's like a bunch of stuff. So we're not sure what O1 Pro actually is, but we, we do know that it costs them a lot of money because they gave us. The access to it now, and it costs us a lot of money, like a ridiculous amount folks. so we're talking about 150 for a 1 million input tokens and 600, which I think could be the highest API price that I've, that I've seen in the API world for 1 million output tokens, 600 for 1 million output tokens. if you pay for the 200 bucks a month tier, I don't think they limit you over there. It's only if you really, really, really want to build this into your tool and use it for the best one imaginable now. It's also to be said, I think the O1 Pro is still one of the best performing models generally. And, nobody has been able to eval this yet because they didn't have it over in an API. So what this opens up is somebody with very deep pockets to make sure and actually start running evaluations. I'm not gonna do it. Although, I'm not going to do it. I'm not going to spend ways and buys them. I definitely, I'm not going to spend my own money to run evals of like, comparisons, but this model is supposedly probably the best powered model right now that you can access via API, but maybe with a little bit of intelligence with Tupper or one, maybe there could be like 10 percent of difference. And for this 10%, you're going to pay 600 bucks, for one million tokens. DeepSeek is like. What's DeepSeek at latest? But I think it's like very, like, it's, it's, it's ridiculously cheap. It don't, it don't make sense. And you get a reasoning

crisis with DeepSeek.

Don't think you get them for this price. You don't.

Let's debug this.

What do we have confirmed? Just, just to, to make, to make it clear, what do we have confirmed We know it's necessarily not a different model or we don't know that it's a, it's a, it's an interesting detail.

They claimed specifically, I can go and find the suite.

I think I mentioned it in Thursday at one point, they confirmed that it's not only, so this is all one. Just to be clear, but it's not only a one with a lot of thinking. so all three, when it was released to remember our KGI price, they said like all three was a lot of thinking beats this like 87 percent or something. they confirmed that this is not only that, so this is not only more GPU and more like power. So supposedly there's some sort of. I don't know, maybe they do like search and then come back.

You can be best of N, which significantly is gonna, is

gonna cost them more and obviously gonna get you far better result. Now, the interesting thing is Whether or not it's the other one is the same model, or it is also, a more powerful model, which is different. It's a really interesting detail, but yeah, we don't know. But,

we can, we can from the price that this could be like,

like a breast search or something. We also saw some research from Google this week that I should probably add to the show notes as well, about, specifically. search getting better results and like one potential more scaling law from search. we should, we should, we should add this to show notes, but we don't have time to chat about this. Unfortunately.

Just one thing about the pricing.

You, you have to talk about where you can go to the batch API and then it costs only, only half the you saw. When you are patching it, but it does not do a prompt caching. There's no cached input price. So yeah, that's that.

all right, folks.

So we're moving on. we'll do a brief GTC recap. GTC is, NVIDIA's flagship. convention, meeting, I don't, I don't know what to call this Woodstock of AI. Jensen is now basically Taylor Swift. I don't know what to call this, but I know that like Jensen's absolutely going live on stage and saying, Hey, I'm here without the safety net. And I have only the slides. I don't have any teleprompter. I'm just going to go for two hours and goes like insane detail deep. Like, it's absolutely incredible to see how deep Jensen can go, and it's, it's been incredible. So the, the, the recap for GTC, this is a brief recap that I, that I took from our friend, Rowan Cheng from, from, Rundown AI. And, so we're just going to thank Rowan for, For the recap, they announced like a bunch of stuff. not a lot of ai. So we talked about the nitron stuff as well. they, they announced this like cool robot called blue, star Wars inspired robot. And they have like a new physics engine with intelligence there. the robot looks super cool folks. it really like is endearing. We also have, some announcements for like, Newton, which is their physics engine, open source, I keep seeing this physics engine things and I keep getting frustrated about, they're not like measuring up to the level of, I want to type something and it generates like a physics scene. I think we've seen like one physics engine like this. You guys remember we got all excited and then it ended up being nothing. and nobody was able to actually do the things that they claimed in the video. So I'm, I, I, I'll keep being skeptical, until such a time where I can just like type stuff. And with that physics engine, things will happen. However I will say, we don't, we're not going to cover this too much, but, there's a new thing blowing up Blender MCP for some reason, out of the all MCP tools, Blender MCP became like the coolest thing around. And Blender is this like 3d software. It's very difficult for folks, Blender MCP lets your AI basically know how to do stuff in Blender and tons of people just like getting blown away by this. It's just one example of, tools, basically. MCP is a protocol, but basically tools. Um, So what else in GTC? Blackwell Ultra, a new generation of Blackwell with 1. 5X computational power. Nistan, you want to comment on this super quick? Or Yam, you want to comment on this? Have you guys looked at this? Second half of 2025, Blackwell Ultra. We, we don't know a lot yet. yeah. 1. 5 computational power. they also showed a future GPU roadmap after Ultra, Vera Rubin. And this is the next. So after, Blackwell, we have Vera Rubin and then Feynman in 2028. So I love the naming schemes. Absolutely. And we'll see, like everyone is going to jump some level and Jensen's going to show some graph that like, this is like two X better than before. Just a reminder that we are on multiple exponential curves. Jensen is showing us the hardware exponential curve. OpenAI often shows us the algorithmic exponential curve in. Scaling laws with 4. 5. We talked about and in test time compute, which is another scaling law that we're writing folks We're writing multiple exponential curves. Some folks corrected me and said that these test time compute is not a exponential scaling law It's a log level scaling. I don't know. I did not take any statistics They may be right, but I know that shit is moving fast. That's what I know basically Oh, they also released Groot. Groot is a customizable foundation model for training robots. So shout out Dr. Jim Fan, who leads kind of the, the NVIDIA robotics team. And Groot looks super, super cool. They released the model as well. Nissan, I think you sent the link to this as well. They released like a robotics model. And a super cool tiny computer. We talked about another small one, but this one is even smaller. DGX Spark. DGX Spark is a, a tiny supercomputer that Jensen showed. It has a thousand. Tops of AI performance with 128 gigabytes of Ram and grace Blackwell architecture. So a bunch of announcements, and DJ station as well. So it's always, there's always a picture of Jensen holding something cool. And only for like, for people who don't follow, this looks the same, but I will say the, the. Hope not to offend anybody with this metaphor. It's like, folks who don't follow soccer, or football, and for them every match looks the same, they're pretty much doing the same thing. But for folks who follow the actual thing, they get like so excited about what's going to happen on the field. For folks who follow Nvidia stuff, and obviously for investors as well, Every little thing that Jensen shows is like 2x better. Less than a year after. So it's, it's crazy. So we're basically going to pause and there's a bunch of partnerships as well. So that's anyone else

feel that, Nvidia has taken the spark of Apple and

he's like a new Steve Jobs in a way I've seen the comparison and I thought, Hmm, that's interesting.

I will say they're doing innovation.

And meanwhile, I had the tweet today. I got so pissed at Apple. we didn't cover this last week, but basically there were, there's some stuff about Apple's, intelligence. Apple intelligence that we got all excited about during last summer that that's not gonna come all of what they announced They had fucking ads for this thing ads of like, hey Siri do this thing blah blah blah That's not gonna come until like 2027 folks. HDI could be here in 2027. What are you talking about Apple? I got so weirded out by this news. Not only me Wolfram, thanks for prompting, because I didn't really want to cover this. Um, Daring Firewall, which covers Apple for the last 20 years, John Gruber, he basically said something's rotten within Apple, and that's been, like, absolutely crazy, because, hey, he's been their fan service f for the last 20 years. If he says something, then absolutely something is wrong. They're planning the releases for, like, two years out. Do you know how much is going to change in those two years? Like, I'm not sure if I'm going to be here fully. physical or like have digital by then. And they're like, Oh yeah, we're only going to release like the smart Siri. Meanwhile, Siri right now is so dumb that I asked it to precondition my Tesla. And I said, turn preconditioning on and said, which one? And show me two options. Turn on and turn off. Like it new to transcription. It literally says turn on preconditioning. And it asked me if I wanted to turn on or off. It's 2025 folks. We've passed first quarter. We're talking about models running on my device that beat like GPQA and Diamond and go like AI me and solve the competition. And my dumb ass Siri doesn't know if I want to turn something on when I asked her to turn something on. That's the state of AI Apple intelligence we're in. It's absolutely abysmal. Abysmal. And hopefully folks are going to wake up. I

don't understand.

They can be a GPT wrapper and I'm like getting done seriously. Or they

could just add whisper.

I have to actually take my recordings voice recordings on Apple and then convert them to wave files and then just run whisper because the Siri transcription is, is, Usually okay. And quite often garbage, so we'll just get it wrong. And, yeah,

transcription got a little better, just a tiny bit better.

but not enough that Siri actually does anything with it. All right. give it two more years, at least give it until the singularity. And, and then, they're going to

sell more hearing aids in those two years that all of the AI

company profits combined, right? That's just. That's true.

Alrighty.

we now move on to, we should move on to this week's buzz, but I'm going to wait for Morgan to join us. so we're going to talk about this week, but then, I want to start covering the 3d stuff before, we have, around 30 minutes until the opening a live stream that we're going to cover, and we also have Joseph Nelson, a friend of the pod that's going to join us. I think Joseph is already in the space. And Joseph, if you're listening to us, join us also in the StreamYard link that I DM'd you. So we can actually see you and see like what you guys are about to announce. We, we're about to have breaking news. Before this though, I really want to cover this folks. It's, it's absolutely incredible, to see how fast this field also develops. and then let me show you what we're talking about. So, entire screen and this one. Oh, actually, yeah, we have Joseph here, so we're gonna do 3D afterwards. let's wait until he connects, and then we'll do an introduction, a brief chat with a friend of ours. basically, while Joseph connects, we'll add him to the stage, and hopefully, you'll set up a microphone, and we can see you. If you guys remember multiple times, when anything happens in the world of VLMs or in the world of vision models as well, we get, we get our friend Peter Skalsky that hasn't been on the part for a while, but Peter is our friend and a vision expert that we, we, we, we invite to tell us all about the cool new segmentation models and different things. Peter works at a company called Roboflow. Joseph is a co founder of Roboflow that, hopefully is with us. Joseph, are you here? I am. Hey, man. What's up? are you able to, Oh, there he is. Let's go. Welcome, man. so this is your first time on the pod, I believe, but you're probably listening, but the company was represented on the pod as well, due to, no, you've been there before. We talked. I somehow believe we talked.

Yeah, actually, I think when, segment anything too came out,

I think the last time we caught up.

Yes.

So you've been here. my apologies. but, for those. since segment anything two came out, multiple things that happened and many more folks joined Thursday. I saw, how about you start with maybe just an introduction of who you are and also, what Roboflow is. I'm

Joseph.

I am the CEO at Roboflow. Roboflow, we build tools, open source tools, a platform, community for developers and enterprise to create and deploy computer vision models to production. So on the, open source side. We maintain projects like supervision, inference, notebooks, and a community called Roboflow Universe, where folks create and share open source data sets and models. We have a hosted platform for creating and deploying those models, and we have an inference service for both doing cloud inference and local on device, inference. And, about a million developers build, build with our tools. we've been at it for about four years. So I like to say like, maybe like weights and biases, like maybe new GPT that this AI thing might be, might be, might be big. of course, the acceleration of the speed of computer vision. And, capabilities across, the AI field have only been accelerating. And so one of the things that we've been eager to do is whenever there's new models that come out, we want to make it really easy for folks to. See how those models do on their visual data and put them into production. today we're actually, maybe have something to share and announce on that front, but I can let you, guide the conversation. Have you see?

so I just want to, before we dive into the stuff, the

reason why you're here today is not. Just like pump Roboflow, which is well deserved and we love Roboflow. supervision is incredible. The fact that you guys are working and opening like in open source and doing all kinds of things. shout out to Piotr Skalski also like doing incredible education for the community of how to actually use these models. Because a lot of times segment anything 2 comes out or like another model and that only does like one specific thing. and folks don't actually know how to like utilize this. So not only do you have supervision maintained, which is incredible, like 26, 000 stars. I'm going to add another one. I don't know why I wasn't started yet. but, supervision is fully open source. So you guys like dedicated to like actually provide those tools. And have an inference engine of your own and a bunch of stuff that you provide 1. 7 million downloads a month. This is insane. Huge congrats on the success and the work that you guys are doing in, in supervision, but also, this week you came here because, you, you showed me, like a sneak peek, on your ex account with the weights and biases, log that looks like insane to me. And you also said that you guys have something cool to announce. So I would love to hear, what you guys have and talk about it.

we're introducing two, two things today.

the first is, a new state of the art, real time object detection model. it's actually something that we've wanted to do since RoboFlow started is enable the field to move forward in its research and development capabilities with new model capabilities. And we've always been a place where there's new models that come out. We want to make it easier for folks to understand what they can do, where they're good, maybe where they can be better. How did you on your data create and deploy them? We've actually never released a model that we've developed. And so this is the first time where we've taken a lot of those learnings and put that into a model. And, it does pretty well. I'll tell you more about it in a second. Alongside that model release, we introduced a new eval, new benchmark for benchmarking VLMs and object detection models alike. And these two releases are actually batched together for a very deliberate reason. When we went to benchmark our new model, we realized and have seen over the span of time that object detection, evaluation, is a little bit broken. Like the benchmark that everyone uses is, the cocoa benchmark, which was introduced in 2014. It's common objects in context, about 200, 000 images. from the Microsoft team hasn't been updated since 2017, but models have continued to get really, really, really good. And so they're saturated the cocoa benchmark. And so that coincides with, at the same time, like as Robo flow has grown and as people create and deploy projects and open source projects, we've seen there's actually a pretty big diversion between the way people are using computer vision and the way that we're measuring model performance. And by that, people are using computer vision for stuff like understanding infrared images. for wanting to do aerial image processing from drones, maybe medical images, maybe documents, maybe digital images like understanding video games. None of these things are presented in cocoa. And so as data sets continue to, as models continue to improve, they might be optimizing for a subset of the real world. And so we, introduced a benchmark that we call RF 100 vision language. And I'll tell you about, maybe some of the zero shot performance that might surprise you from like some of the leading models and how far we have to go as a research community. as well as maybe some of the details on, RF data, the, the model, but those are the two headline releases.

So let's cover them.

Headline releases. RoboFlow finally releases your own model. we'd love to hear a few more details about this, but also an eval benchmark, which is well, like, needed, like we know if, if there's one thing that I can always say, and that could be the reason because I work with voice and biases on the evals platform. There's always a need for new and better evals always and so it's great to see those couple as well And so you're releasing together this model and the new eval to show that The previous benchmarks is like outdated and I agree with you Coco has been around for a while And I think what you mentioned is resonates is because that's the eval, many folks optimize maybe to like test their models on this eval and this leads to very specific optimizations. and that maybe doesn't generalize well. So what can you tell us about the model? remind us the name again and, where can folks find it and yeah, give us some details.

Yeah.

the model is called RFDeter. So it's from the detection transformer family of models. So back in 2023, there was a model that was introduced called RTDeter, real time detection transformer. And, RF data is an homage to that, but with Roboflow pre appended, it's a fully open source Apache to commercially permissive license to take, use, deploy, fine tune. it is the first real time object detection model to exceed 60 mean average performance on Cocoa, and it performs better than all other current object detection models on what we call real world data sets, and that's a part of the eval is a transformer based architecture, as the name implies of being a real time, model, meaning it can run on the edge and papers with code defines it as 25 plus frames per second when it processes things. We're releasing two sizes, a base model size and a large model size. and we benchmark and compare its performance to leading state of the art, transformer models, as well as the YOLO family of models, which are particularly popular for doing, real time, types of work and problems. As you mentioned, we've long supported open source models and actually, we hope that this actually increases the number of, of more models that people release. in a sense, like the introduction of our own models actually meant to make it so that hopefully other folks beat it pretty fast and go in this direction of realizing some of the gains and improvements we've done. The core thing that we've realized that allowed us to produce leading performance is, this observation that transformers what they're incredibly good at is maintaining context in per training. So if you do a large scale pre train and then you apply that to some subset of a domain problem, then the model can do pretty well now the, yeah, yeah, great. We're going to screen share so you can see. So the, the novelty with RF data is we use a large scale pre train and a grounding dyno V2 backbone, and we have various sizes. and what we've observed is that by doing that pre train and then applying to a smaller domain problem, it actually maintains a lot of that context to do better on even the in domain problem contexts. And so we think that basically what we're aiming to show the field is move in the direction of take all the benefits that we know and love about transformers and continue to enable them to be used at real time speeds. And on edge devices at, rates that might previously, not have been, explored as well. So the model is the best for real world data.

yeah.

when you say in domain and folks were not following the vision fields and like the, the, the, the field generally, what, what do you mean? Can you give us an example of like, what, what does like out of domain in domain means? Yeah.

Yeah.

So this kind of gets back to the idea of like the, the eval concept. So maybe I can introduce this, this concept alongside, in domain, out of domain. So generally, a lot of vision models are trained on the, the, or evaluated on the cocoa data set. Now, a lot of vision problems are pretty diverse, pretty different. Um, types of problems people might be using vision for, as we were mentioning, might be like something that's out in the real world, which could be maybe, a document you're processing a lab image, something from underneath the microscope. maybe it's aerial imagery, satellite imagery. maybe it's industrial images could be infrared data. These things are, often out of domain and out of domain distribution and understanding the visual world is, pretty complicated and pretty complex, which usually means fine tuning or at least distillation or domain adaptation is key. The key thing with RF debtor is it is the best model for working on in domain and real world contexts. So out of the box, it's already pretty good on on cocoa and super fast, and it guarantees for all models that are available presently to adapt best to in domain problem types. you can think about it like robo flow in, the four years of folks building and creating open source projects, they've exposed all these types of problems that they're working on. And for a bit we were kind of like, are we crazy? Like these problems aren't just like people and vehicles and things you've seen cocoa like we're seeing all this other stuff like what's going on? And with four years of time and 500 million user labeled and shared open source images, we're like, no, actually, this is just the way people are using computer vision and models are not working in the problems and context that developers need them to. So what can we do to move the field in the direction that allows the types of problems that people are actually using computer vision for to be solved?

That's incredible.

So first of all, congrats on the release. this looks, amazing. the, the graph that you showed from rest of the assets also goes, goes crazy. Nistan, I think Nistan wanted to add to the stage and like maybe jump on and ask you also a question. And then also I, I do want to talk about the eval and how we can find it. Like what, what, what's next for the eval, benchmark as well.

Yeah, I don't.

All the notebooks or demos I've ever made the coolest ones that people like were always some kind of combination of using YOLO and drawing like 60 frames per second squares on screen. So people love the the visual aspect of it can you tell us more because you mentioned edge devices and these are becoming a lot more A lot more usable there. Can you tell us what kind of performance you went into it a little bit What kind of performance do we Expect to see. And if we're getting this right, this will only draw the boxes so far and doesn't really need as much pre training because it seems like it's pretty good out of, out of the box itself. So yeah, I can't tell us a bit more, what kind of performance, what kind of video performance in particular people can expect when there's devices? I see there's 6ms latency and if you can go a bit more into that, that'd be great.

Yeah.

So as you said, a lot of vision problems. they demo really well because you can, like, see what's going on. And that also means usually you're running, near where the problem you're solving is. Like, for example, we have some broadcasters and they want to co locate their compute next to the broadcast event because they don't want to stream it to the cloud, do inference, and then put it live over the wire for a broadcast. Or maybe you're like in a place where there's not great internet, so you don't even have the luxury of streaming if you wanted it. So your, your compute constraint, which means running on the edge is critical for a lot of vision problems. So a key thing about our debtor is supporting edge vision capabilities, the types of performance that folks can expect on the edge. you were sharing the, the screen share of the latency to expect on the model sizes for RF debtor base. The, Latency that we benchmarked is right around six milliseconds end to end on a T4 using TRT, which is what a lot of the benchmarks, will suggest on an edge device. What does that translate to? It means that you'd be able to. So edge devices commonly. Our, NVIDIA Jetson, Intel Nook. there's Qualcomm devices, on an edge device depends on the size of Jetson, but there's like a Jetson Nano to Jetson Xavier. You can expect anywhere from, 15 to, if you get the largest orange sizes, 60 plus, frames per second in real time performance. and remember, papers with code code defines real time. It's 25 FPS. The human eyes is at about 30 FPS. So there's quite a bit of overhead available for other problems and processes someone would want to run on the edge device packaging things up and running them on the edge is why we made the open source inference server that we released because it can be a pain to manage and get things going on the edge and optimize. So that's a key part of, Okay.

Thank you.

And a very brief, other question as, as a company that works in, in open source, what kind of advice do you have for, for other companies or how can other companies reach you that, that, that wants some, some custom work in, in this, in this, in this regard?

I think the key thing about open source, is

building with and for the community. So this model represents that effort where we're seeing folks wanting to run on the edge. We're seeing examples where people want to use transformer based architectures and we're seeing this need that's being on met. And so in fact, like even this release, there's like, we already know there's like already so much more to do. People will probably want more model sizes. They probably want better benchmarks. They'll probably want new task types. They'll probably want, maybe other data sets that it can run on and and so forth. And, we could have said, Hey, what are all those things that we know someone's gonna want to do and just do that out of the box?

Or,

in fact, we could allow a first release and then solicit feedback

from community and continue to update and improve and support the readme. So I think that's a key part of building open source and actually a key benefit is people use your stuff. They break your stuff. They have feature requests for your stuff. and, that's, that's key. And the second thing is like, so much of AI is yet to be fully understood of how, like all the places that will be useful and impactful. And so open source also allows leaning into that advantage. In terms of reaching us, so reaching us about this project, there's the GitHub repository that we just published, and, of course, you can open issues or PRs for things that you like. For reaching me personally, I'm on Twitter, x, at, at Joseph of Iowa, and the company is, handle is at Roboflow. you can find my email pretty easily based on that information, too. and we're, of course, like a hugely collaborative, company for especially those that do an open source AI problem. So happy to be supportive of any ways of integrations or capabilities or deployment or data collaborations. we'd love to be a part of supporting your all's efforts.

That's so awesome.

Joseph, thank you so much for coming up. I know I've been mindful of your time. I just want to, make sure that we don't miss the very visual aspect of what you guys do. Nisten talked about this, but just going to your website, just seeing like real time things move and all of the people in there just like segmented correctly and straight. This chocolate example is like. I looked at this like, I don't know, like in loop, like multiple, multiple times. So there's that addition that what you guys do is very visual, but also open source is very important. Thank you so much for working on this. I want to leave the stage to you before, before you go for like, whatever shout outs you want to do and whatever, like the, the, the place is yours. If you're hiring, you can tell people and reach the people through there as well. feel free to, to pump your own stuff.

Yeah.

each of the things you mentioned are true. We try to make tools that people want to use. So please give a try and let us know what should be better about them. And we are rapidly growing. engineering, machine learning, go to market are all things that we're actively aiming to, bring into the team at the moment. Maybe I'll leave you with a last minute trivia question, Alex. So we introduced this benchmark RF 100 VL and everyone's like all the models are, infinitely great and, work amazingly out of the box. Yeah. My quiz question for you is if you had to guess the zero shot performance of a current state of the art model, you could take your choice. Maybe QuimVL 2. 5 or maybe Gemini. Let's let's do QuimVL because it's open source and people like using it. If you had to guess the zero shot performance of the world's best open source, multimodal Model on a benchmark like real world stuff like RF 100, vl, what percentage mean average precision do you think when VL 72 B gets on RF 100? Vl. Ooh,

that's, that's a tough one.

it's a great model, right? So it's, it's gotta be, it's a great model. Real world is very complex as we know from robotics and everything, so I would say. I don't know. I don't know what are

roughly the other, the other numbers.

Because

that's part of what I'm making it.

I'm making it really hard. I'm not going to make any contact. Let's

go for 12%.

I'm going to get 12 percent from Alex

and I'll add the show back.

Come on folks.

38, 38.

Any other guesses?

42, of course.

42, that's a good, that's a good guess.

You have any idea, throw us a number.

I don't know, I, I think it's true question.

I think there is a good guess.

Yeah, it's a bit of a trick question.

The zero shot performance of QwenVL 2.5 vision language. On real world data, 5.8 5.8%.O we as a field have a long, long way to go before we have zero shot performance on real world context. Just

a question, like what's the, what's the target space?

Like, how do you, like, you measure this in terms of like what targets?

This is, an object detection task and you provide

instructions, almost providing annotator instructions of saying like, Hey, like on this video stream, label all of the people and then based on the box grounding capability.

Yeah, you get the targets beforehand like and it needs to

be like, like to do to detect.

yeah.

So the held out data and I just put the link in the chat to the preprint of the paper, which the table that we're referencing is on page seven. So the, the task is you give the Quin VL an image and you hold out the. ground truth, bounding boxes and names for those bounding boxes. and then you provide annotator instructions and you compare if you give a human the same annotator instructions versus giving the model those annotator instructions. How well does the model do as measured by mean average precision?

Open ended and it's completely open ended.

Okay. That's very hard. Yeah. Yeah. okay. That's a hard problem. Yeah. Yeah.

Yeah.

I want to let me, let me interrupt because we're going live to the next segment. Joseph, I want to thank you so much. One of the best things that we get to do on Thursday is to get the folks that announced the stuff to come and talk about the stuff themselves. I want to just shout out some commenters. This is absolutely not a sponsor segment. Thursday, I does not do sponsors. The only sponsors weights and biases, and we just, Love the folks that we work with, our friends. And this is the reason for, for the segment. I think that this is like state of the art models and I just wanna shout out Robo Flow for all the open source that you guys do. And Pure is a dear friend of the pod that came and helped us through like a bunch of visions. You're obviously a dear friend of the pod, so we really love the work you do and congrats on the state of the art model and the eval. That's not saturated folks best in the world. Clem vl. Like, we always talk about when we all, June Yang from when he's here. And like, we talk about this being like the best vision model 5%, it means this is an amazing benchmark to he'll climb on and the models will just get better. So huge thank you to you and for the work that your team did for this release. And we appreciate your time and, come back to us with new releases as well. We hope to see this model get, get significantly more improved. The, the one thing that we barely mentioned is the sizes. The, these models, like tiny folks, 29 million parameters for base and 128 million parameters for large. Hi folks, we're going over to, open the eyes livestream. Joseph, thank you so much for joining us. Thanks for having me.

Let's go.

Voice agents. today, we're going to enable developers and businesses to build voice agents. Agents which are reliable, accurate, and flexible. And so we're going to announce a bunch of new models and tools for that. Today, we're releasing three new models and a bunch of new tools and capabilities designed to make it really easy for developers to build rich, human like voice experiences. We have two new state of the art speech to text models that outperform our previous model, Whisper, on literally every language that we've tested. We have a new text to speech model that for the first time lets developers control not just what the model says, but how it says it. And then we have a big update to our agents SDK to make it really easy to turn text based agents into voice agents. So let's get into it. A few things to cover. We'll start with speech to text where we have two new models, GPT 40 Transcribe and GPT 40 Mini Transcribe. Yi, you've been working on these models. I'd love to hear how they were built and how they perform. compared to our last generation models, Whisper and Whisper 3, our new generation model is based on our large speech model. This means this new model has been trained on trillions of audio tokens. It also ingests our latest technologies and also architecture of our models. We also distilled the larger model down to a much smaller size one, which is a GPT 40 Mini Transcribe. Nice. So what I'm seeing here is that the new model is state of the art across many languages, English, Spanish, more. and then actually 4. 0 Mini is state of the art, if not for its bigger brother model as well. Yeah. If you are actually looking for a very high accuracy transcription model, our model definitely is the best choice for you. Great. So that's GPT 4. 0 Transcribe 0 Mini Transcribe. 4. 0 is API today for just 0. 6 cents per minute, same price as Whisper, and 4. 0 Mini Transcribe is 0. 3 cents, so half price. We're also enhancing our speech to text APIs with streaming, Next capability is a new text to speech model, GPT 40mini TTS. Yaroslav, we'd love for you to show us how this one works. Yeah, let me pull this up. this is OpenAI. fm. it's a website, we built just to make it easy to play with this new model. so as you can see, there are a bunch of voices that you can choose from. there are different prompts that we pre generated, but you can also type in your own. this is basically a new field that we added. It's instructions field that tells the model how you want it to speak the text. so yeah, let's maybe try, try some, Mad scientist, please. Exactly, yeah. Ahaha! The stars tremble before my genius! The rift is open! The energy surging! Unstable? Perhaps. Dangerous? Most certainly! And just to be clear, the tone, the personality is not tuned into the model. It's just prompted. Exactly, yes. You can just prompt it, and you can be as specific as you want. We thought the last thing to show would be how this all comes together. And to do that, we're releasing an update to our Agents SDK. Yaroslav, I'd love for you to show us the code changes involved to make that conversion. Now, let me open the backend server code. So it's a very simple, uh, WebSocket, uh, backend. Awesome. Alright, so that was like nine lines of code. Yes. To build the voice agent and now it's a voice agent miracle.

All right, folks, I'm going to pause this here because, I

want to introduce a friend of the pod. Quinla co founder of daily, a friend of the pod. I consider you a friend now as well, and, also our absolutely most world renowned expert in voice AI, which I am incredibly thankful for you being here, because we need to cover what they just announced. It's great that they're showing like the additional stuff, but the main stuff that that we need to talk about, they just announced with breaking news. So just before you begin, I'm going to hit the breaking news button because we just did like the live stream of OpenAI's new voice, text to speech and speech to text. AI breaking news coming at you only on Thursday night.

All right, folks. So you just heard it here first. And in case you didn't hear OpenAI just went live and announced two major things and a bunch of little other things. one thing that they've announced is a new. Speech to text engine. and that's not just a speech to text model like Whisper was before. This is a speech to text model that's built on top of their transformer, models. And that one is significantly outperforming, I don't know if significant, but outperforming Whisper 2 and Whisper 3. And they claim state of the art on multiple languages as well. So that's crazy. and that's fairly cheap as well. There was a distilled version, even smaller version. One thing that I love about this is that it allows you to actually prompt the transcription understanding versus just providing a bunch of text words like we do for whisper to lecture. Let's say we're in Thursday. I can give it like the synopsis for Thursday. Let's say, And the other thing that just announced is a new. Text to speech from the other side, right? So like the speech to text first, converse your speech into text. So you can do some stuff like a transcription, and then, and translation as well. and then the other one is GPT for all mini TTS text to speech, which, I believe also is, yeah, it's GPT for all. So it's also based on the transformers, models and that one. you can prompt to be emotional. You can ask it to do some stuff. You can prompt the character a voice, and which is super, super cool. and to cover all this and to give us all the other stuff that they also announced. And, Quinn, welcome to the stage. I don't believe you need an introduction, but in case people still don't know who you are, we have a bunch of new people joining. Feel free to, cover yourself and daily super quick.

I'm a network engineer.

I do low latency stuff obsessively, to a fault. and so I have a lot of fun trying to squeeze latency out of these pipelines that we're all building for voice AI. you can follow me on Twitter because I tweet about this stuff all the time. And I, I'm so excited to be here. This has been a crazy week. I'm at GTC. I just got my like golden ticket for a 50 90. Let's go. Congrats. I'm excited about that. I heard

they ran out super quick.

That's what I heard.

Yeah, they're releasing a few every day and you have to stand in line. So it's a total NVIDIA like classic thing. Yeah, this is a really fun release. I think there are a couple of two really high level themes here. One is the architecture stuff, exactly as you were talking about, like building all of this stuff on top of these big LLM architectures as the labs and the open source community figures out how to train multi modality into the core, just opens up so many possibilities. So these, these are like fully promptable models as they showed in the live stream. The example that my colleague at DailyChad kind of put into one of the pipe cat like little single file examples for this was. He prompted the transcription model that he was going to talk about dogs and you can measure the difference between whether you know the transcription is right for like dog breed names or not with just that like really simple addition because the models are so good at context and they have so much context embedded in their weights already that you're just trying to like steer latent space. At least that's the way I think about it.

This basically opens up a whole field of prompting for these

models, which is crazy, which is like, we've been prompting for the last two and a half years for chat GPT. we've been doing way more than this for image models, right? Like prompting is a craft. And now, Whisper was basically just like, do this thing. Here's maybe a few keywords, but now for this model, for, for at least the transcription model, but for this one, you can just like prompt a whole thing into existence. This is crazy. There's like a whole new field open up.

There's not really a difference between.

translation and transcription internally in the model, right now, they're going to fine tune these models so that they do what you want to do for different use cases. And that's going to be a little bit intention with, with like making it totally promptable. But like, that's the same thing we're doing to add reasoning to models, right? Like you're trying to expand and steer and optimize like where you end up in the latent space. It just feels like it's a whole new territory in this domain. That's incredible. so

we have, I want to talk to you about the, the text

speech one, because text to speech one is very interesting to me. obviously they revolutionized the field with Whisper. Like I remember Whisper being, for me personally, changed my whole life trajectory because I started opening a startup using Whisper and then like it led to Thursday Eye. But generally Whisper just absolutely like changed the world in like how cheap a translation, how quality, high quality translation can be. Whisper is no longer state of the art. For, for many reasons, like there's multiple folks, that have trained, like better models as well. and that was a very specific model. This one is based on their series of like, of, of transformer models. What I haven't played with it yet, but like, what can we expect? Have you guys like looked at, cause I know like, daily, there's like, a bunch of transcription happening behind the scenes that you like offering as well. Have you guys tried those versus like whisper? Could you give us like a tidbit about that?

Yeah, we've been playing with it.

I think the two big things are Whisper wasn't really designed to be real time. So it's been pressed into service for some voice AI use cases, but mostly people have not used Whisper for conversational voice. The latency is just long enough that it blows your latency budget when you're trying to respond really quickly.

There's also the chunking thing with like 30, 30 second chunks

that you need to pass the chunk to the next chunk and that doesn't really like drive a live stream, yeah.

Yeah, there's always this trade off between latency and like

accuracy of the transcription, like how big, how big a buffer you're feeding it, how much context it has, all that stuff. And, and Whisper's fantastic, but just wasn't designed for the real time stuff. This stuff is, they, they really have real time top of mind as they're putting out these new models. And for those of us building the voice agent and the like voice and video agents, that's really exciting. I think the results here are really good, although I'm always careful about that because like when a model is brand new, we like, we haven't run it through all the evals, but I think these models are, I think they put OpenAI sort of back, back in the mix of like, would you use these for production use cases that, some other models have eclipsed Whisper for like real time stuff.

So one example of this is Whisper 3 when it came out we got all

excited about Whisper 3 and then many people just like Nope, I'm gonna use Whisper 2 like it's not good enough for this case use case that use case as well

I think that's another theme for me just because

I, I'm super interested in like how the, the big labs do this stuff. And I think the big labs, as much as they know internally are really learning as everybody externally build stuff. like, part of what is happening here with the prompting of these models, I'm sure they figured it out internally too, but like a lot of people have been using Gemini 2 flash for transcription. Because it's a big large language model and there's like, there's so much capability there and it's audio native. And so if you ask it to transcribe stuff, it actually does an amazing job. And then you can steer it and you can ask it to transform while it's transcribing. And that's, I think a lot of stuff is in this nice feedback loop between the big labs releasing stuff and the community build stuff. And then that informs how the big labs. Both prioritize what they release and train the next models.

One other thing they've added, and I want to talk

to you specifically about this. Cause you told me about this, back in, or engineer, semantic VAT. Let's talk about semantic VAT. You guys train the model, you released it and you kind of like, Hey, let's start a community and then open the accounts with semantic VAT. Could you explain the concept of semantic VAD to folks who have no idea what this is, and then let's talk about like what they released.

Yeah.

So the basic problem is like, how do you know when a human is done talking and the AI should respond? So you call it turn detection or in the academic literature, it's often called phrase in pointing. We're, we're also now calling it semantic VAD, which stands for voice activity detection. Voice activity detection is traditionally like a really small AI model that you train. On segments of speech and you, you, you categorize speech as speech or non speech. You categorize audio as speech or non speech. And you can use that to decide when, a human is done talking. But it's, it's a dumb way to do it, right? Because it's basically just a pause. I think because

everybody who listens knows when you talk to any voice mode and

you stop to think for just a second. If you talk to a person, that person would wait for you. When you talk to an AI, the AI immediately soaks in with like a bunch of bullshit. And you're like, no, I haven't finished my thought yet. I haven't wait, hold on. This happens with kids all the time. Kids take longer to like come up with something. This is why like my kids just absolutely hate talking with JGPT. Just like they, dad, dad, tell her to do this. They go to me cause like they know how I interact with the model. And I assume that this is experienced with everybody else. This is because the VAD is dumb. It's literally just waiting for silence, right? Like there's some, something like similar to this or, or maybe not silence, but like

Whatever a really small AI model thinks is not human speech.

You're waiting for a certain amount of time of those frames in a row. And so obviously the next step is to try to get more human like, and, thinking about how humans actually figure out when, somebody's done talking, it's a lot about. tone of voice, pace of speech patterns near the end of a, of a sentence, the grammar of the whole sentence. And so you're trying to bring semantics like language level knowledge in there. and like last year we were all just trying to figure out how to wire up all these pieces. So they worked at all. And now this year we're like, Oh, it's not good enough. Like we got it working, but it's not good enough. So what are the next things? And very clearly like one of the big pain points in 2025 is this like turn detection problem. And all of a sudden, a bunch of people are working on it, which is really exciting. Like I talked to people at all the big labs, and so like, everybody now understands this is a big problem. So it's super exciting that OpenAI released like SemanticVAD this week, and it's good. I don't know if it's as good as it's gonna get, but it's good. Like it's a big step forward. we also a couple weeks ago, released an open source, model we call SmartTurn for smart data detection. You can, you can participate in that. Effort to build like a truly open source, open data, fine tunable, like multi language, semantic VAD or smart turn model. It's get dot new slash turn G I T dot new slash turn. and the idea is to build a SOTA, like totally open source smart turn model because I want to see the big labs make amazing stuff. And I want to see like open, open source stuff also like pushing the envelope there.

There's also a great potential of learning

from the way they did this. Let's, let's say, let's give it at this point. semantic value is absolutely, absolutely important for folks getting to AI voice agents, to the level where people can interact with them properly and not. wait, et cetera. I think that I've changed my speaking patterns because of the way that I know how the VAD works. I'm sure that I'm not the only one. Like I'm sure that I go like, like I do all kinds of things to like, have a thing that I'm still talking and this is stupid. I don't want to change my stuff for AI, to change itself. So I think semantic VAD is like absolutely incredible.

We 100 percent do that.

We've become really bad testers of this stuff. My favorite story about that is like from that, that book about the development of the Macintosh where they didn't realize there was a bug in the mouse. Driver low level code until they released it to the public, where if you made a very small hiccup in holding the mouse button down with your finger, the whole system crashed. They didn't find it because everybody working on the Mac had practiced so much. They never, ever made that mistake. That's what we're all like talking to these boys. Yeah. luckily

we have, and I know you have, and I have as well, we

have amazing testers, kids, kids absolutely hate talking to it because of this one thing, because of like. The kid is still thinking like the sentence in the middle, the kids trying to wrap the brain around. So I'm very excited to see how that performs. I have a real life test with my kids. I think we covered pretty much all of the releases, right? So like two fully new, promptable ASR models, GPT 4. 0 transcribe and GPT 4. 0 mini transcribe. I think they said 0. 6 cents and 0. 3 cents for, for each of them. I think point sense something, we need to confirm this once, once the actual, documentation lands. There's also the new voice model called GPT 4. 0 Mini TTS, which is, all of them are based on GPT 4. 0. This, distilled and like pruned and tiny versions of this, they run also for real time, which is incredible. we talked about semantic We didn't talk about noise reduction and Conversation. item. retrieves, Quinn, would you like to cover those and tell us what those are about?

Another problem for, like, production voice agents is,

like, picking up background noise that, that does sound like speech. like, if you have the radio on in the room and you call a customer support line, typically that really messes up the voice agent because you end up with the background speech being transcribed. So they released a new noise reduction model that filters out things like background speech and it's tunable, like, it's got a few different Keywords for like near or far or something like that. I can't remember exactly, but if you're building voice agents, you should experiment with that and you should do some testing with things like having a radio on in the background when you call the agent because that tends to be a real world thing that you hit as soon as you deploy to production conversation item. Retrieve is a really nice. improvement to being able to manage the context on the, on the kind of agent framework side. So you're, you're like talking to the, the OpenAI real time API in code, and you're just sending it audio, and it's chunking up that audio and creating conversation turns. But previously before they released conversation item retrieve, you didn't have any ability to introspect into what those turns actually were. So you couldn't do things like transcribe one turn at a time and save it in your like back end persistent storage system. Now you can, now you know exactly what the model heard because you can ask to get That back from the API, which is actually really, really valuable.

we need to move on because, they took a big chunk of our show.

Usually with live streams, there's like a bunch of stuff they talk about. We can condense, hopefully on the, on the recording. We can condense Quinn. Thank you so much. The one thing that you barely mentioned, and I want to mention for you is daily is a company that does real time stuff. Like we can basically do this meeting on daily, let's say, and have like the super low latency as well. as you said, network engineer, but also pipe cat is something that you guys released and working on. There's the open source. Software, that, that allows to play with all these things was like one line of example. So if, for folks who are listening and like very interested in like, Hey, I want to try this new TTS. So I want to try this like new transcription, et cetera. daily is an absolutely greatest way to go about this and like super quickly install this. So as we had a chat before with Joseph Nelson from herbal flow and their supervision stuff, you guys are. similarly approach as well. You have a company that does this thing and inference and like all of the servers as well, and you also have the open source that you support. We love open source. You don't say, yeah, Quinn, thank you so much. We love daily as well. And thank you as always coming and giving us like the absolutely like. Most professional breakdown of everything that happens in voice and we expect to hear from you more. Don't leave yet Because the next item that we're gonna talk about is also from this world and maybe you have thoughts on this as well What we there's a bunch of voice and audio stuff that happened. So the live stream of OpenAI we just covered We're gonna add the details. There's two new things that happened They may not be as interesting as what just announced but Canopy Labs drops Orpheus 3b Orpheus is a also a language model. So folks, basically what we just talked about about GPT becoming like a language model that you can prompt for emotions. Orpheus is a 1 billion parameter, 500 billion and 150 million version of a natural sounding speech language model. Eh. I will tell you upfront, it's unfair to compare open source to the absolutely state of the art, the open AI releases. Always, we always talk about this, right? When Mistral comes out, et cetera. however, they sound good folks. I don't know, like, Wolfram, you had a chance to play with this? We can't be left stuff. They, they have zero shot cloning, which OpenAI does not give you yet, as far as I saw, like this TTS new thing, I cannot put Alex's voice and have the TTS, like make Alex, I can try to prompt it, Hey, Ukraine is really, accent, very excited dude that speaks very passionately about stuff, but likely they won't get my voice. these open source models give you voice, like zero shot voice cloning, which is super cool. They released like fine tunes on voices as well, trained on a hundred thousand tokens, with a hundred millisecond latency and Apache to license. You cannot beat Apache to license. Apache to is incredible because you can train with a bunch of stuff on your voice, et cetera. Wolfram, any, any thoughts from you on Orpheus?

Yeah, the one thing not on the list is

that it is not multilingual. So that is not a feature multilinguality. I tested it in German and it, it had a heavy accent, but I'm still excited because it's open source. It can be put in and you can run it locally and it, yeah, it, it fits on your local systems. The three B isn't all that big, so you can do it and it will only get better as a support. I think there will be a lot of support for this.

We will

hear about it a lot.

And you can train it for other languages too.

Yeah, you can train it.

You can fine tune this. Quinn wanted to ask you about just open source models and your approach. I know that like, Pipecat supports like the production models, MNAs, etc. what's, what's generally the thought, on your side from like open source models such as this one for agents and building things?

I love these.

I think there's two really interesting things about these models. One is they're exposing like new architecture approaches that, there's lots of this stuff inside the big labs, but we don't always get to see it. And, and building like true full duplex streaming mode, for these models is something that like the open source community is really pushing on in a really cool way. so there's like at least four kind of speech to speech models that are pushing on the architecture in different ways. There's like Sesame, there's Moshi, there's Ultravox. I haven't actually looked at the code for Orpheus yet. I just like played around with it. It's really good. I want to look at the code, cause I think it's really, in that same category. And then there's also super small. speech models, which, what those open up is the ability to run on device. On device, yeah. Like Kokoro is, is amazing for 82 million parameters. Like it's unbelievable to me that it's so good and so small.

We're blown away by Kokoro.

Absolutely. Everybody like was like absolutely blown away.

Yeah.

So in, in Pipecat, like the open source stuff, we just try to support all the models. Like I try to write on the weekends, I try to write like Pipecat services for all the models when they come out or, people submit PRs because like, if you're playing with on device stuff. you still need an orchestration layer. It's not like you only need the orchestration layer if you're, if you're calling out to the, the big lab APIs. so I'm super excited about like all, all this stuff and it's, and it's developing so fast.

And we love it that it's an open source, like helps

other things develop as well. Last thing that we will talk about here, NVIDIA released Canary, which is 1 billion and 180 million flash speech recognition translation, which kind of pales in comparison to the other news, like with all my love to NVIDIA, the TTS, the state of the art now from OpenAI, but, but I think that Canary is actually like open source. So that's something. CC by license, which allows commercial use. this is a model that you can run. Many folks may not want to go to OpenAI for this latest model. I actually, let me, you guys know, I love OpenAI, but Whisper being open source, absolutely changed the game for everyone. The fact that this new TTS model is not open source. I don't love, I love folks from OpenAI know that I get excited and hyped about all the things, for the, sorry for the TTS version. Okay. May not release this open source. There's problems, but the, the speech to text model, what, why, what's, what's wrong with speech to text model being open source folks? You release whisper, you change the world. give us speech to text in open source, please. Give us this like new GPT. I think it's because it's based on GPT. They will never open source it until such a point or the open source GPT 4. 0. but Hey, we need this in open source, please. But, this new one from, from NVIDIA and NVIDIA Canary, looks pretty cool. Get second place on this. Speech recognition, leaderboard on Hug Face. Shout out to Vaibhav and everybody who runs it from the team. very tiny size, 180 million, can run on device as well. German, French and Spanish. There's a few, I think Whisper supports like 190 languages. This supports four as well. so that's pretty cool. So shout out to the folks from Nvidia for releasing this. And this concludes our. Voice and audio segment, which was big. And this also, with this, we'll say thank you. Gwendolyn will be mindful of your time. we're also be mindful of the fact that we're over two hours here, but we'll open, open the eye stuff that usually pushes us a little bit over. so the last two things that we need to cover, let's do things we need to cover. is the 3d stuff from Tencent. I absolutely want to show, but before this, I want to say hi to a friend of ours. Lucas. Hey, Lucas. How are you?

Hey, I'm good.

Can you guys hear me?

I you're coming through loud and clear.

Okay, so Lucas, we're going to get to what you guys have announced in just a second, just before this, though, for folks who've been waiting, I absolutely want to show what we're talking about here, in. Intense and 3d stuff. So Tencent updates, who knew on 3d, if you guys remember, we talked about this before, who knew on is one of the best performing video models. That's open source at Tencent is a Chinese company, obviously. also released like image to video, but just like super cool stuff that you can find on these models, et cetera. Hunyan 3D is their state of the art 3D generation model from one picture or multiple pictures. And they also released like a turbo thing. It's, it's all super cool. So I just want to show you guys like what we're talking about. This is an example of me generating a mouse character, just by going to Gemini native image output. Just ask you for a character, literally drag and dropping that image that generate, Gemini generated into the, into the Huion 3D Turbo. And they released this, state of the art model that generates 3D images for you. it, like, it, it's state of the art both in open source and closed source. That's what they claim. So this open source model that I, I don't believe it's Apache, but we can check on the model, but it's definitely the open source. It's stated in geometry details, conditioning alignment, and texture quality. Texture is also ridiculous. Like you can, you can see how close this matches this one image. This was one image. The process from creating an image into 3D now is just like, like this, like this. Not only that. They released a, a new acceleration technique called Flash VDM that they also open sourced that makes this model go brrr and, and go up to 30x with the turbo model. This is ridiculous, folks. You can generate this quality of 3D imagery, with less than one second on an H100. One second of a image to 3D model. This means we're getting to the point where it's near real time with consistency as well. Cause those are the image inputs folks. We're getting to near real time 3D generation with like the example of these models. I had to zoom in on my face to see the excitement that I have. The additional thing they released is this thing. generally, these 3D models, when you provide one image, let's say the frontal image of a character, these models have to come up with what's behind, right? Like in the case of this mouse, for example, you can see as a person, you can say, okay, he has like a tail that goes and curves, et cetera. so when this model generates the stuff, they need to kind of like assume, the model needs to assume what's happening on the backend. I think this is the first time that I've seen something like this. There is MV, multiple view, where you can provide to the same model, the frontal view, the side view, the back view, and the other side view. And that will generate, and you can see the difference here between Hunion 3D 2 and then 2 MV. You can see that the, the, the, We're showing a picture of, I don't know, a dog with, sunglasses, whatever. So you can see multiple things that are actually significantly better. You can see that the, the, the strap around the sign, the glasses is correct on the multiple view thing. You can see the tail is correct. You can see like a bunch of stuff. So you basically can generate like a full 3d image. That's perfect. And not just like a hallucination of the rest of it. And the amazing thing is folks, we're now at a time where like Gemini. Literally can generate those for you. I just want to show you that I did this. Let me go and find, I'll show you here. You can ask Gemini to generate this image or upload a different image that you have only the front face of. And you can ask Gemini, Hey, rotate this in 90 degrees. Gemini will fucking do it. This is insane. This to me is like absolutely mind blowing. You need four images from a character reference sheet, let's say, of like front, left, back, and etc. And you can go to Gemini, upload one, and say, Hey, Gemini, What will this look like rotated and general will absolutely give it to you with like just straight up This is an absolutely mind blowing time that we're living in right now I just wanted to highlight the fact that the pipeline from your thoughts to a 3d world is like this And if you have a vision pro you can Pull those 3d things and put them in your physical space. If you have a 3d printer, you can print them and put them on your actual physical desktop. This is super cool. So shout out to Tencent, folks who are into the 3d world. I wanted to highlight the multi view and the turbo are actually improving our lives. As Jensen says, every pixel will be generated and not rendered very soon. We're going to the world where like holodeck is like right there. Like we literally can. Talk to the voice agent that OpenAI just released, have it transcribed into, into the Gemini commands that will then generate the images, then take those images, shove it into 3D, 3D generates them in less than one second, they just like pop in front of you. The possibilities are endless. The addition of like the MCP Blender stuff that's been blowing up, to put those into actual games, those are crazy. So I just wanted to highlight this. By the way Yes.

By the way, you can install it.

If you have just six gigabyte VRAM is enough, then you can one click and install it from Pinocchio and use it locally, just six gigabyte.

That's not going to take one second to generate,

but that's absolutely awesome. You can just like run this locally and pipe it into a 3d printer. We have more breaking news. Can you believe it? Like I want to end the show, but we have breaking news. I don't think my co hosts know what I'm talking about, by the way. AI breaking news coming at you only on Thursday night. All right, folks are breaking news for today. Additional breaking news is, we have, the biggest lab that was late to this thing and tropic releases, web search. And so far when you talk to Claude on AI, it was just Claude, they were the last, I believe the last company, even open router added like XAI, whatever. so finally Claude can now do search on the web. you can enable this and each response include inline citations and you can verify the sources. what else is interesting? You can just like. Toggle it, in all the U S only pay plans. So they're still not caught up fully, but it's a small thing that for folks who use the cloud AI stuff for artifacts, for the best models, for coding, et cetera. that's been missing for a while. there's been a for a while. And then, they're also rolling support to a free plan as well, for more countries soon. So they're going to get to the point of like free tier as well. and tropic is known for their safety, assumptions, et cetera. So that's why probably they didn't release it for a while, but Hey. Shout out to Entropic for this and I just want to say that we're at the two hours and 20 minutes and we probably need to start. Closing down, but a huge thanks for everybody who tuned in. the, the thing that we wanted to do for this week's buzz and with somebody so we're probably going to do next week. We're going to talk to you about the MCP server as well. but shout out to my whole team who worked on this and we'll definitely cover that work. it's been an incredible show this week. Incredible show. So much open source, folks. Yama's back, OpenAI released, 01 Pro and now Whisper Killer and also a new TTS they can prompt. We had daily, we had so many guests, incredible guests and friends of the pod, we had like Joseph from Roboflow talk to us about state of the art models that they just released, he released it. Right here. Like we're basically Niton had to go and like, sleuth their GitHub for, for finding the links. I didn't even get the links before. So it

was 11 minutes?

Yes. That it had been updated. Yeah.

we actually are bringing you breaking news, not only

discussing breaking news as well. and we also then talked with, Quinn from daily covering everything, the open air list with open source n TTS as well. We always got somebody from Nvidia. and then NVIDIA PR said, no, so NVIDIA PR, we're an amazing show to come and talk to your new models about open source specifically. So please send your people here. with that, I want to thank Wolfram, I want to thank Yam, I want to thank Nistan. Thank you guys, for coming up and co hosting this. thank you for all our guests today, Lucas and, Kwin and from Daily and Joseph as well. And thank Most of all to everybody who tunes in and supports the show from multiple ways by sharing with your friends, by giving us five star reviews, by subscribing to YouTube channel, folks, we're almost at a thousand subscribers. Please subscribe to YouTube channel. it's great. You'll get the, the, the things when we go live as well. the best way to support the show is this one comment that I shouldn't be triggered, but triggered me. We are not, when we will be sponsored, you will know. I will definitely absolutely make sure that if we have a sponsor segment, you will know the only sponsor for this show currently is Weights Biases and our product called Weave, which is LLM experimentation, tracking, LLM observability, and evaluation. That tool I talk about all the time. That's the only sponsor everybody else is here because we think that telling you about them is a service to our community. And those are friends that we love and know they're doing good, great work in open source as well, including everybody who was here on stage today. with that, I want to wish you a great weekend, a happy Thursday and a lot more AI releases until next week. Thank you everyone for coming up. and, As always, if you missed any part of the show, it's going to be available on thursday. ai. every place where you get your podcast and the newsletter. Leave us a five star review. Thank you much. Bye bye.