WEBVTT 00:00:00.052 --> 00:00:05.192 Alex Volkov: AI engineers are sleeping on the personal agents renaissance explosions happening in 00:00:05.212 --> 00:00:06.612 the real world. It's kind of crazy. 00:00:06.802 --> 00:00:11.022 Nathan Benaich: Americans say that they're much, that they much prefer living near a nuclear power 00:00:11.121 --> 00:00:12.241 plant than a data center, 00:00:13.162 --> 00:00:14.621 which is, uh, quite wild. 00:00:14.812 --> 00:00:17.731 Alex Volkov: Anthropic is also going to do an explosive IPO. 00:00:17.951 --> 00:00:23.212 they reported an insane quarter, uh, 16.8 billion dollars, about 00:00:23.352 --> 00:00:26.272 45% from the second quarter increase. 00:00:26.462 --> 00:00:28.362 Omar Sanseviero: we've been doing optimizations for Gemma. 00:00:28.462 --> 00:00:33.881 We have been able to achieve over 100,000 tokens per second with the E2B, so our 00:00:33.982 --> 00:00:34.741 smallest model. 00:00:34.942 --> 00:00:36.002 Alex Volkov: Andrej Karpathy says 00:00:36.621 --> 00:00:39.421 6 billion people have barely touched an LLM. 00:00:39.622 --> 00:00:44.121 Peter Gostev: I'm very pro rumor, but, you know, uh, the- I'm also a psycho, so, you know, 00:00:44.822 --> 00:00:45.661 you don't have to be. 00:00:45.841 --> 00:00:47.722 Alex Volkov: Welcome to ThursdAI for October 15. 00:00:48.122 --> 00:00:49.282 Uh, I'm live from New York. 00:00:49.362 --> 00:00:53.762 Uh, my name is Alex Volkov, I'm an AI evangelist with CoreWeave, and with me 00:00:53.882 --> 00:00:58.041 Wolfram Ravenwolf, also AI evangelist, also from CoreWeave, and Peter Gostev, 00:00:58.522 --> 00:01:01.482 evals and model behavior at Arena. 00:01:02.042 --> 00:01:04.692 And I just spent 3 days at AI Engineer here in New York. 00:01:04.752 --> 00:01:06.712 I can't wait to tell you guys all about this. 00:01:07.431 --> 00:01:11.992 And while I was there, there was an incredible week just blowing up, so, so we 00:01:12.032 --> 00:01:13.272 had to try and catch up. 00:01:13.712 --> 00:01:15.671 Uh, Meta shipped Muse Wonder, 00:01:16.351 --> 00:01:17.331 which was rumored 00:01:18.032 --> 00:01:22.881 by them as as this big watermelon model, and it smells big. 00:01:23.062 --> 00:01:26.662 It definitely smells like a big model, and you can use it right now in in Muse, and 00:01:26.702 --> 00:01:30.262 we're gonna talk about this. Uh, it's number 2 on Arena, which is 00:01:30.942 --> 00:01:33.701 puts Meta Square in the Frontier Lab race. 00:01:34.282 --> 00:01:36.302 As we told you before multiple times, 00:01:36.982 --> 00:01:41.061 do not, under any circumstances, do not bet against Zach. 00:01:41.382 --> 00:01:43.582 We told you. We told you this 2 years ago. 00:01:43.982 --> 00:01:48.301 We told you this after Llama 4 dropped, it was a disaster, and we told you this, uh, 00:01:48.422 --> 00:01:50.541 when he started hiring people with soup. 00:01:51.062 --> 00:01:52.372 And so now we're gonna feel it. 00:01:52.452 --> 00:01:53.701 Peter, I think you you tested it out. 00:01:53.802 --> 00:01:54.801 I think you have some thoughts. 00:01:55.441 --> 00:01:56.322 Uh, would love to hear 00:01:56.362 --> 00:01:56.462 Peter Gostev: Yeah. 00:01:56.482 --> 00:01:56.801 Alex Volkov: from you. 00:01:57.362 --> 00:02:02.042 Peter Gostev: Yeah, maybe to, to differentiate, we do have few different leaderboards, and they, 00:02:02.202 --> 00:02:04.261 they are second on the text arena, 00:02:04.282 --> 00:02:04.661 Alex Volkov: Mm-hmm. 00:02:04.682 --> 00:02:08.261 Peter Gostev: but they're actually 12th on the agent arena. 00:02:08.762 --> 00:02:09.681 Alex Volkov: Interesting. 00:02:10.202 --> 00:02:12.502 Peter Gostev: Yeah, and it, it's kind of interesting to interpret. 00:02:12.562 --> 00:02:15.142 I think we do have this kind of divergence now going on. 00:02:15.202 --> 00:02:18.921 So, for example, Argon, uh, from Google as well did really well on text. 00:02:19.202 --> 00:02:20.241 So it's actually first. 00:02:20.922 --> 00:02:25.502 Um, so you can roughly interpret it as, how much people like to chat to the model. 00:02:26.182 --> 00:02:30.781 So it's interesting that there is a divergence between what people want, who 00:02:31.142 --> 00:02:32.601 people want to talk to versus 00:02:33.342 --> 00:02:37.702 agentic capabilities and coding and all of that, and it looks like it's different. 00:02:37.862 --> 00:02:43.721 So yeah, Argon is fourth on Agent Arena, and Muse won the max, so it's not the full 00:02:43.902 --> 00:02:44.242 version. 00:02:44.262 --> 00:02:44.481 Alex Volkov: Mm-hmm. 00:02:44.502 --> 00:02:46.061 Peter Gostev: It's on, it's ranked 12th. 00:02:46.862 --> 00:02:49.672 Alex Volkov: Uh, very interesting. We, we also have to talk about the whole thing where I saw the 00:02:49.772 --> 00:02:50.211 timeline, 00:02:51.212 --> 00:02:56.762 react negatively because, Muse Wonder wasn't able to recreate the 3D scenes in, 00:02:57.462 --> 00:03:00.622 and people were like, oh no, this is a bad model because it, it can't compete with 00:03:00.722 --> 00:03:02.641 Astra on like the 3GS thing. 00:03:03.082 --> 00:03:06.162 Uh, we have to address this because I don't know if my assistant 00:03:07.062 --> 00:03:11.522 every day needs 3GS, so I don't know if this is like a personal thing that I need 00:03:11.582 --> 00:03:14.142 my assistant to do. Okay, uh, what else? 00:03:14.262 --> 00:03:18.201 Okay, so folks, la- later today, Omar Sanseviero from Google DeepMind joins us 00:03:18.242 --> 00:03:20.572 to talk about it, about Argon specifically. 00:03:20.732 --> 00:03:25.811 We also have, thanks to Peter, we have, Nathan Benaich from the State of AI Report 00:03:26.032 --> 00:03:28.712 joining us in an hour to talk about the state of AI. 00:03:28.952 --> 00:03:30.552 Uh, they've been running this for a decade, Peter? 00:03:30.592 --> 00:03:32.311 Is is that, is it's like a crazy thing. 00:03:32.382 --> 00:03:33.682 I'm really excited to hear from him. 00:03:34.142 --> 00:03:34.322 Peter Gostev: Yeah. 00:03:35.182 --> 00:03:36.942 Yeah, no, he's a, he's a great guy. 00:03:37.062 --> 00:03:39.041 Yeah, he's uh, he's been doing this for ages. 00:03:39.102 --> 00:03:41.901 Yeah, I actually haven't checked how long, but yeah, I've been reading it for many 00:03:41.942 --> 00:03:42.181 years. 00:03:42.462 --> 00:03:46.315 Alex Volkov: I think it's, I think it's a decade, but Slopos prepared the news while I was 00:03:46.395 --> 00:03:49.315 running at the engineer, so hopefully Slopos did a good job. 00:03:49.595 --> 00:03:51.195 Uh, good job, Slopos. Um, 00:03:51.835 --> 00:03:53.035 and then what else? 00:03:53.195 --> 00:03:55.234 Oh, Anthropic had an investor day. 00:03:55.435 --> 00:03:57.535 So before we only had rumors and leaks, whatever. 00:03:57.635 --> 00:03:59.015 Anthropic is about to go IPO. 00:03:59.195 --> 00:04:00.034 We have some numbers 00:04:00.835 --> 00:04:06.274 and a specific thing for kind of the only sponsor for this podcast, CoreWeave, with 00:04:06.315 --> 00:04:07.414 our Agent Lens product. 00:04:07.515 --> 00:04:11.314 Uh, Anthropic Claude code specifically now supports open telemetry. 00:04:11.635 --> 00:04:16.644 So you literally can start running Claude code and then see your results in, um, 00:04:17.525 --> 00:04:20.005 in Forge Agent Lens, which is super cool. 00:04:20.365 --> 00:04:23.424 I think I've seen some examples already, uh, so we're definitely going to talk 00:04:23.445 --> 00:04:24.684 about this at least a little bit. 00:04:25.245 --> 00:04:29.314 And then, uh, so after Nathan Benaich, we'll talk about state of the art, but 00:04:30.155 --> 00:04:30.495 I think 00:04:31.515 --> 00:04:35.954 since I was a little bit of out of the loop, I do have a whole prep thing here. 00:04:36.515 --> 00:04:39.554 Would love to hear from you guys about the highlight of the week. 00:04:40.155 --> 00:04:41.354 I think we'll start with Peter. 00:04:42.915 --> 00:04:46.755 Peter Gostev: Well, my highlight was, uh, I was at the NVIDIA event yesterday. 00:04:46.915 --> 00:04:47.974 Alex Volkov: Oh, you met Jensen, right? 00:04:48.994 --> 00:04:49.774 Peter Gostev: Yeah, yeah, he's 00:04:49.795 --> 00:04:50.535 Alex Volkov: Tell us, tell us. 00:04:50.714 --> 00:04:52.274 Peter Gostev: He was watching on the show. 00:04:52.355 --> 00:04:57.204 Yeah, they they had a kind of, it was very confusing afternoon because during the 00:04:57.845 --> 00:05:01.884 day, during the afternoon, they were building, they had this kind of builders 00:05:01.965 --> 00:05:07.274 event, so it was all like, people you meet at AI engineer conference kind of thing. 00:05:07.755 --> 00:05:08.954 So that was really fun. 00:05:09.355 --> 00:05:09.535 Alex Volkov: Yeah. 00:05:09.755 --> 00:05:14.674 Peter Gostev: Um, and yeah, met a few people from Twitter who I haven't met before. 00:05:14.995 --> 00:05:18.954 They had like, uh, it was in the office in London of a company that does like 00:05:19.075 --> 00:05:23.755 humanoid robots, uh, which I had no idea that it was central London, you know, so 00:05:23.955 --> 00:05:24.634 that was awesome. 00:05:25.235 --> 00:05:29.754 Um, yeah, they were kind of a, it was all about OpenClaw kind of thing, so they were 00:05:29.795 --> 00:05:34.835 kind of showing how they also launched the laptop, right, the, the one with the kind 00:05:34.875 --> 00:05:35.634 of Spark, 00:05:36.355 --> 00:05:38.815 um, or, or rather did it start shipping or something? 00:05:38.875 --> 00:05:41.014 I think they announced it a few months ago, right? 00:05:41.355 --> 00:05:41.574 Alex Volkov: Mhm. 00:05:41.715 --> 00:05:44.394 Peter Gostev: Um, so yeah, a few cool things like that. 00:05:44.515 --> 00:05:47.435 Yeah, and then in the evening it was very different. 00:05:47.755 --> 00:05:49.334 It was in a fancy, 00:05:49.995 --> 00:05:55.934 uh, kind of, uh, reception lounge, uh, of a, I know there's a weird converted church 00:05:56.155 --> 00:05:57.655 building in, in London, and there was 00:05:57.675 --> 00:05:57.974 Alex Volkov: Nice. 00:05:58.235 --> 00:06:02.075 Peter Gostev: AI minister and all of the fancy people from, I don't know, 00:06:02.914 --> 00:06:07.594 universities and industry and a very different, very different vibe. 00:06:08.595 --> 00:06:10.514 Alex Volkov: That's so cool, man. I- it's kind of like, 00:06:11.478 --> 00:06:15.995 besides the AI engineer London I, that we both went to and we recorded from, this is 00:06:16.015 --> 00:06:18.255 the first time that you're saying something about, like, somewhere else 00:06:18.355 --> 00:06:20.154 outside of the States that gets me FOMO. 00:06:20.715 --> 00:06:21.994 I'm like, oh, that sounds super cool. 00:06:22.435 --> 00:06:23.514 I wish I would have been there. 00:06:23.955 --> 00:06:25.314 Uh, that's your highlight, obviously. 00:06:25.475 --> 00:06:28.305 Uh, the, folks are welcome to check out Peter's timeline for this. 00:06:28.585 --> 00:06:32.645 Uh, you posted, uh, very well from there, um, and you can tell us more about this. 00:06:32.785 --> 00:06:34.325 And then, Wolfram, what's your highlight? 00:06:34.424 --> 00:06:35.845 What's your one highlight from this week? 00:06:36.745 --> 00:06:40.805 Wolfram Ravenwolf: There's so much stuff, but if I have to pick one, just one that I'm currently most 00:06:40.864 --> 00:06:43.964 excited about, it's an open weights release, uh, of Fish Audio. 00:06:43.985 --> 00:06:44.364 Alex Volkov: Fish Audio. 00:06:44.385 --> 00:06:45.984 Wolfram Ravenwolf: Fish Audio Drama 3. 00:06:46.745 --> 00:06:50.924 Yeah, that is currently, I mean, there's so much, I could na- name multiple things, 00:06:50.985 --> 00:06:51.125 but 00:06:51.745 --> 00:06:57.505 Alex Volkov: Since we're going to cover, Gemini 4 Argon, and you are our Google guy with the 00:06:57.904 --> 00:07:02.014 Pixel phone and whatever, and you didn't choose, Argon, I think that I'm very 00:07:02.075 --> 00:07:04.994 interested to hear what you think about Argon, since this wasn't your highlight. 00:07:05.475 --> 00:07:06.054 Uh, I think it's 00:07:06.075 --> 00:07:07.935 Wolfram Ravenwolf: Yeah, definitely. I can say something about this. 00:07:07.955 --> 00:07:09.834 Alex Volkov: Okay, yeah, we'll definitely, we'll definitely get to that. 00:07:09.995 --> 00:07:12.194 Um, highlight of the week for me. 00:07:13.034 --> 00:07:13.474 Um, 00:07:14.115 --> 00:07:15.634 okay, this is a highlight and a hot take. 00:07:15.915 --> 00:07:20.885 I've spent the last 3 days among the top AI engineers, in New York, and also this 00:07:20.925 --> 00:07:21.525 AI engineer 00:07:22.245 --> 00:07:22.704 conference 00:07:23.325 --> 00:07:23.985 was adjacent 00:07:25.325 --> 00:07:27.304 to obviously Wall Street, so this was a 00:07:28.258 --> 00:07:29.738 finance and AI engineers together. 00:07:29.818 --> 00:07:33.538 So there's a lot of finance bros, a lot of, big trading companies, a lot of banks. 00:07:34.619 --> 00:07:38.559 The highlight for me is I realized something that I kind of Wolfram and I 00:07:38.579 --> 00:07:40.498 talked about on the show, Peter, maybe with you as well. 00:07:40.939 --> 00:07:46.259 Um, AI engineers are sleeping on the personal agents renaissance explosions 00:07:46.298 --> 00:07:48.059 happening in the real world. It's kind of crazy. 00:07:48.499 --> 00:07:51.459 Uh, so folks who are listening to ThursdAI, I know you guys are AI. 00:07:52.059 --> 00:07:55.279 Hopefully we do a good job to tell you about the stuff that's happening outside 00:07:55.319 --> 00:07:56.139 of our bubble as well. 00:07:56.779 --> 00:08:00.439 It is kind of crazy. It feels to me that because everybody's building their own 00:08:00.539 --> 00:08:04.238 harness, or they're so used to Claude Code doing stuff for them, they're trying to 00:08:04.499 --> 00:08:05.298 force this, 00:08:06.832 --> 00:08:09.792 behavior into tools that don't need to do these tools. 00:08:10.232 --> 00:08:15.232 Meanwhile, the world is changing for normies because of Grokbot and Muse and to 00:08:15.272 --> 00:08:17.382 some extent bots, or dots, 00:08:18.022 --> 00:08:22.262 to some extent dots. Uh, and and this is like the summary for me because I'm like 00:08:22.342 --> 00:08:26.352 very, personal assistant pilled or personal agent pilled, and I see that I 00:08:26.432 --> 00:08:29.792 talk to folks and they don't have one running around 24/7 doing stuff for them. 00:08:30.322 --> 00:08:33.922 this is my highlight of, this is not the AI news, but this is news to me, 00:08:34.882 --> 00:08:38.922 that even the engineers that we talk to every week for the past 3 years that love 00:08:38.962 --> 00:08:42.482 to stay at the forefront of what's happening are missing a huge, huge thing 00:08:42.522 --> 00:08:45.921 that's happening. So hopefully we're here to tell you about this huge new thing, and 00:08:45.962 --> 00:08:50.041 we're here to make sure that you guys don't miss it, because I think it is huge. 00:08:51.002 --> 00:08:52.682 Because most of the normies that I talk to 00:08:53.562 --> 00:08:55.542 did hear about some of these, personal agents. 00:08:55.662 --> 00:08:56.222 Wolfram, go ahead. 00:08:57.102 --> 00:08:59.161 Wolfram Ravenwolf: What really excited me this week, it's another thing. 00:08:59.222 --> 00:09:03.282 It's not an AI release or something, but it is something someone I very much 00:09:03.422 --> 00:09:08.621 respect said, because Linus Torvalds really said that not using AI to create 00:09:08.782 --> 00:09:09.542 code is like, 00:09:10.182 --> 00:09:14.121 it's just a tool, so if you don't use it and you are a smart person, you should be 00:09:14.182 --> 00:09:18.542 using it. He, he said if anyone is submitting, as an example, it's, it's as 00:09:18.662 --> 00:09:21.742 crazy to say you can't use AI for doing something. 00:09:22.142 --> 00:09:26.821 It's like saying you can't use a compiler to create code, because that is also a 00:09:26.982 --> 00:09:31.062 machine tool that you are using, and you are not using assembly, assembler to 00:09:31.222 --> 00:09:35.582 create the code. And that was an example, and it's great to see the mindset change. 00:09:36.182 --> 00:09:39.921 I mean, he already, already had an open mindset, and he says, as long as you 00:09:39.982 --> 00:09:41.942 understand the code, I don't care how you create it. 00:09:42.262 --> 00:09:45.701 That is the important thing, and I think more people should be thinking like that. 00:09:46.022 --> 00:09:50.261 It doesn't matter what tool has been used, but did you, what you ever, whatever you 00:09:50.302 --> 00:09:52.742 created, do you understand it, and does it work? 00:09:53.102 --> 00:09:55.121 That is the important thing, not how you create it. 00:09:55.582 --> 00:09:57.055 Alex Volkov: The do you understand the part is interesting. 00:09:57.112 --> 00:09:59.991 we should definitely talk about this because I definitely slop. 00:10:00.232 --> 00:10:03.362 I turned my slop cannon towards some projects that, Peter, I think you you can 00:10:03.462 --> 00:10:07.141 resonate. I have no idea what's going on under the hood, as long as I see it and it 00:10:07.222 --> 00:10:11.011 works and it's dope. Uh, but I do agree, there's some stuff, especially in critical 00:10:11.072 --> 00:10:13.591 infrastructure, so the folks at OpenAI, uh, 00:10:14.232 --> 00:10:18.131 let me scratch that. The folks at AI Engineer from banks, for example, there's 00:10:18.152 --> 00:10:18.712 a lot of stuff 00:10:20.312 --> 00:10:22.171 that needs to be human readable. 00:10:22.312 --> 00:10:24.432 There's a lot of stuff that they need to understand everything. 00:10:24.512 --> 00:10:25.772 Somebody taught me the concept of 00:10:26.432 --> 00:10:27.512 toxic access, 00:10:28.392 --> 00:10:28.672 where 00:10:29.872 --> 00:10:33.491 the loan book, the master loan book record or whatever, something inside the bank 00:10:33.552 --> 00:10:35.721 that, all the money lives in for everybody. 00:10:36.062 --> 00:10:39.522 if there's people with access that can change some stuff, that's like toxic 00:10:39.782 --> 00:10:41.061 access or something like that. 00:10:41.422 --> 00:10:44.981 Uh, so that, obviously, people need to understand what's going on there and be 00:10:45.062 --> 00:10:48.711 accountable as well. let me run to TLDR, which is a section of ThursdAI where we 00:10:48.772 --> 00:10:53.371 tell you briefly about the most interesting, exciting parts of what 00:10:53.452 --> 00:10:55.891 happened, curated to our interests. 00:10:57.132 --> 00:10:59.991 I will, I think I'll stop saying that soon, but we used to be able to cover all 00:11:00.032 --> 00:11:04.311 the news. Uh, I no longer, with fair conscience, can say this is all everything 00:11:04.332 --> 00:11:05.451 that happened in the world of AI, 00:11:06.092 --> 00:11:09.132 but here is the most important thing that happened in the world of AI. 00:11:09.172 --> 00:11:09.571 Let's go. 00:11:19.752 --> 00:11:24.392 All right, here's everything on ThursdAI that we're covering on October 15th in 00:11:24.491 --> 00:11:26.291 about 5 minutes. Let's get into it. 00:11:26.755 --> 00:11:30.042 So we're live from New York after AI Engineer, uh, where I think the highlight 00:11:30.082 --> 00:11:30.941 was Capital One 00:11:31.562 --> 00:11:36.242 Max Kanat Alexander made the best case I've heard when code quality still 00:11:36.322 --> 00:11:37.342 matters. This is 00:11:38.162 --> 00:11:41.331 probably the MVP talk, and I would recommend folks to, to tune into the live 00:11:41.392 --> 00:11:42.672 show and wait for the release. 00:11:42.912 --> 00:11:43.271 Um, 00:11:43.872 --> 00:11:47.062 swyx also came on stage, at his own conference and talked about how to kill 00:11:47.082 --> 00:11:48.442 your SaaS, uh, hackathon. 00:11:48.481 --> 00:11:52.781 There was a hackathon this weekend, and 72 clones of big SaaSes, spent 100 billion 00:11:52.862 --> 00:11:54.581 tokens over this weekend to build, clones. 00:11:54.982 --> 00:11:58.491 And then Anthropic shows dynamic workflows, as well. 00:11:58.852 --> 00:12:00.931 Let's see what else we have with the TLDR. 00:12:02.972 --> 00:12:03.931 Yeah, um, 00:12:04.812 --> 00:12:08.332 in big companies and APIs, Meta's first 00:12:09.052 --> 00:12:10.381 big and heavy and 00:12:11.782 --> 00:12:13.401 like frontier-ish 00:12:14.302 --> 00:12:15.772 model, or maybe frontier model. 00:12:15.872 --> 00:12:16.951 No, it's actually a frontier model. 00:12:17.351 --> 00:12:22.791 Meta Muse Wonder is number 2 on on Arena, and this is the best version, 00:12:23.351 --> 00:12:23.691 uh, 00:12:24.432 --> 00:12:24.611 and 00:12:25.512 --> 00:12:27.032 it is now available in different tools. 00:12:27.112 --> 00:12:30.351 So it's available in MetaCode and Muse only for subscribers. 00:12:30.872 --> 00:12:32.571 Uh, some builders are split about this. 00:12:33.152 --> 00:12:34.432 Also, there's 2 versions there. 00:12:34.552 --> 00:12:37.731 There's Muse Wonder and Muse Wonder Full. 00:12:38.672 --> 00:12:39.092 Um... 00:12:41.392 --> 00:12:42.532 Did you see what I did there? 00:12:42.632 --> 00:12:44.152 Muse Wonder and Muse Wonder Full? 00:12:44.192 --> 00:12:47.252 That's, I think, what they did, because everybody calls their models Max on the 00:12:47.312 --> 00:12:50.551 Max reasoning. They call it Full, and it's wonderful, and it's pretty cool. 00:12:51.032 --> 00:12:51.831 Uh, Gemini 4 00:12:52.432 --> 00:12:54.171 finally launched. Welcome, Gemini 4. 00:12:54.272 --> 00:12:57.712 Again, as a reminder, we'll have the folks from Gemini, Omar, Sanseviero, and 00:12:57.832 --> 00:13:01.741 Geneviève, joining us in about an hour to talk about Gemini 4 Argon. 00:13:01.822 --> 00:13:06.662 It's out for everyone. It's number one on Arena and visual evals, uh, and we can't 00:13:06.702 --> 00:13:09.851 wait to talk to these folks, and hear all about the use cases about Gemini. 00:13:10.412 --> 00:13:14.791 And then, uh, we talked about Anthropic opening, open telemetry traces and talking 00:13:14.812 --> 00:13:15.632 about their IPO. 00:13:17.152 --> 00:13:20.391 And Agent SDK still runs on your cloud subscription. 00:13:20.512 --> 00:13:21.831 Wolfram, hopefully you caught that. 00:13:22.352 --> 00:13:27.632 There was an update, and now there is an Agent SDK, and, um, if you guys remember 00:13:27.712 --> 00:13:28.072 the whole 00:13:29.472 --> 00:13:31.431 maybe Anthropic will block you from the 00:13:32.672 --> 00:13:36.191 subscription service if you use it for like Hermes Agent or whatever. 00:13:36.312 --> 00:13:36.491 Now 00:13:37.352 --> 00:13:40.551 we're back at the place where Agent SDK supposedly runs on your Claude 00:13:40.632 --> 00:13:40.951 subscription. 00:13:41.632 --> 00:13:41.991 Um, 00:13:42.752 --> 00:13:46.991 Anthropic also showed investors a 6- 16.8 billion dollar quarter, 00:13:48.112 --> 00:13:52.951 uh, ahead of its reported November IPO, and there's a rumor about Fable 5.5 00:13:53.152 --> 00:13:56.412 showing up in Azure, but it's unconfirmed, just a rumor. 00:13:56.952 --> 00:13:57.232 Uh, 00:13:57.952 --> 00:14:01.811 folks from TypeSafe, this was just this morning, this is like nearly breaking 00:14:01.872 --> 00:14:03.712 news, like like I think it's in the breaking news category. 00:14:04.192 --> 00:14:08.711 Uh, Jev 1.14, which can now see images. 00:14:08.832 --> 00:14:13.721 So Image Jev, we heard that it's coming, and Image Jev is now here, so it's now in 00:14:13.842 --> 00:14:14.201 preview. 00:14:14.962 --> 00:14:15.401 Um, 00:14:16.402 --> 00:14:17.971 and I think that's wonderful. I really want to try it. 00:14:18.052 --> 00:14:21.312 I didn't have time running around, but like Image GeV is something I want for, I 00:14:21.412 --> 00:14:22.291 waited for a long time. 00:14:23.092 --> 00:14:25.972 Microsoft shipped its inference chip Maia 300 00:14:26.652 --> 00:14:28.892 and its own decision model, Microsoft Decision 1. 00:14:29.172 --> 00:14:33.132 So decision models are taking over, and I'm very, very interested to see how, 00:14:33.512 --> 00:14:36.431 Decision 1 stacks up with ImageJev. 00:14:38.112 --> 00:14:39.312 Open source LLM. 00:14:39.792 --> 00:14:41.031 In open source LLM, 00:14:41.672 --> 00:14:43.291 4 open decision models 00:14:44.312 --> 00:14:48.231 was released, Amos 1, from 0.5B to 27B. 00:14:48.672 --> 00:14:52.131 Uh, they also answer with bounding boxes, so it's not only Jev. 00:14:52.272 --> 00:14:54.631 Jev answers with bounding boxes, so do these models. 00:14:55.232 --> 00:14:55.692 Uh... 00:14:57.492 --> 00:14:57.912 haven't, 00:14:58.585 --> 00:15:00.171 I haven't had the time to test them. 00:15:00.612 --> 00:15:06.592 MiniCPM returns with 4.7 and runs a 35 billion param vision model on iPhone 00:15:06.612 --> 00:15:10.711 in about 3 gigs. Uh, Liquid, shout out to our friends from Liquid AI. 00:15:10.812 --> 00:15:14.771 They released their biggest Liquid foundation model, 24 billion parameters, 00:15:15.182 --> 00:15:19.742 built for local agents, and we constantly talk about running agents locally. 00:15:20.502 --> 00:15:20.821 And then 00:15:22.342 --> 00:15:22.661 QwenGyre, 00:15:23.391 --> 00:15:26.391 RL framework for long horizon agents came out. 00:15:27.032 --> 00:15:28.191 Uh, I think it's a paper. 00:15:29.992 --> 00:15:35.332 In this week's buzz, we told you about Forge, CoreWeave Forge, the rebrand for 00:15:35.472 --> 00:15:37.811 our, agentic cloud offerings on SaaS. 00:15:38.332 --> 00:15:40.851 Forge model distillation is now in preview, 00:15:41.452 --> 00:15:41.711 and, 00:15:42.756 --> 00:15:47.035 I think that's super cool. And also Agent Lens in preview, and Agent Lens is 00:15:48.116 --> 00:15:52.836 just hot this week because now you can connect Claude, uh, via OpenTelemetry and 00:15:53.036 --> 00:15:57.475 see the most, you know, the most useful and not useful traces from your Claude, 00:15:57.876 --> 00:15:59.645 uh, run directly in Agent Lens. 00:15:59.726 --> 00:16:02.996 I think that's super cool. apparently XAI dropped Grok Voice 3. 00:16:03.716 --> 00:16:06.416 Do you guys, here see this, speech-to- speech leaderboard? 00:16:06.695 --> 00:16:10.915 It tops the speech-to-speech leaderboards, and, apparently all of the Starlink 00:16:10.976 --> 00:16:13.655 support calls are now used by Grok Voice 3. 00:16:14.296 --> 00:16:17.306 And, uh, Wolfram, you also mentioned Fish Audio, correct? 00:16:18.026 --> 00:16:18.485 Drama voice. 00:16:18.506 --> 00:16:19.385 Wolfram Ravenwolf: Correct. 00:16:19.466 --> 00:16:21.265 Alex Volkov: Just summarize for us. 00:16:23.026 --> 00:16:25.646 Wolfram Ravenwolf: Yeah, uh, it's a new, um, 00:16:26.546 --> 00:16:30.825 a model, a speech model that you can direct, where you can give instructions 00:16:30.986 --> 00:16:32.726 for how the voice should be sounding. 00:16:32.786 --> 00:16:36.865 I think this is best if we, when we talk about it, we we show some examples. 00:16:37.346 --> 00:16:38.325 Alex Volkov: Yeah, we maybe play some examples. 00:16:38.346 --> 00:16:40.825 Wolfram Ravenwolf: It's open source as well, so you can run it locally, 00:16:41.466 --> 00:16:45.625 and uh, it's also at the top of what is possible with voice right now. 00:16:46.146 --> 00:16:47.825 Alex Volkov: Oh, that's super cool. Um, 00:16:48.946 --> 00:16:49.306 all right. 00:16:50.346 --> 00:16:52.105 Let's go to industry. 00:16:52.986 --> 00:16:56.365 Industry is where we talk about stuff that are not releases, but I think is important 00:16:56.386 --> 00:16:58.646 from this week. Uh, Andrej Karpathy says 00:16:59.265 --> 00:17:02.065 6 billion people have barely touched an LLM. 00:17:02.866 --> 00:17:06.366 Uh, he he did some back of the napkin math, and 00:17:07.186 --> 00:17:11.355 don't we love a good, like, Karpathy tweet that sets the industry ablaze, and it 00:17:11.436 --> 00:17:13.555 feels like he's been tweeting less since he joined Anthropic. 00:17:13.916 --> 00:17:18.475 Uh, so he literally had this, like, funnel of saying 6 billion people barely touched 00:17:18.516 --> 00:17:23.176 an LLM and AI, and the gap, he says, is getting wider, uh, and then he kind of 00:17:23.196 --> 00:17:27.125 does calculate. So hopefully, everybody who tunes into ThursdAI, folks, you're on 00:17:27.146 --> 00:17:30.976 the top of under Karpathy's funnel, but he does have this thing where, like, only a 00:17:31.076 --> 00:17:33.795 handful of people touch the models at the Frontier Labs. 00:17:33.861 --> 00:17:37.341 and last week, State of the AI, state of AI report came out. 00:17:37.541 --> 00:17:40.631 Nathan Benaich, the author, the the guys who's been running this for a while, is 00:17:40.671 --> 00:17:46.500 gonna join us, was at 10 AM Pacific, which is going to be in an hour, to talk about 00:17:46.541 --> 00:17:46.661 it. 00:17:47.301 --> 00:17:47.500 And 00:17:48.415 --> 00:17:50.825 I'm excited. Peter, you connected us to Nathan. 00:17:51.185 --> 00:17:52.304 Uh, I'm very, very excited. 00:17:52.345 --> 00:17:55.015 He's been running this State of AI report f- for a while now. 00:17:56.150 --> 00:17:59.269 Peter Gostev: Alex, I think, uh, did you mention Nemotron? 00:17:59.669 --> 00:18:00.610 I don't think so, right? 00:18:00.950 --> 00:18:01.050 Alex Volkov: No. 00:18:01.310 --> 00:18:03.330 Peter Gostev: The, it just launched about half an hour ago. 00:18:03.710 --> 00:18:04.050 Alex Volkov: Ooh. 00:18:04.150 --> 00:18:04.370 Peter Gostev: And 00:18:04.830 --> 00:18:05.289 Alex Volkov: No, half an hour. 00:18:05.310 --> 00:18:05.889 Peter Gostev: Nemotron, 00:18:06.710 --> 00:18:08.349 yeah, Nemotron 3.5 00:18:09.230 --> 00:18:10.169 Super VL, 00:18:10.830 --> 00:18:16.389 120, uh, billion total parameters and 12 billion active parameters. 00:18:16.990 --> 00:18:17.409 Alex Volkov: No, I do not. 00:18:17.430 --> 00:18:17.789 Peter Gostev: Um, 00:18:18.669 --> 00:18:20.570 it looks like they're focusing on, 00:18:21.350 --> 00:18:26.050 uh, more, it's multimodal, so the idea is that it's good at video understanding, 00:18:26.150 --> 00:18:29.909 image understanding, and long-running agents, and roughly where they're kind of 00:18:30.070 --> 00:18:30.849 pitching it at 00:18:31.750 --> 00:18:34.990 DeepSeek V4 flash, including small, 00:18:35.990 --> 00:18:38.289 that kind of level, but it's meant to be quite fast and, 00:18:38.910 --> 00:18:42.369 um, and let's say it's performant at, at that sort of level. 00:18:42.590 --> 00:18:43.210 So, yeah. 00:18:43.230 --> 00:18:44.710 Alex Volkov: But just a new release. We haven't tested it. 00:18:44.750 --> 00:18:46.169 It just literally came out. Yeah. 00:18:46.510 --> 00:18:46.649 Peter Gostev: Yeah. 00:18:46.750 --> 00:18:47.009 Alex Volkov: Um. 00:18:47.190 --> 00:18:47.489 Peter Gostev: Perfect. 00:18:47.510 --> 00:18:51.370 Alex Volkov: So shout out to our friends from f- f- from Nemotron, Team Green, and uh, and our 00:18:51.429 --> 00:18:54.629 friend Chris Alexiuk, who works on that team and, and always joins us and talks 00:18:54.670 --> 00:18:55.910 about it. So shout out to NVIDIA. 00:18:56.269 --> 00:18:57.410 Uh, there's also rumors 00:18:58.430 --> 00:18:59.629 that NVIDIA, did you guys see this? 00:18:59.990 --> 00:19:02.329 And NVIDIA, is in talks to buy, 00:19:03.084 --> 00:19:03.643 Reflection? 00:19:04.444 --> 00:19:07.854 I think it was Reflection. Well, the lab behind Beam that we talked about Beam last 00:19:07.894 --> 00:19:09.513 week. I'm pretty sure it's Reflection. 00:19:09.534 --> 00:19:13.433 This is like, there's rumors that Jensen's collecting the Infinity Stones of open 00:19:13.894 --> 00:19:18.818 source. So Hugging Face, Reflection, also rumored, also friends of the pod, the 00:19:18.898 --> 00:19:22.218 Israeli company rumored that NVIDIA is like talking about purchasing them as 00:19:22.258 --> 00:19:27.277 well. All right, I think it's time for us to go to Frontier Labs and talk about Meta 00:19:27.338 --> 00:19:28.017 Muse Wonder. 00:19:39.338 --> 00:19:41.678 Alrighty, folks, let's talk about... 00:19:43.378 --> 00:19:46.837 do not bet against Mark Zuckerberg, and now Alex Wang. 00:19:47.018 --> 00:19:48.258 Uh, so in addition to 00:19:48.898 --> 00:19:49.277 having, 00:19:49.958 --> 00:19:50.917 great meme taste, 00:19:51.998 --> 00:19:53.717 Meta is now back at the frontier. 00:19:54.398 --> 00:19:58.767 Meta's frontier model is number 2 on Arena and, just 12 points behind Argon, which 00:19:58.788 --> 00:20:03.287 we're going to talk about. Uh, but some builders are calling these benchmarks, and 00:20:03.528 --> 00:20:05.317 I think, several things can be true. 00:20:06.198 --> 00:20:09.197 Meta Superintelligence Labs released Muse Wonder and Muse Wonder Full. 00:20:09.958 --> 00:20:12.337 Uh, Wonder Full is kind of the over-the- top 00:20:13.498 --> 00:20:15.398 model that we don't get in the API. 00:20:15.538 --> 00:20:17.778 It only exists in Meta subscription tiers. 00:20:18.898 --> 00:20:22.817 Alex Wang called it their largest and most capable model yet. 00:20:23.537 --> 00:20:25.198 Uh, I joked with somebody yesterday 00:20:25.898 --> 00:20:29.458 that all I need to say every week on the show is that this new model launch, 00:20:29.578 --> 00:20:31.867 numbers are bigger, and they claim it's their best yet. 00:20:32.388 --> 00:20:34.607 And it does, we're kind of there, right? 00:20:34.627 --> 00:20:38.047 We're kind of there where it's like the new iPhone 18 moment for every lab, and 00:20:38.468 --> 00:20:39.887 this is the best iPhone we've released yet. 00:20:40.428 --> 00:20:44.827 Uh, both Meta models, though they do smell like a big model to me. 00:20:44.948 --> 00:20:46.827 So both have 1 million token. 00:20:47.388 --> 00:20:47.747 Um, 00:20:48.707 --> 00:20:52.187 Muse Wonder via API, it's 4 bucks in and 20 00:20:52.788 --> 00:20:56.747 dollars out per million tokens, basically Opus 5.5 pricing. 00:20:57.028 --> 00:20:57.688 But if you see 00:20:58.582 --> 00:21:03.142 the evals charts, they actually show that they beat Opus on multiple interesting 00:21:03.222 --> 00:21:05.981 benchmarks, and those benchmarks are not your standard benchmarks, besides the 00:21:06.062 --> 00:21:09.571 SWE-bench one. I'm pretty sure that those benchmarks are, a little bit different. 00:21:10.132 --> 00:21:10.532 Um, 00:21:11.372 --> 00:21:12.091 Muse Wonder, 00:21:12.812 --> 00:21:15.062 in parentheses, full, is the better model. 00:21:15.142 --> 00:21:19.421 That's the one that kind of, they say, beat Opus and beat Astra and beat Fable, 00:21:19.792 --> 00:21:21.751 only for Muse code subscribers. 00:21:22.232 --> 00:21:23.631 And uh, 50 00:21:26.052 --> 00:21:27.431 or 200 dollars a month. 00:21:27.672 --> 00:21:31.132 This is the tier that you need to pay, either 50 dollars a month or 200 dollars a 00:21:31.152 --> 00:21:32.312 month to get it on Muse Code. 00:21:33.272 --> 00:21:34.872 No open weights for it either, but 00:21:35.752 --> 00:21:37.232 here's the news that excites us. 00:21:38.232 --> 00:21:42.791 Uh, Muse Spark, the model that was there before Muse Wonder, they told us about 00:21:42.912 --> 00:21:45.952 open source, and Muse- Mu- Muse Spark is basically the new Llama. 00:21:46.392 --> 00:21:49.271 They're going to, not yet, but they're going to open source Muse Spark. 00:21:49.792 --> 00:21:53.152 Uh, I hope this is the latest version, 1.3, but I'm very, very excited. 00:21:53.512 --> 00:21:54.431 Uh, Meta says, 00:21:55.234 --> 00:21:55.354 the 00:21:56.035 --> 00:21:57.594 Meta Wonder models are too big 00:21:58.315 --> 00:21:59.954 and maybe even too risky for open source. 00:22:00.115 --> 00:22:01.794 That's a, that's an interesting call out. 00:22:02.635 --> 00:22:02.995 Um, 00:22:04.375 --> 00:22:05.744 right, let's, let's just dive in there. 00:22:05.945 --> 00:22:08.225 Uh, Peter, I know you guys tested Muse Wonder. 00:22:08.465 --> 00:22:09.945 Um, would love to hear from you. 00:22:10.905 --> 00:22:16.405 Peter Gostev: Yeah, so you, you're right to say that it was second on the text arena, which means 00:22:16.425 --> 00:22:17.744 that people like talking to it. 00:22:17.945 --> 00:22:20.304 It is, I think, 12th on the 00:22:21.185 --> 00:22:23.364 agent arena, and agent arena is when 00:22:24.265 --> 00:22:29.005 all people like you and I go on arena and use it in various kinds of ways, which 00:22:29.185 --> 00:22:31.825 could include coding, could include white collar work. 00:22:32.465 --> 00:22:37.505 So that doesn't, that's not amazing, I must say, 12 is not, it's not that great. 00:22:38.185 --> 00:22:43.664 Um, wi- which is kind of interesting, like people say, or is it benchmarks or, or 00:22:44.024 --> 00:22:45.144 like Alex Wang is, 00:22:45.985 --> 00:22:50.405 uh, fighting back and saying, oh, you know, who needs, uh, 3D and all of that. 00:22:50.465 --> 00:22:50.604 But 00:22:50.745 --> 00:22:50.885 Alex Volkov: Yeah. 00:22:50.985 --> 00:22:55.784 Peter Gostev: you know, 3- 3D aside, I have tried also building websites with it, and I must say 00:22:55.945 --> 00:22:57.925 it was kind of awful, that websites. 00:22:57.985 --> 00:22:58.084 Alex Volkov: Mm. 00:22:58.105 --> 00:23:01.424 Peter Gostev: I, I don't know, maybe it was my experience, maybe I did something wrong, 00:23:02.105 --> 00:23:04.185 and I didn't spend, like, many hours with it. 00:23:04.425 --> 00:23:04.605 Alex Volkov: Yeah. 00:23:05.025 --> 00:23:05.225 Peter Gostev: But 00:23:06.025 --> 00:23:07.864 I think that's kind of an important as well. 00:23:07.905 --> 00:23:08.505 I've never seen 00:23:09.305 --> 00:23:14.505 a- a model that's really bad, but it's good at 3D and it's good at websites, but 00:23:14.585 --> 00:23:16.504 it's fundamentally bad. Like, I- I think- 00:23:17.145 --> 00:23:21.405 I- I think it's kind of unreasonable to say, well, it- it's much as Opus, but 00:23:21.445 --> 00:23:24.505 actually it's really bad at half of the things that people use it for. 00:23:24.585 --> 00:23:28.245 So it- it's kind of, I I don't know, I don't want to be unfair to them. 00:23:28.305 --> 00:23:29.864 It's clearly a better model, 00:23:30.505 --> 00:23:34.744 but equally, I think it's still behind on some things, like front end for sure. 00:23:34.865 --> 00:23:38.524 3D aside, front end is seems like really quite bad, actually. 00:23:38.825 --> 00:23:38.858 Alex Volkov: So 00:23:38.858 --> 00:23:39.805 Peter Gostev: This is the frontier. 00:23:40.165 --> 00:23:40.844 Alex Volkov: I will say, 00:23:42.525 --> 00:23:47.445 while 3D is not a use case that I expect my assistants to kind of use, and it's 00:23:47.525 --> 00:23:48.864 obvious that Meta is 00:23:49.485 --> 00:23:52.185 fine tuning Muse Wonder for the Muse Assistant, right? 00:23:52.205 --> 00:23:55.784 Like, it's very obvious to us why they have the model, and the way they gonna put 00:23:55.805 --> 00:23:59.904 this capability in people's hands is via their personal agent or AI assistant, 00:23:59.965 --> 00:24:01.204 whatever you want to call this category. 00:24:01.565 --> 00:24:03.214 And so they're optimizing f- f- for that thing. 00:24:03.238 --> 00:24:07.144 We, we should probably mention the two innovation type things that they're adding 00:24:07.205 --> 00:24:12.445 there that are not evaluated, which is, dynamic memory and dynamic context stuff. 00:24:12.645 --> 00:24:12.924 Um, 00:24:13.845 --> 00:24:15.374 however, uh, Peter, to your point, 00:24:16.335 --> 00:24:22.295 while asking my assistant to create 3D castles and 3D series or whatever Nisten 00:24:22.335 --> 00:24:26.375 created and they showed us last week, a virtual copy of Toronto with 100,000 00:24:26.495 --> 00:24:27.134 agents running, 00:24:27.775 --> 00:24:29.535 that's not what I'm gonna ask my assistant to do. 00:24:29.895 --> 00:24:34.065 Building websites is kind of what I think, many people would like their agents to 00:24:34.125 --> 00:24:34.724 help them with. 00:24:35.405 --> 00:24:38.445 Building websites, one-off websites for like a use case, 00:24:39.085 --> 00:24:41.005 or like more, more stuff as well. 00:24:41.045 --> 00:24:44.715 So I think that this is definitely in the realm of, something that people will ask 00:24:44.795 --> 00:24:48.784 their assistants to do. Wolfram, do you have any thoughts on, uh, Meta Wonder, 00:24:49.005 --> 00:24:49.665 Meta Muse Wonder? 00:24:49.885 --> 00:24:54.385 Wolfram Ravenwolf: Oh, on one hand, uh, that is exactly this what I've seen less technical people who 00:24:54.425 --> 00:24:55.605 are not as much into AI. 00:24:55.685 --> 00:24:59.924 They use AI assistants to, uh, create websites or update their websites. 00:24:59.965 --> 00:25:02.525 So that is very important to be good in this regard. 00:25:02.685 --> 00:25:07.445 3D, on the other hand, of course, it's a nice demo, but yeah, the practical use 00:25:07.485 --> 00:25:09.304 case for us most of the time is not that high. 00:25:09.685 --> 00:25:13.785 I tested Wonder, and it is excellent in German, which is not true with all the 00:25:13.845 --> 00:25:14.285 models, so 00:25:14.525 --> 00:25:14.744 Alex Volkov: Mhm. 00:25:15.045 --> 00:25:17.725 Wolfram Ravenwolf: It has the colloquialisms and stuff, so this is great. 00:25:18.245 --> 00:25:22.924 Um, yeah, it's, it's a good model, and I think that they have their own agents, uh, 00:25:23.125 --> 00:25:27.304 that they are offering personal agents means that it can also be used very well 00:25:27.345 --> 00:25:28.884 in Hermes Agent and the other ones. 00:25:29.204 --> 00:25:30.484 It's available with the API. 00:25:30.925 --> 00:25:31.045 Alex Volkov: Yeah. 00:25:31.165 --> 00:25:33.764 Wolfram Ravenwolf: Um, yeah, I think it's great to have more choice here. 00:25:35.085 --> 00:25:36.764 Alex Volkov: I think a very interesting thing is, 00:25:37.485 --> 00:25:40.845 I haven't seen the Muse code subscription being used in like Hermes Agent. 00:25:41.204 --> 00:25:44.945 I don't know if that's possible, but I think that'd be cool, because now, we're 00:25:44.965 --> 00:25:47.804 gonna get there, but the Claude SDK looks to be back up. 00:25:48.605 --> 00:25:49.044 Um. 00:25:49.245 --> 00:25:52.445 Wolfram Ravenwolf: What's always possible is to have your Hermes Agent just use another agent. 00:25:52.525 --> 00:25:56.885 I'm using all the agents like that, that Hermes on my own system is my main, and 00:25:57.045 --> 00:26:00.585 connecting to CrockPot and the other ones and just taking control of them all. 00:26:00.925 --> 00:26:02.164 Alex Volkov: Yeah. How did you set it up? 00:26:02.285 --> 00:26:03.765 Yam, you had a comment as well about Muse? 00:26:03.965 --> 00:26:06.364 Yam Peleg: Yeah, that's uh, that's a good idea, actually. 00:26:06.405 --> 00:26:08.844 I never thought about, about trying it as well. 00:26:09.405 --> 00:26:09.964 How do you, 00:26:10.725 --> 00:26:12.785 if we have time, how do you, how do you set this? 00:26:12.845 --> 00:26:13.965 Like, what do you use it for? 00:26:14.804 --> 00:26:18.364 Wolfram Ravenwolf: Oh, I've been using this for a long time, even before the agent stuff came up. 00:26:18.565 --> 00:26:19.644 I was calling it minions. 00:26:19.805 --> 00:26:25.325 Amy has her minions, and she uses computer use if there is no API or CLI tool. 00:26:25.565 --> 00:26:29.644 It's just a s- a skill, basically, a minion skill where she can use, we did it 00:26:29.685 --> 00:26:34.125 in the, in the, ah, when, when the first computer use stuff came out, she was going 00:26:34.145 --> 00:26:36.804 to the ChatGPT website and using that as a minion 00:26:37.045 --> 00:26:37.185 Yam Peleg: Yeah. 00:26:37.205 --> 00:26:40.364 Wolfram Ravenwolf: from Claude and stuff like that, and it still works the same way. 00:26:40.445 --> 00:26:45.845 You just give it a tool, like it can use a CLI tool, or it can use a website or 00:26:46.045 --> 00:26:50.604 Crocbot, the app can be controlled with computer use, and it's just another agent 00:26:50.685 --> 00:26:53.044 for a sub-agent for the main agent. 00:26:53.405 --> 00:26:54.505 So very easy to set up. 00:26:54.645 --> 00:26:58.644 Yam Peleg: Okay. I do it all the time in the terminal, uh, Claude calling Codex, Codex 00:26:58.765 --> 00:26:59.364 calling Claude. 00:26:59.405 --> 00:26:59.945 Wolfram Ravenwolf: Mm-hmm. Yeah. 00:27:00.045 --> 00:27:02.385 Yam Peleg: The variety gives, gives you a lot for 100%. 00:27:02.965 --> 00:27:04.964 I'm j- I'm just wondering because these are, 00:27:05.805 --> 00:27:09.385 these are different. They're extremely powerful, connected to many different 00:27:09.445 --> 00:27:13.784 services. So just, I don't know, just, just the use case, like what you use it 00:27:13.845 --> 00:27:14.265 for. I mean, 00:27:14.405 --> 00:27:14.785 Alex Volkov: I think, yeah. 00:27:14.805 --> 00:27:15.648 Yam Peleg: It can be interesting. Okay, 00:27:15.745 --> 00:27:15.925 Wolfram Ravenwolf: Just 00:27:16.705 --> 00:27:21.804 one use case is a council like GPT Pro, which you can't just use with the API and 00:27:21.825 --> 00:27:23.545 so on. So we are using it that way. 00:27:23.625 --> 00:27:24.145 If we are 00:27:24.785 --> 00:27:27.645 creating a concept for something, it goes through all these. 00:27:27.665 --> 00:27:33.645 I have a whole council of 12, 13 agents that all, uh, work together to 00:27:33.785 --> 00:27:38.065 make sure that this is, uh, if it's very important, to be the best one we can make. 00:27:38.705 --> 00:27:38.844 Yam Peleg: Cool. 00:27:40.145 --> 00:27:42.065 Alex Volkov: We definitely should talk about use cases, though. 00:27:42.145 --> 00:27:45.945 So we talk about models and numbers, et cetera, but I think, as we started to say 00:27:45.985 --> 00:27:48.964 in the beginning, we're approaching the level of capability where, like, for most 00:27:49.005 --> 00:27:53.355 of us, um, the stuff that we need to do, we can probably already do. 00:27:53.715 --> 00:27:53.934 So, 00:27:55.401 --> 00:27:55.622 Go ahead. 00:27:55.642 --> 00:28:00.221 Peter Gostev: Maybe one, one, one thing to add as well, I think, about the release, and I imagine 00:28:00.281 --> 00:28:03.621 they probably just don't have enough compute, but it so- sounds like at least 00:28:03.802 --> 00:28:04.342 from, like, 00:28:04.602 --> 00:28:05.601 Alex Volkov: Theo, you know, the- 00:28:06.842 --> 00:28:10.962 Peter Gostev: I think they don't. Yeah, it's like the- yeah, I don't think so, because why, why 00:28:11.062 --> 00:28:13.802 is the full one not available via API? 00:28:14.162 --> 00:28:16.021 Like, I don't think they're just being stingy, right? 00:28:16.122 --> 00:28:17.401 Maybe they want to cross-sell the 00:28:18.202 --> 00:28:22.001 subscriptions, but that feels, I don't think that's working that well. 00:28:22.442 --> 00:28:28.421 So, for example, I think Theo, right, the the YouTuber, he, uh, he was 00:28:28.522 --> 00:28:33.281 very complimentary about the Muse model, which I was confused by, and then he came 00:28:33.322 --> 00:28:35.802 out and said, well, actually the the full one is great. 00:28:35.962 --> 00:28:38.162 The regular one that we got, he thought was bad. 00:28:38.562 --> 00:28:38.701 Alex Volkov: Mm. 00:28:38.722 --> 00:28:43.241 Peter Gostev: So I I haven't tested the full one, so my impression is very like meh about the 00:28:43.322 --> 00:28:43.922 model, but 00:28:44.762 --> 00:28:47.722 maybe the full one is amazing, but we don't have access to it. 00:28:47.762 --> 00:28:52.441 So I, I, I wonder if there's also that kind of mismatch that they think, oh, 00:28:52.482 --> 00:28:56.761 their model is amazing, then the reception is like, oh, is it benchmarks or what? 00:28:56.962 --> 00:29:00.081 But actually, maybe it is amazing, but they, they mean a different model. 00:29:00.362 --> 00:29:01.122 It's not ideal. 00:29:01.722 --> 00:29:06.222 Alex Volkov: This is kind of on the same topic of, folks, let's say inside Anthropic, they're 00:29:06.242 --> 00:29:08.811 probably not using Opus, they're probably using like t- Opus 00:29:09.572 --> 00:29:11.012 plus plus, whatever model. 00:29:11.052 --> 00:29:14.591 And so when they talk about the stuff that works for them in the public, I'm not sure 00:29:14.652 --> 00:29:16.392 that they mean kind of the models that are available to us. 00:29:16.492 --> 00:29:17.831 That's a very interesting, very interesting point. 00:29:17.852 --> 00:29:21.402 And I think Theo also gets early access to like the models that are not available, in 00:29:21.442 --> 00:29:26.641 the API. Um, you can get Muse Wonder, not Wonderful, Muse Wonder within the product. 00:29:26.682 --> 00:29:30.461 So I gotta wonder if any of you guys used it within the product itself. 00:29:31.582 --> 00:29:32.982 Because I switched immediately, so 00:29:33.742 --> 00:29:38.802 I, I've been using Muse, as you guys know, because I've talked about this a lot, and 00:29:38.942 --> 00:29:42.582 I will continue talking about this because I think Muse is wonderful, not the model. 00:29:43.032 --> 00:29:46.591 I, when I realized that I can only get access to this new intelligence, 00:29:47.311 --> 00:29:50.071 essentially what happens is, Wolfram, you know all about this, you've been doing 00:29:50.092 --> 00:29:54.111 this for Amy for a while. Essentially, I realized that I can upgrade my assistant's 00:29:54.192 --> 00:29:55.112 intelligence with a click. 00:29:56.472 --> 00:29:57.551 And that is incredible to me. 00:29:57.952 --> 00:30:02.931 That is just like, um, this is the scene from The Matrix where Neo gets a thing, 00:30:03.032 --> 00:30:05.192 and now he's like, boom, and I know kung fu. 00:30:05.632 --> 00:30:08.992 So skills is kind of like that, but also like upgraded intelligence is kind of like 00:30:09.032 --> 00:30:10.551 that. And so I upgraded, 00:30:11.735 --> 00:30:14.036 muse, uh, Wolfred, my muse, to 00:30:14.916 --> 00:30:16.876 Wonder, and I gave it a lot of tasks. 00:30:17.076 --> 00:30:19.275 Here's use cases time. Let's talk about use cases a little bit. 00:30:19.596 --> 00:30:23.605 Uh, the podcast studio that I'm in, shout out, Virgo and Ezra, for helping me set 00:30:23.625 --> 00:30:25.886 this up super quick. Muse found this out. 00:30:26.026 --> 00:30:28.715 Muse emailed Ezra a couple of times, including the the notes and the 00:30:28.756 --> 00:30:30.076 requirements I need for the podcast. 00:30:30.436 --> 00:30:32.276 On the way here, Muse got me coffee. 00:30:32.676 --> 00:30:33.635 I did a, 00:30:34.276 --> 00:30:39.325 meet up yesterday, liquid cooling IRL, AI engineers in sauna, and you guys can see 00:30:39.346 --> 00:30:43.156 this wristband, Muse designed and ordered it and it shipped to the hotel. 00:30:43.796 --> 00:30:45.136 And all of this was on Muse Wonder. 00:30:45.176 --> 00:30:48.336 I was just kept like shoving tasks to it, and it was executing them 00:30:49.836 --> 00:30:54.129 mostly perfectly. The failure modes of this model, Peter, it's really hard to 00:30:54.189 --> 00:30:57.608 benchmark, so we need to talk about how we benchmark those like use cases, but the 00:30:57.669 --> 00:30:59.669 failure modes of this model were not the 00:31:00.309 --> 00:31:03.388 logic part. The failure modes of this model were the website doesn't work, the 00:31:03.428 --> 00:31:05.749 website is black and last because Cloudflare doesn't like agents, 00:31:06.509 --> 00:31:09.449 or, the Stripe link doesn't quite connect or something. 00:31:09.689 --> 00:31:12.828 So the failure were real world failures, including a very funny ones 00:31:13.639 --> 00:31:17.289 where, Muse called some businesses and businesses hung up because they don't want 00:31:17.328 --> 00:31:19.288 to talk to a personal assistant. 00:31:19.549 --> 00:31:20.909 It was really funny. So, 00:31:22.229 --> 00:31:22.748 I think, 00:31:24.349 --> 00:31:26.869 again, I think we're approaching a level of capability where most people 00:31:27.509 --> 00:31:29.649 do not care, um, and 00:31:30.869 --> 00:31:32.228 as long as it does stuff for them. 00:31:32.349 --> 00:31:34.468 And then the question is, does it do stuff for them? 00:31:34.509 --> 00:31:38.748 So I think when Meta released the models, they had some evals in there. 00:31:39.109 --> 00:31:40.008 I think we should pull them up 00:31:40.629 --> 00:31:42.489 that are not the standard ones. 00:31:42.669 --> 00:31:44.228 There are some, like, personal assistant ones. 00:31:44.509 --> 00:31:47.788 And Peter, the question back to you is, you are saying on the agent arena, 00:31:48.709 --> 00:31:49.969 Muse Wonder is kind of suffering. 00:31:50.108 --> 00:31:50.869 I gotta wonder if 00:31:51.549 --> 00:31:54.908 my experience with like the personal agent stuff versus the agent arena, 00:31:55.749 --> 00:31:57.168 where are those tasks falling? 00:31:57.229 --> 00:31:58.748 Where's the daylight between those tasks? 00:31:59.989 --> 00:32:02.428 Peter Gostev: Yeah, it's, it's definitely hard to say, and I think, 00:32:04.269 --> 00:32:07.828 yeah, it, it could, the, the kind of failures that you're talking about, they 00:32:07.869 --> 00:32:09.109 do sound more like the 00:32:09.949 --> 00:32:12.088 harness in a, in a broader sense, like 00:32:12.149 --> 00:32:12.368 Alex Volkov: Mhm. 00:32:12.389 --> 00:32:14.588 Peter Gostev: connections, integrations, like things like that. 00:32:14.669 --> 00:32:16.989 So, yeah, it's definitely, it's definitely hard to say. 00:32:17.509 --> 00:32:20.429 One thing to say, though, about the, the benchmarks, and you mentioned the 00:32:20.549 --> 00:32:21.189 different ones, 00:32:21.828 --> 00:32:22.148 um, 00:32:22.869 --> 00:32:24.848 this is kind of a pattern, which is a funny one. 00:32:24.909 --> 00:32:29.528 I, I did a bit of analysis on this, maybe I should, I should publish it there, where 00:32:29.589 --> 00:32:33.489 I kind of pulled together the different benchmarks that labs were using over the 00:32:33.549 --> 00:32:35.709 last year, just all of the releases. 00:32:36.389 --> 00:32:37.348 And the half 00:32:38.149 --> 00:32:43.348 life of a benchmark that they quote is, is, uh, basically like one release, 00:32:43.949 --> 00:32:47.568 uh, where so many benchmarks just get quoted once 'cause they get a good number 00:32:47.669 --> 00:32:49.109 and then they're never seen again. 00:32:49.469 --> 00:32:53.868 So I wonder it, uh, I'm not accusing them of, of this in this case. 00:32:53.989 --> 00:32:57.728 Maybe they're meaningful benchmarks, I, I don't know, but that's also something to 00:32:57.829 --> 00:33:01.369 look out for, is that quite often I would open a release and I'm looking at it, I 00:33:01.429 --> 00:33:04.289 don't recognize even the titles of half of the benchmarks. 00:33:04.509 --> 00:33:04.648 Alex Volkov: Yeah. 00:33:04.709 --> 00:33:08.349 Peter Gostev: And, you know, I- I'm in the industry, so I- I don't really know how, 00:33:09.269 --> 00:33:13.508 how reliable, how are they gonna stick to those benchmarks or- or not. 00:33:13.789 --> 00:33:18.048 But if they're inventing new ones or kind of citing new ones because it has a good 00:33:18.149 --> 00:33:21.868 score, I think that that's also not- not necessarily indicative of the performance. 00:33:22.589 --> 00:33:25.338 Alex Volkov: Uh, that's true, and I- I think, to some extent this makes sense because 00:33:25.599 --> 00:33:28.859 capability, let me, try to find the, the exact thing you're talking about, but I 00:33:28.879 --> 00:33:32.758 think the capability moves and different benchmarks need, need to switch as well, 00:33:32.799 --> 00:33:34.999 right? Like, we've we've seen that OpenAI decides not to do 00:33:35.679 --> 00:33:39.709 SWE-bench anymore because SWE-bench was like leaked in full, and then SWE-bench, 00:33:40.209 --> 00:33:43.629 Verified came out as SWE-bench Pro and like a bunch of like, updates to this 00:33:43.749 --> 00:33:47.368 paradigm. Um, but I definitely want to pull up some of them so we can talk about 00:33:47.389 --> 00:33:48.148 this meta... 00:33:49.949 --> 00:33:53.559 Wonder. Nisten, meanwhile, any comments on Muse Wonder from your end? 00:33:55.599 --> 00:33:57.139 Nisten Tahiraj: No, it's is Wonder a new thing? 00:33:57.238 --> 00:33:58.038 I'm lo- I 00:33:58.639 --> 00:34:01.539 I have Muse running in circles all night. 00:34:01.599 --> 00:34:02.238 I don't know what it 00:34:02.839 --> 00:34:04.599 does. It tests websites and stuff. 00:34:05.039 --> 00:34:08.219 It's pretty dumb sometimes, but it's just so energetic. 00:34:08.238 --> 00:34:09.838 Alex Volkov: Nisten, you don't know what Muse Wonder is? 00:34:10.238 --> 00:34:11.172 Nisten Tahiraj: No idea. Just 00:34:11.219 --> 00:34:14.349 Alex Volkov: I love how, how co-hosts of ThursdAI sometimes joins as co-hosts with deep 00:34:15.209 --> 00:34:17.809 insights and knowledge and building things in the engineering, and sometimes they 00:34:17.849 --> 00:34:20.988 join as guests and just like chill here and like get their news from the live 00:34:21.009 --> 00:34:21.148 show. 00:34:21.809 --> 00:34:24.188 This is incredible. Muse Wonder is their new model. 00:34:24.289 --> 00:34:26.708 This is their Opus that beats Opus on multiple things. 00:34:26.849 --> 00:34:29.658 So I think, it's time for us to show some numbers. 00:34:29.712 --> 00:34:29.714 see. 00:34:29.734 --> 00:34:31.854 Nisten Tahiraj: Oh wow, holy cow. How does that compare to to Argon? 00:34:31.974 --> 00:34:32.693 Holy shoot. 00:34:32.814 --> 00:34:34.003 Alex Volkov: Yeah, so so this is the thing. 00:34:34.344 --> 00:34:37.004 They did not, they did not do any Argon comparisons. 00:34:37.143 --> 00:34:43.053 Uh, the Muse Wonder full and Muse Wonder on Max, are compared only to Claude Fable 00:34:43.254 --> 00:34:46.334 5.1, Astra 6, and Claude Opus 5.5. 00:34:46.694 --> 00:34:49.113 And, I think, Peter, maybe this is the one that you called out. 00:34:49.514 --> 00:34:52.194 They have a new benchmark called Personal Superintelligence Index. 00:34:52.754 --> 00:34:55.473 Internal Composite Index of Personal and knowledge work. 00:34:55.514 --> 00:34:58.854 So basically doing kind of like a artificial analysis, ish thing, 00:34:59.054 --> 00:35:00.093 capability-ish thing. 00:35:00.453 --> 00:35:02.334 I'm not quite sure how that works out. 00:35:02.574 --> 00:35:03.654 And on that index, 00:35:04.374 --> 00:35:09.253 Muse Wonder, the one that we don't get access to, on Muse Wonder Full is 59, and 00:35:09.354 --> 00:35:10.594 Muse Wonder is 54, 00:35:11.314 --> 00:35:14.354 beating pretty much everybody besides Opus 5.5 Max. 00:35:15.154 --> 00:35:15.473 Um, 00:35:17.474 --> 00:35:19.893 which other ones here, Peter, would you say? 00:35:19.914 --> 00:35:22.194 Nisten Tahiraj: This is crazy. Where is, uh, like- 00:35:22.514 --> 00:35:24.214 Alex Volkov: Wait, hold on, Nisten, just one second. 00:35:24.294 --> 00:35:28.303 Peter, which ones of these would you say are one times or benchmark or, or new 00:35:28.364 --> 00:35:28.643 ones? 00:35:29.444 --> 00:35:31.003 Peter Gostev: Well, like, if you scroll down 00:35:31.604 --> 00:35:33.323 a little bit, so we've got GDP 00:35:34.124 --> 00:35:36.063 PDF, Job Bench. 00:35:36.284 --> 00:35:38.803 Alex Volkov: Yeah, GDP PDF I have never seen before. 00:35:39.284 --> 00:35:40.683 Job Bench I have never seen before. 00:35:40.724 --> 00:35:41.263 I think we're missing. 00:35:41.284 --> 00:35:41.923 Peter Gostev: Memory Bench. 00:35:42.524 --> 00:35:45.443 Alex Volkov: Missing LDJ here, he knows a lot about some of these. 00:35:45.724 --> 00:35:49.964 Uh, I will say on the artificial analysis index, which I think a new one is coming, 00:35:50.957 --> 00:35:53.876 this model does not beat Astra and Fable on that specific index. 00:35:53.977 --> 00:35:57.476 But also that index, again, it's testing generality of models, and I think coding 00:35:57.537 --> 00:35:59.416 agents and not necessarily personal assistants. 00:36:00.057 --> 00:36:00.376 Um, 00:36:02.337 --> 00:36:04.437 OS World, for example, we know and love, right? 00:36:04.537 --> 00:36:08.626 OS World is a benchmark that measures computer use and and browser use, and on 00:36:08.767 --> 00:36:09.686 this benchmark, 00:36:10.407 --> 00:36:11.487 uh, let's see, 00:36:12.447 --> 00:36:14.487 Muse Wonder, the one that we do have access to, 00:36:15.157 --> 00:36:16.396 matches Fable 00:36:17.197 --> 00:36:19.236 and is significantly below 00:36:19.957 --> 00:36:24.137 Opus 5.5 on computer use and even below, Claude. 00:36:24.377 --> 00:36:26.577 So computer use, this model is not the best. 00:36:27.577 --> 00:36:27.896 Um, 00:36:28.600 --> 00:36:31.616 w- which, okay, so going back to what assistants need, 00:36:32.217 --> 00:36:34.377 computer is like the number one thing that I need for my assistant. 00:36:34.497 --> 00:36:37.577 It needs to go and click the right buttons, otherwise it's not useful to me. 00:36:38.057 --> 00:36:38.417 Um, 00:36:39.197 --> 00:36:40.846 a very interesting set of releases here. 00:36:42.007 --> 00:36:43.766 And uh, Nisten, you mentioned Argon. 00:36:43.807 --> 00:36:48.016 We're going to talk with folks from uh, Gemini about Argon in about an hour. 00:36:48.097 --> 00:36:52.337 We have Omer back and Geneviève, so we're gonna talk about that with them. 00:36:54.417 --> 00:36:57.876 I think we're good on, uh, uh, the news coverage or anything else left there? 00:36:58.277 --> 00:36:58.836 Wolfram, go ahead. 00:36:59.837 --> 00:37:03.516 Wolfram Ravenwolf: Uh, yeah, since we were looking at the benchmarks, I, as I'm creating Wolfbench 00:37:03.557 --> 00:37:07.536 2.0, I, I have a crisis moment, and I realized something. 00:37:07.837 --> 00:37:11.337 It's so hard to benchmark agents because they are complete systems. 00:37:11.397 --> 00:37:12.436 They are not just the model. 00:37:12.797 --> 00:37:16.256 You have the skills and memories and all this, and everybody using an agent has a 00:37:16.337 --> 00:37:19.117 different one, even if it's the same system, because of all the different 00:37:19.397 --> 00:37:23.816 personalizations, connectors, and so on, and even the internet access and the 00:37:24.117 --> 00:37:25.997 search providers, they all affect the results. 00:37:26.477 --> 00:37:30.737 And the thing is, nowadays we are at a, we are now at a point where you can have your 00:37:30.837 --> 00:37:35.397 agent create a benchmark for your own purposes and use the same system. 00:37:35.477 --> 00:37:39.737 I'm doing it with Hermes, so I can put a Hermes in the sandbox with my, my soul and 00:37:39.837 --> 00:37:44.797 skill and all and so on, and use a model and do some tests specifically for the 00:37:44.997 --> 00:37:47.677 users I have, and I think everybody can now do this. 00:37:48.317 --> 00:37:49.436 Just have your agent build it. 00:37:49.517 --> 00:37:51.517 You can build all the software in the world now. 00:37:52.037 --> 00:37:53.476 You can also build an evaluation. 00:37:53.677 --> 00:37:57.636 I mean, as someone building evaluations professionally, that is a strange moment, 00:37:57.997 --> 00:38:01.736 but it is really here now where you can have a customized evaluation for your own 00:38:01.877 --> 00:38:02.357 purposes 00:38:03.196 --> 00:38:06.276 with the agents you are using and test new models, test new agents. 00:38:06.397 --> 00:38:09.536 All of this is now possible, which wasn't possible just 00:38:09.597 --> 00:38:10.176 Alex Volkov: There was. 00:38:10.196 --> 00:38:11.517 Wolfram Ravenwolf: It was a lot of work now. 00:38:12.037 --> 00:38:12.676 Now it's possible. 00:38:13.317 --> 00:38:17.216 Alex Volkov: There was a talk at AIE that calls out this paradigm where you're like, you talk 00:38:17.237 --> 00:38:17.536 to your 00:38:18.407 --> 00:38:21.686 slop generator. You're like, hey, how can I know that you're right? 00:38:22.007 --> 00:38:24.846 And the slop generator says, well, I can do an eval for you. 00:38:25.287 --> 00:38:27.186 Uh, here's a few things to test myself on. 00:38:27.247 --> 00:38:28.566 And then you say, okay, do it yourself. 00:38:28.647 --> 00:38:32.526 And so the slop generator generates the evaluations for itself, and they're also 00:38:32.607 --> 00:38:32.927 slop. 00:38:33.807 --> 00:38:36.407 And where are we if nobody reads those traces? 00:38:36.527 --> 00:38:39.047 Uh, we're talk- we're gonna talk about reading traces in a bit, in the Claude 00:38:39.087 --> 00:38:41.556 section. Um, so this is Muse 00:38:42.337 --> 00:38:46.417 Wonder, and the call out there for our open source folks is that the Muse Spark 00:38:46.457 --> 00:38:47.476 model is going to get open source. 00:38:47.537 --> 00:38:49.937 This is a new Llama, and I think that's one, that's great. 00:38:50.777 --> 00:38:53.477 I need to start using the word wonderful because it's confusing, but I think that's 00:38:53.537 --> 00:38:53.736 great. 00:38:54.617 --> 00:38:56.497 Uh, right, let's talk about Anthropic. 00:38:56.657 --> 00:38:58.377 Anthropic is big in the news this week. 00:38:59.577 --> 00:39:01.176 Anthropic gave us 4 stories this week. 00:39:01.337 --> 00:39:06.016 You can trace your agent now, which is a very big one for, at least the sponsor of 00:39:06.057 --> 00:39:11.416 the show. Uh, you can now, or I guess again, we're so back, you can run the 00:39:11.477 --> 00:39:13.036 agent SDK on your subscription. 00:39:13.677 --> 00:39:15.697 So this means that you can plug it into Hermes. 00:39:15.757 --> 00:39:20.377 I see Wolfram shaking, he said, uh, uh, very strongly because, folks, I have told 00:39:20.397 --> 00:39:20.997 you before, 00:39:21.637 --> 00:39:24.317 so all of this personal agent type revolution, 00:39:25.327 --> 00:39:29.697 what I, hold on, so what I mostly need from this is that personal agent ability 00:39:29.857 --> 00:39:31.777 with Opus 5.5 capability. 00:39:32.537 --> 00:39:37.456 And now it seems that while I was able to pay API prices for this before, now I 00:39:37.497 --> 00:39:39.256 could do this with my, uh, Claude subscription. 00:39:39.317 --> 00:39:41.376 So I'm definitely excited to try this out. 00:39:41.476 --> 00:39:44.470 I've seen this in multiple coding harnesses and agents. 00:39:44.950 --> 00:39:46.229 Uh, what else do we have? 00:39:46.710 --> 00:39:51.869 Uh, Anthropic also showed investors a 16.8 billion dollar quarter, 00:39:53.054 --> 00:39:53.574 and also 00:39:54.214 --> 00:39:59.404 rumored Fable 5.5, uh, in Azure configs, which is a very interesting thing because 00:39:59.564 --> 00:40:01.104 Opus 5.5 is kind of 00:40:02.404 --> 00:40:05.182 beating Fable 5.1 for most stuff. 00:40:05.324 --> 00:40:07.323 They're not like super, super, strong. 00:40:08.284 --> 00:40:09.924 So let's talk about some of the Anthropic stuff. 00:40:10.924 --> 00:40:11.703 How is this lab? 00:40:11.724 --> 00:40:12.043 Wolfram Ravenwolf: One more. 00:40:12.564 --> 00:40:13.123 Alex Volkov: Yeah, one more. 00:40:13.524 --> 00:40:14.884 Wolfram Ravenwolf: One more. There's even one more. 00:40:14.924 --> 00:40:18.983 They posted a, an article about strengthening their defenses against 00:40:19.084 --> 00:40:20.124 distillation attacks. 00:40:20.484 --> 00:40:25.203 So after clamoring for, ah, we have to regulate and ban this stuff, uh, now they 00:40:25.444 --> 00:40:27.124 basically did it by themselves, 00:40:27.964 --> 00:40:33.023 increasing their defense. They, they reduced the amount of distillation that 00:40:33.244 --> 00:40:39.083 happened to the tracing, reasoning traces and so on by 90 to 99%. 00:40:39.523 --> 00:40:44.583 So it will be much harder to distill Claude reasoning now, and we'll see what 00:40:44.724 --> 00:40:46.083 effect that has on other models. 00:40:46.604 --> 00:40:48.003 Alex Volkov: I know the effect that it has on me. 00:40:48.724 --> 00:40:52.884 There's not a day goes by that my Claudes are disconnected, I need to re-log in 00:40:52.924 --> 00:40:57.213 again, and then the remote control thing fails, and so from New York I can control 00:40:57.254 --> 00:41:00.373 my Mac. I don't know if this is the effect of their security on the distillation, but 00:41:00.414 --> 00:41:02.713 it affects me personally. I don't want to call this out. 00:41:02.914 --> 00:41:04.073 Anthropic, please, 00:41:05.154 --> 00:41:07.134 please let me use my Claudes wherever I am. 00:41:07.794 --> 00:41:09.593 This is getting old. This is really getting old. 00:41:09.834 --> 00:41:10.914 All right, talk about this. 00:41:11.034 --> 00:41:11.353 Uh, 00:41:12.594 --> 00:41:13.873 let's talk about OpenTelemetry, 00:41:14.834 --> 00:41:17.074 which is kind of a nerdy subject, but it's an important subject. 00:41:17.154 --> 00:41:22.194 OpenTelemetry is this open standard for how, systems communicate their traces to 00:41:22.253 --> 00:41:23.454 each other, basically the logs. 00:41:23.574 --> 00:41:27.303 OpenTelemetry has been around for a long, long while, and then folks like, the We 00:41:27.424 --> 00:41:30.664 folks from Weights and Biases, and now the Agent Lens folks, which we're now 00:41:30.784 --> 00:41:33.378 launching in, in deploy preview. 00:41:34.058 --> 00:41:36.258 This is how systems communicate with those platforms. 00:41:36.458 --> 00:41:37.998 Um, and now Claude 00:41:38.618 --> 00:41:42.417 itself, the Claude harness is now supporting OpenTelemetry. 00:41:42.818 --> 00:41:47.178 What I'm saying is, where before you had to run something like a 1B Hive mind, 00:41:47.298 --> 00:41:50.468 which we told you about, that shows you all, all the Claude, conversations. 00:41:50.848 --> 00:41:55.048 Now Claude itself can send its traces to whatever tool of choice that you have. 00:41:55.448 --> 00:42:00.068 Obviously, on this pod, we strongly, we strongly ship, the CoreWeave Agent Lens, 00:42:00.268 --> 00:42:04.067 product we just launched, but any anything that supports OpenTelemetry, Claude code 00:42:04.108 --> 00:42:06.188 now supports. I think it's incredible, 00:42:07.228 --> 00:42:11.788 and I think it's very interesting how what you just said, Wolfram, where they try to 00:42:11.868 --> 00:42:15.957 prevent distillation while also simultaneously allowing Claude, Claude to 00:42:15.998 --> 00:42:18.088 send his traces in uh OpenTelemetry. 00:42:18.768 --> 00:42:20.307 Though this was never a problem. 00:42:20.408 --> 00:42:21.767 Claude was always writing the 00:42:22.648 --> 00:42:24.047 conversations to local, right? 00:42:24.088 --> 00:42:26.008 So you always able to get it on disk. 00:42:26.968 --> 00:42:28.347 They were not, they're not encrypted. 00:42:28.368 --> 00:42:31.447 The reasoning traces are encrypted, but the conversations themselves aren't. 00:42:31.847 --> 00:42:33.257 Uh, but I think a, it's a cool tool. 00:42:33.458 --> 00:42:35.717 You can, send your traces to observability stacks. 00:42:36.398 --> 00:42:40.208 Uh, and this includes everything, agent turns and tool calls and MCP calls and 00:42:40.448 --> 00:42:44.247 latency and token usage, so you can, like, properly measure how much Claude is 00:42:44.328 --> 00:42:47.047 costing you. Uh, and if you're a company, you can do this at scale. 00:42:49.808 --> 00:42:51.408 Oh, there's one stat here from Ramp. 00:42:51.528 --> 00:42:56.767 They said this helped them to cut token use to 10 to 20% on some agents because 00:42:56.848 --> 00:42:59.448 they learned that some of their agents are running through loops. 00:42:59.728 --> 00:43:03.007 Will you guys be, are you guys following your agents in any way? 00:43:03.128 --> 00:43:04.647 Are you evaluating them based on this way? 00:43:04.728 --> 00:43:06.608 Will you be using this from Claude Code? 00:43:08.448 --> 00:43:10.067 Nisten and Yam both shook heads. 00:43:10.688 --> 00:43:11.468 Nisten Tahiraj: Definitely gonna try. 00:43:11.488 --> 00:43:12.247 Alex Volkov: Somebody pick it up. 00:43:13.808 --> 00:43:15.587 Nisten Tahiraj: Yeah, this sounds good, especially for 00:43:16.248 --> 00:43:18.408 stopping loops and getting over all 00:43:19.328 --> 00:43:22.747 agent stuff. I- I have noticed that, uh, Claude Code does, 00:43:23.728 --> 00:43:27.468 does tend to stop often if you have anything to do with the reasoning, now it 00:43:27.488 --> 00:43:30.048 just like triggers it as a, as a, as a word. 00:43:30.448 --> 00:43:34.428 Like, if you're working with any type of LLM, like I've been working on WebGPU, the 00:43:34.488 --> 00:43:37.947 moment I even like mention anything about reasoning, now, now it just triggers it 00:43:38.008 --> 00:43:38.408 right away, 00:43:39.048 --> 00:43:41.608 uh, which is like for a small model, but yeah. 00:43:43.408 --> 00:43:43.807 Alex Volkov: Um. 00:43:44.168 --> 00:43:47.087 Nisten Tahiraj: Still good, though. It's, you can't beat Opus, so. 00:43:48.568 --> 00:43:53.167 Alex Volkov: Opus inside the Claude Code harness, I think it's still the best tool 00:43:54.087 --> 00:43:56.568 for overall tasks and doing some stuff. 00:43:56.768 --> 00:43:58.887 Uh, Opus also helped prepare the show. 00:43:59.128 --> 00:44:00.168 Nisten Tahiraj: Just Opus Low. 00:44:00.368 --> 00:44:02.827 If you need to do something, just try 00:44:03.527 --> 00:44:05.367 all in Opus Low, guys. 00:44:06.008 --> 00:44:07.527 It is just straight up vibing. 00:44:08.087 --> 00:44:12.047 Alex Volkov: Um, so I think, and because of this reason, Anthropic is also going to do an 00:44:12.148 --> 00:44:13.107 explosive IPO. 00:44:13.708 --> 00:44:19.568 Um, they reported an insane quarter, uh, 16.8 billion dollars, about 00:44:19.708 --> 00:44:22.628 45% from the second quarter increase. 00:44:23.867 --> 00:44:27.708 Adjusted profitable to the second quarter in a row and cash flow positive. 00:44:28.588 --> 00:44:33.748 Anthropic is profitable and cash flow positive and gets billions of dollars and 00:44:33.908 --> 00:44:34.227 going 00:44:35.068 --> 00:44:35.748 45% 00:44:36.548 --> 00:44:38.347 from the previous quarter still. 00:44:39.308 --> 00:44:40.807 it's absolutely insane. 00:44:40.998 --> 00:44:43.977 Nisten Tahiraj: I think they're- they have way better opinionated, 00:44:44.598 --> 00:44:44.938 uh, 00:44:45.798 --> 00:44:48.317 coding than the other models. 00:44:48.558 --> 00:44:52.078 I think people are impressed more like by the, the agentic and completeness ability 00:44:52.118 --> 00:44:56.077 of Astra, but like the actual quality of the code, it's, it's not that good. 00:44:57.078 --> 00:45:00.958 I mean, it's pretty good, uh, but it, it's not as good as Opus. 00:45:01.038 --> 00:45:04.297 And when it comes to DevOps, even Opus is bad, and Astra is like even 00:45:04.958 --> 00:45:09.537 below that. Yeah, I, I think they still have a lot of tricks, and it's mainly 00:45:09.678 --> 00:45:10.917 because of Haiku, honestly. 00:45:11.558 --> 00:45:14.037 It's- Haiku is like the most disruptive, uh, 00:45:14.638 --> 00:45:15.857 model I think they've ever 00:45:16.558 --> 00:45:21.678 released at that price, and they even give you 200 dollars in credits now to just use 00:45:22.478 --> 00:45:26.357 throughout apps. That's not gonna run out at 1 cent at 100K. 00:45:26.398 --> 00:45:30.118 Like, whatever app you're gonna serve to people, if you just keep it under 100K 00:45:30.278 --> 00:45:34.657 context for Haiku, which most conversational UX stuff is, is good 00:45:34.678 --> 00:45:35.037 enough. 00:45:36.118 --> 00:45:40.138 That is more than enough, and on the max plan, that 200 dollars, that's probably 00:45:40.198 --> 00:45:42.557 gonna support like a thousand users pretty easily. 00:45:42.958 --> 00:45:43.097 Alex Volkov: Yeah. 00:45:43.438 --> 00:45:46.278 Nisten Tahiraj: 10 cents in, 1 cent cash, that's, 00:45:47.198 --> 00:45:49.777 uh, yeah, it's just, it's just way too cheap. 00:45:49.907 --> 00:45:51.248 Alex Volkov: so let's talk about the subscription saga. 00:45:51.308 --> 00:45:52.548 Wolfram, I'm gonna put you up there, 00:45:53.751 --> 00:45:58.192 As we said, we would love to use personal agents with the Opus 5.5 level. 00:45:58.312 --> 00:45:59.431 Now it seems like it's possible. 00:45:59.565 --> 00:46:00.622 could you cover that a little bit? 00:46:00.682 --> 00:46:04.162 Did you see this? The Claude SDK and the slash P stuff. 00:46:05.202 --> 00:46:08.281 Wolfram Ravenwolf: That's why I was waiting for the new Hermes update, where this is now possible 00:46:08.321 --> 00:46:08.842 to do. So 00:46:09.602 --> 00:46:14.642 I cancelled my subscription with Opus when, or with uh, Anthropic, when uh, you 00:46:14.802 --> 00:46:17.381 weren't able to use it without getting banned. 00:46:17.522 --> 00:46:21.921 Some people apparently got banned for using it, and uh, so that was when I 00:46:22.042 --> 00:46:25.681 decided I don't want the subscription if I can't use it with the agent I'm using. 00:46:26.202 --> 00:46:30.022 I pay for this, and if it's Claude Code or if it's Hermes, it shouldn't matter, in my 00:46:30.082 --> 00:46:34.661 opinion. And now it's possible to use your personal agent, be it Hermes, OpenClaw, 00:46:34.802 --> 00:46:40.502 whatever you use, and connect the SDK, the agent SDK, and use Opus as the 00:46:40.602 --> 00:46:45.122 main model in your own agent that way, again, in a legal way without being 00:46:45.282 --> 00:46:50.082 banned. So yeah, now it's possible again, and I have to reconsider resubscribing. 00:46:50.922 --> 00:46:55.201 Alex Volkov: So the cool thing about, what happened 2 weeks ago is that not only did OpenAI 00:46:55.362 --> 00:46:56.142 always allow this 00:46:57.162 --> 00:47:01.421 use case, they also literally shipped a button and an integration called, like, 00:47:01.802 --> 00:47:05.902 sign with your ChatGPT, and then the other products could use your tokens from your 00:47:06.002 --> 00:47:07.021 subscription, right? 00:47:07.082 --> 00:47:07.202 So 00:47:07.841 --> 00:47:09.421 Devin is one example from Cognition. 00:47:09.482 --> 00:47:11.181 It supports that use case from the get-go. 00:47:11.562 --> 00:47:14.981 Hermes Agent was called out on stage, Nous Research, shout out to Nous Research, our 00:47:15.042 --> 00:47:17.962 good friends from there, and congrats on their round again. 00:47:18.401 --> 00:47:22.422 Um, and Hermes Agent is now going to support both, and this was always the use 00:47:22.442 --> 00:47:25.489 case with OpenAI. It looks looks like Anthropic kind of needs to follow suit. 00:47:26.264 --> 00:47:27.363 so now it's going to be possible. 00:47:27.504 --> 00:47:31.953 Quinn Slack, also a friend from AMP, the coding agent, said that this is why now 00:47:32.134 --> 00:47:33.333 you can use their models. 00:47:34.334 --> 00:47:36.053 Uh, Peter Gostev, will you take the risk? 00:47:36.173 --> 00:47:40.053 Will you use your Anthropic subscription within external tools, or will you just 00:47:40.094 --> 00:47:41.494 say, ah, Claude Code is good enough? 00:47:43.173 --> 00:47:43.532 Peter Gostev: Yeah, 00:47:44.994 --> 00:47:48.233 it is kind of a, a funny one, because I think they're clearly trying to push you 00:47:48.274 --> 00:47:50.274 in the direction of, of using Claude Code. 00:47:50.614 --> 00:47:50.793 Alex Volkov: Yeah. 00:47:50.934 --> 00:47:54.013 Peter Gostev: Um, so yeah, it's kind of annoying. 00:47:54.294 --> 00:47:57.974 I don't know, aren't they making enough money already to just let people be? 00:47:58.574 --> 00:48:01.334 Like, uh, I f- I feel like there, there should be more freedom. 00:48:01.374 --> 00:48:06.313 But yeah, I mean, I, I don't personally, I, I just don't because I, I've got enough 00:48:06.654 --> 00:48:09.753 stuff going on. I use other models via Pi, for example, if I need to. 00:48:09.854 --> 00:48:11.614 I find that, uh, more convenient. 00:48:12.174 --> 00:48:14.513 Um, because I, and I'll tell you why, is that I think 00:48:15.134 --> 00:48:20.033 with Anthropic, with OpenAI, they clearly, their main priority is to use their own 00:48:20.134 --> 00:48:21.893 models, right, and to, 00:48:22.694 --> 00:48:27.953 uh, and to kind of, uh, to, to coordinate everything, and I think it's just easier 00:48:28.054 --> 00:48:28.813 to, to, 00:48:29.454 --> 00:48:32.373 uh, use other models in other ways, but yeah, it, 00:48:33.054 --> 00:48:37.714 uh, I, I prefer to just match the harness to the model, basically, and simply put. 00:48:39.174 --> 00:48:42.013 Alex Volkov: Harness to the model, I think, is very important and something that folks miss. 00:48:43.574 --> 00:48:46.133 Claude Code and Claude definitely go hand in hand. 00:48:46.414 --> 00:48:48.934 Again, Muse Wonder and Muse probably go hand in hand as well. 00:48:49.758 --> 00:48:54.518 Codex is also very good with Astra and less so in other places, and that's why 00:48:54.798 --> 00:48:55.897 it's, for agents like 00:48:56.558 --> 00:48:58.587 Hermes, it's some time to catch up. 00:48:58.788 --> 00:48:59.627 And there's a rumor, 00:49:00.548 --> 00:49:04.428 and we don't love rumors here, but we will say it, that Fable 5.5 00:49:05.468 --> 00:49:05.947 showed up. 00:49:07.308 --> 00:49:07.511 And I 00:49:08.552 --> 00:49:13.412 I think I'm like, I'm getting tired of rumors, so I'm gonna do this for just one 00:49:13.512 --> 00:49:14.872 second. This is not planned. 00:49:15.272 --> 00:49:15.711 But hey, 00:49:16.872 --> 00:49:22.404 uh, Anthropic is going to release Fable 6 also, and then, Opus 6 is going to come 00:49:22.445 --> 00:49:26.304 out, and then Opus 6.1 or 6.5 is going to come out after that, it's going to be even 00:49:26.365 --> 00:49:28.045 better, and then Opus 7. 00:49:28.485 --> 00:49:31.005 Uh, so if you heard it here first, Opus 7 is coming. 00:49:31.564 --> 00:49:35.444 We just don't know when or how, but I'm getting tired of this, like, oh, this 00:49:35.565 --> 00:49:39.555 model is coming and this getting just a lot of, o- okay, so, so when we, when 00:49:39.595 --> 00:49:43.115 we'll get it, we'll get it. I don't understand why this, why the X algorithm 00:49:43.155 --> 00:49:45.254 for all of us is so fine-tuned on just like 00:49:46.355 --> 00:49:51.234 trying to snipe the news of something that we all know is on a continuous improvement 00:49:51.355 --> 00:49:52.154 loop all the time. 00:49:52.955 --> 00:49:56.295 Could you just tell me if that's interesting to you when you see somebody 00:49:56.355 --> 00:49:57.075 and just like, oh, 00:49:57.885 --> 00:49:59.605 I've seen this leak or whatever. 00:50:00.565 --> 00:50:02.405 We know the models are gonna get better. 00:50:02.485 --> 00:50:06.244 We know it's very important for these labs to tell you how better they were gonna 00:50:06.285 --> 00:50:08.765 get. It's, it was especially important pre-IPO. 00:50:09.325 --> 00:50:12.884 Um, I'm interested in testing them out, playing with them, hearing what Peter 00:50:12.925 --> 00:50:17.365 says, what Nisten and Wolfram and Yam say, rather than just like knowing the news. 00:50:18.045 --> 00:50:21.045 But given said all that, somebody did spot 00:50:21.805 --> 00:50:24.684 Fable 5.5, so I'm very excited to see how that's gonna go. 00:50:25.765 --> 00:50:28.404 Uh, comments? Did you guys hear my, do do you push back? 00:50:28.465 --> 00:50:31.275 Do you feel like it's, it's important to snipe and like do breaking news, et 00:50:31.295 --> 00:50:33.575 cetera, at this age, or are we getting a little bit of 00:50:34.295 --> 00:50:36.935 tired of models getting better, numbers got better? 00:50:37.775 --> 00:50:40.475 Nisten Tahiraj: I want to see them run a full factory or something. 00:50:40.575 --> 00:50:41.734 That's that rumor. 00:50:42.615 --> 00:50:46.875 Like, I want to see a whole bunch of humanoids running whatever model 00:50:47.655 --> 00:50:48.775 that, uh, yeah. 00:50:49.815 --> 00:50:51.814 Yeah, they're, yeah, the models are going to get better, of course. 00:50:52.295 --> 00:50:54.255 There's going to be, there's going to be all Opus 6. 00:50:54.495 --> 00:50:58.394 I think Anthropic's going to not allow everyone to just stay on top of the 00:50:58.455 --> 00:50:59.314 benchmarks. They always 00:51:00.015 --> 00:51:01.295 have something in the, 00:51:01.935 --> 00:51:03.274 in the back end, but uh, 00:51:03.895 --> 00:51:06.654 yeah, rumors now are more acceptable for 00:51:07.695 --> 00:51:12.855 uh, like actual humanoids running a factory, running a hospital or something. 00:51:13.215 --> 00:51:13.814 We gotta, 00:51:14.615 --> 00:51:18.254 uh, we gotta leave the rumors for more like actual physical stuff. 00:51:20.055 --> 00:51:21.154 Yeah, I agree. 00:51:21.415 --> 00:51:21.814 That's, 00:51:22.495 --> 00:51:23.414 I'm not sure why. 00:51:24.935 --> 00:51:26.634 Like everybody else, it's common sense. 00:51:26.695 --> 00:51:27.175 I don't know. 00:51:27.534 --> 00:51:27.754 Alex Volkov: Yeah. 00:51:28.619 --> 00:51:29.099 Wolfram, 00:51:30.058 --> 00:51:32.219 from an evaluation standpoint, do you think that, 00:51:32.863 --> 00:51:36.562 s- sniping the news and telling you, oh, this model is coming is still important, 00:51:36.623 --> 00:51:37.662 or are we like past that? 00:51:38.623 --> 00:51:42.862 Wolfram Ravenwolf: I mean, right now, the most important thing for most people using agents is to 00:51:42.943 --> 00:51:45.143 have a way to use a subscription, so 00:51:45.823 --> 00:51:49.562 if the new model comes out and it's API only, or it's so expensive that you run 00:51:49.623 --> 00:51:51.282 out of, uh, your tokens as 00:51:51.943 --> 00:51:55.883 as quickly and you can't really use it, it's not as useful as a model that is just 00:51:56.023 --> 00:51:59.463 smart enough. And I think we are reaching a point where a model is smart enough for 00:51:59.583 --> 00:51:59.782 most 00:52:00.383 --> 00:52:03.562 users, and the much higher intelligence is only for 00:52:04.343 --> 00:52:09.263 bigger projects and stuff that you as a, as a user of a personal agent are not even 00:52:09.423 --> 00:52:10.203 needing for 00:52:11.023 --> 00:52:14.823 book me a flight or, uh, plan my vacation and stuff like that. 00:52:15.743 --> 00:52:15.883 Alex Volkov: No. 00:52:15.983 --> 00:52:20.242 Wolfram Ravenwolf: I think there's a limit to what we really need, and everything beyond that is on a 00:52:20.383 --> 00:52:22.822 company level and not on a personal level anymore. 00:52:24.143 --> 00:52:26.542 Peter Gostev: Um, I, uh, completely disagree. 00:52:26.823 --> 00:52:31.843 I think we're gonna look back at this a year later, gonna be like, oh, we were so, 00:52:31.983 --> 00:52:33.143 so stupid and naive. 00:52:33.343 --> 00:52:34.463 I think it's gonna be amazing. 00:52:34.502 --> 00:52:39.723 And I, I, I get your point, like, I think it's kinda, kinda tedious, you know, this 00:52:39.983 --> 00:52:42.162 number is gonna go up. But I think 00:52:42.783 --> 00:52:43.443 the fact that 00:52:44.703 --> 00:52:46.122 we, we shouldn't take it for granted. 00:52:46.223 --> 00:52:50.002 I think OpenAI and Anthropic are the two labs that actually ship models very 00:52:50.183 --> 00:52:50.643 regularly. 00:52:51.023 --> 00:52:51.203 Alex Volkov: Mhm. 00:52:51.223 --> 00:52:53.862 Peter Gostev: But we haven't seen a full pro model from, 00:52:54.543 --> 00:52:58.403 uh, Google for, what, 8, 10 months? 00:52:58.502 --> 00:52:59.822 I, I don't know, haven't counted. 00:52:59.903 --> 00:53:02.422 So they, they were only shipping, uh, flash models. 00:53:02.983 --> 00:53:07.883 So I think, well, you, I, I find it perfectly reasonable for people not to pay 00:53:07.903 --> 00:53:11.963 attention to rumors, but I think there is information in it when, when, uh, 00:53:12.143 --> 00:53:13.143 companies actually ship. 00:53:13.543 --> 00:53:18.562 So let's say if there is a scenario that Anthropic doesn't ship 5.5 Fable for 00:53:18.623 --> 00:53:22.262 another 2 months, I think that's also kind of newsworthy, like what happened there? 00:53:22.703 --> 00:53:22.842 Alex Volkov: Mm. 00:53:22.863 --> 00:53:28.603 Peter Gostev: Um, so yeah, it's a, I that, so I think it can kind of go both ways, um, 00:53:28.863 --> 00:53:33.343 which I don't think you have to have rumors for, but I still, I I still get a 00:53:33.383 --> 00:53:34.862 little bit of information out of it. 00:53:34.943 --> 00:53:38.522 So yeah, we'll see, like I haven't counted, uh, how many weeks 00:53:39.263 --> 00:53:42.903 OpenAI, when did they first kind of, um, had Astra 00:53:43.543 --> 00:53:46.763 released? It's probably what, like a month ago, 6 weeks ago, something like that. 00:53:46.863 --> 00:53:49.353 Alex Volkov: Astra is around a month ago, 5 month, 5 weeks ago or something, yeah. 00:53:49.773 --> 00:53:54.613 Peter Gostev: So, so yeah, you'd expect them to, I'm gonna leave 6.1 Sol aside because I think 00:53:54.653 --> 00:53:59.713 that was like maybe out of cycle a little bit, but eh, the next Astra, you'd think 00:53:59.773 --> 00:54:00.872 that's probably another month 00:54:01.573 --> 00:54:05.433 away, right? So I think, but if it's 2 months away, then I think we'll get some 00:54:05.533 --> 00:54:08.973 information out of it because it's, uh, like they're probably spending more time, 00:54:09.213 --> 00:54:11.593 I know, aligning the model or something like that. 00:54:11.693 --> 00:54:11.813 So 00:54:12.453 --> 00:54:17.353 yeah, so, you know, I'm very pro rumor, but, you know, uh, the- I'm also a psycho, 00:54:17.413 --> 00:54:18.032 so, you know, 00:54:18.733 --> 00:54:19.572 you don't have to be. 00:54:19.853 --> 00:54:25.372 Alex Volkov: I- I- I think first of all, the descending opinions is, I talked to our fans, in 00:54:25.513 --> 00:54:29.273 IRL, they love this. That when we disagree, we give like multiple, sides of 00:54:29.313 --> 00:54:31.433 the story, and I think they actually do love this. 00:54:31.673 --> 00:54:35.482 So that's great, Peter, and I do think that, you're giving an additional insight 00:54:35.543 --> 00:54:38.782 there where now Anthropic is going to be public in IPO, 00:54:39.543 --> 00:54:42.402 whether or not they're going to release a new model actually will move the news, 00:54:42.543 --> 00:54:44.943 will move markets, given how big Anthropic is. 00:54:45.303 --> 00:54:47.302 Uh, that's a very interesting kind of thing. 00:54:47.423 --> 00:54:51.496 So maybe we're, after doing this for quite a while, maybe we need to find the passion 00:54:51.536 --> 00:54:53.496 again to say, oh, this model's coming, yeah, let's go. 00:54:53.896 --> 00:54:56.236 Um, but yeah, we'll definitely get there. 00:54:56.316 --> 00:54:56.436 So 00:54:57.236 --> 00:54:59.656 I think that's enough about Anthropic, um, 00:55:00.316 --> 00:55:02.956 and given, let me see if I can make this connection here. 00:55:03.356 --> 00:55:07.036 Given that Anthropic is going to IPO likely on Wall Street, and given that I am 00:55:07.116 --> 00:55:10.155 here very, very close to Wall Street, and the reason I'm here is AI Engineer. 00:55:10.236 --> 00:55:12.276 Let's talk about AI Engineer in New York. 00:55:12.596 --> 00:55:12.915 Uh, 00:55:13.636 --> 00:55:16.696 nice? Yeah, okay. this week, AI Engineer in New York. 00:55:16.896 --> 00:55:19.106 This is the second New York, AI Engineer conference. 00:55:19.146 --> 00:55:22.846 For those of you who are new to our show and not familiar with AI Engineer, uh, 00:55:23.026 --> 00:55:26.605 ThursdAI started as a Twitter space and then became a podcast, and the first time 00:55:26.666 --> 00:55:30.285 that I ever told anyone else when they said, hey, what do you do? 00:55:30.466 --> 00:55:34.745 I told them I have a podcast, was on my way to AI Engineer in 2023 to cover it. 00:55:35.266 --> 00:55:41.075 Uh, since then, the conference has blown up, and swyx, the co-host of Latent Space 00:55:41.116 --> 00:55:43.836 and the creator of AI Engineer category and the 00:55:44.516 --> 00:55:44.996 conference, 00:55:45.716 --> 00:55:49.675 has been curating incredible folks throughout these conferences. 00:55:49.836 --> 00:55:52.475 So we went live, earlier this year twice. 00:55:52.956 --> 00:55:57.926 One time from the Moscone West was this, with Wolfram during the, in the middle of 00:55:58.006 --> 00:56:01.286 show The Engineer, and then, uh, Peter actually interviewed you when we chatted 00:56:01.326 --> 00:56:03.486 in London as well. So we went live from London, 00:56:04.126 --> 00:56:04.446 um, 00:56:05.246 --> 00:56:07.046 Wolfram as well, uh, and 00:56:07.726 --> 00:56:11.405 this time, for some reason, they decided not to do it on Thursday, but we're here 00:56:11.486 --> 00:56:13.026 in New York, after AI Engineer. 00:56:13.546 --> 00:56:18.366 And what interests me about this specific conference is the zeitgeist is incredible 00:56:18.426 --> 00:56:22.226 because a lot of folks come there, first of all, to learn so that you know that the 00:56:22.286 --> 00:56:24.745 speakers are gonna get deliver something, but also the hallway conversations. 00:56:24.846 --> 00:56:28.455 And so I have a little bit of a, like, debrief for you because I know that many 00:56:28.496 --> 00:56:29.615 people did not make it to New York. 00:56:29.756 --> 00:56:31.515 None of my co-hosts are here in New York right now. 00:56:31.956 --> 00:56:34.475 Um, and so here's a little bit of a debrief. 00:56:34.916 --> 00:56:35.115 This 00:56:35.756 --> 00:56:40.355 specific conference, specific- because it was around adjacent to Wall Street, this 00:56:40.436 --> 00:56:41.995 focused on finance and AI. 00:56:42.476 --> 00:56:44.116 The big hedge funds were all here. 00:56:44.395 --> 00:56:44.855 Um, 00:56:46.316 --> 00:56:48.196 Two Sigma, let me go find them. 00:56:49.116 --> 00:56:54.815 Capital One and just Bridgewater, Apollo, Citadel, Bloomberg, like all of those 00:56:54.916 --> 00:56:57.516 folks, and then OpenAI and Anthropic were also here. 00:56:57.876 --> 00:57:02.236 Uh, the big theme from Latent Space, what they said, was autonomy versus security. 00:57:02.756 --> 00:57:05.196 Now that the big companies are actually running agents. 00:57:06.116 --> 00:57:10.075 And when I say now that the companies are running agents, to contrast, a year ago, 00:57:10.596 --> 00:57:15.515 2025 was the year of agents in quotes, and the zeitgeist was agents don't work. 00:57:16.276 --> 00:57:18.135 This was before Opus 4.5. 00:57:18.235 --> 00:57:21.235 This was before the kind of the November, January revolution. 00:57:22.316 --> 00:57:26.316 Just scroll back just a year ago, agents did not do work, and now all these folks 00:57:26.396 --> 00:57:29.456 are saying, oh shit, agents do so much work that we need to like put guardrails 00:57:29.476 --> 00:57:34.235 on them and do like, how do we, how do we deal with autonomy and models crossing the 00:57:34.316 --> 00:57:36.795 threshold from non-coding work to like making decisions? 00:57:36.916 --> 00:57:40.935 There's now agents inside banks, there's golden data sets, there's evals, there's a 00:57:41.016 --> 00:57:45.595 bunch of stuff that the industry now contends with that they didn't have to a 00:57:45.656 --> 00:57:46.896 year ago, because a year ago 00:57:47.576 --> 00:57:51.655 agents did not work. Most of the AI in, in companies were software engineers trying 00:57:51.676 --> 00:57:53.335 to get, lines of code out. 00:57:53.736 --> 00:57:57.296 Now there's like agentic things happening inside most of these ones. 00:57:57.856 --> 00:58:02.995 Nisten Tahiraj: Like 2 years ago, I think I posted something about I've never seen an agent 00:58:03.256 --> 00:58:06.975 actually do anything useful other than Plinys buying crypto, 00:58:07.576 --> 00:58:09.455 and that was, uh, that was only 2 years ago. 00:58:09.696 --> 00:58:12.415 So just remember, everyone's opinion is always wrong 00:58:13.136 --> 00:58:14.695 at some, at some point. 00:58:15.056 --> 00:58:18.356 Yeah, there was a tweet from me yesterday, I don't know what people are doing with 00:58:18.456 --> 00:58:18.835 agents. 00:58:19.096 --> 00:58:20.716 Alex Volkov: No, but also we're changing so fast. 00:58:20.736 --> 00:58:22.595 Nisten Tahiraj: And then it just completely 00:58:23.016 --> 00:58:23.156 Alex Volkov: Yeah. 00:58:23.296 --> 00:58:24.975 Nisten Tahiraj: flip one, one, yeah. 00:58:25.296 --> 00:58:27.109 Alex Volkov: And this is what we talk about when we talk about acceleration. 00:58:27.109 --> 00:58:30.655 you, you can't be looking at the current state of the world right now and say, hey, 00:58:31.056 --> 00:58:34.596 AI is this. You, you have to start, this is why we're talking about acceleration. 00:58:34.616 --> 00:58:37.395 This is why we're try- trying to bring you the news and the use cases because, we 00:58:37.496 --> 00:58:38.595 kind of feel that this is happening. 00:58:38.616 --> 00:58:41.806 So all the banks are now, thinking about this, like, very strongly. 00:58:41.946 --> 00:58:45.626 Not only AI engineers are spending tokens, banks are actually using agentic 00:58:45.906 --> 00:58:46.936 processes. Max 00:58:47.656 --> 00:58:53.436 Kanat Alexander, he is, I believe he's a, a distinguished engineer in, at 00:58:53.816 --> 00:58:54.416 Capital One. 00:58:55.775 --> 00:59:00.175 He was the code simplicity guy, uh, and he did a bunch of Kubernetes stuff at Google. 00:59:00.216 --> 00:59:01.536 He's a fascinating dude to talk to. 00:59:01.976 --> 00:59:03.735 His talk was the highlight. 00:59:04.176 --> 00:59:08.055 Uh, he said, like, he talked about, does code quality even matter? 00:59:08.456 --> 00:59:11.395 Uh, given the background of this person, I think it's very, very important. 00:59:11.715 --> 00:59:15.337 A principal software engineer and open source, back in Google. 00:59:15.438 --> 00:59:17.468 He- He- he led a bunch of the bug stuff in Google. 00:59:17.948 --> 00:59:21.918 Uh, he basically says the code quality matters in proportion to how durable your 00:59:22.018 --> 00:59:22.657 software is, 00:59:23.298 --> 00:59:27.178 uh, because AI is about to make a whole new kind of software way more common, 00:59:27.498 --> 00:59:28.657 disposable software. 00:59:29.218 --> 00:59:32.068 Just write once, use it, and then just throw it away. 00:59:32.308 --> 00:59:34.868 So when you have disposable software, does 00:59:35.508 --> 00:59:36.868 quality matter if it works? 00:59:37.548 --> 00:59:43.257 If you build a calculator or an app for yourself, party planner, like, who, like, 00:59:43.358 --> 00:59:47.037 does it matter? So this was one great, uh, 00:59:47.718 --> 00:59:52.318 show, and I recommend folks to tune into the AI Engineer live stream on YouTube, 00:59:52.768 --> 00:59:54.127 which you can rewatch if you want to. 00:59:54.248 --> 00:59:55.728 The talks will get released. 00:59:56.608 --> 01:00:00.587 The other kind of like hallway conversation was, first of all, about 01:00:00.648 --> 01:00:02.607 evals and whether or not agents are working. 01:00:03.048 --> 01:00:06.797 Um, but as I said in the beginning of the show, the thing that was highlight for me, 01:00:07.238 --> 01:00:08.957 which I do want to ask you guys about as well, 01:00:09.718 --> 01:00:09.877 is 01:00:10.918 --> 01:00:15.418 the disconnect between how advanced all those AI engineers are in use cases and 01:00:15.718 --> 01:00:17.117 harnesses and personal agents 01:00:17.998 --> 01:00:20.478 with the revolution of personal agents for normies. 01:00:21.478 --> 01:00:24.877 And how, when I talk to some of these AI engineers, I'm like, hey, do you use 01:00:25.078 --> 01:00:26.778 Instinct, Muse, Grokbot, or 01:00:27.558 --> 01:00:32.257 whatever? And I don't get an answer yes from them, and that's really, really 01:00:32.398 --> 01:00:35.808 something, because many of the engineers just like build their own harnesses or use 01:00:35.888 --> 01:00:37.548 Claude Code or Codex, and why not 01:00:38.208 --> 01:00:39.647 use Codex for some of this stuff? 01:00:40.208 --> 01:00:41.687 And so there's like a big disconnect, 01:00:42.568 --> 01:00:45.598 which is very interesting to me specifically because how big and wide 01:00:45.698 --> 01:00:49.158 OpenClaw and the Hermes Agent got, and those are harnesses you can tweak and play 01:00:49.178 --> 01:00:49.957 with. Um, 01:00:50.578 --> 01:00:52.658 so I did, I did notice this in the halls as well. 01:00:53.338 --> 01:00:57.368 And, specifically, this is highlighted for me because I did talk, like Wolfram does, 01:00:58.008 --> 01:01:00.608 to every person in the world that is not an engineer. 01:01:00.648 --> 01:01:03.407 Wolfram talks to stewardesses on planes to try to AI pill them. 01:01:03.728 --> 01:01:07.107 Uh, so I did the same. Wolfram, I took a page out of your book and I talked with, 01:01:07.648 --> 01:01:08.087 there was 01:01:10.488 --> 01:01:12.888 the Token Billionaires Lounge we w- told you about before. 01:01:12.951 --> 01:01:14.467 This, this AI engineer had this again. 01:01:14.708 --> 01:01:18.887 So if you are spending more than 1 billion tokens a week or something, you get this 01:01:18.988 --> 01:01:19.627 golden card. 01:01:20.588 --> 01:01:23.507 I think I have it, hold on. Wolfram, there's a new card that you need to get. 01:01:24.708 --> 01:01:28.607 The token billionaires card kind of looks like this, and then this gives you access 01:01:28.668 --> 01:01:29.588 to this, like, lounge. 01:01:30.508 --> 01:01:33.107 That this lounge is kind of like for billionaires lounge. 01:01:33.548 --> 01:01:36.288 There were a lot of service providers in that lounge that were not part of the 01:01:36.348 --> 01:01:38.268 conference, and I tried to talk with them. 01:01:38.988 --> 01:01:41.307 Here's the thing that I got from them, not from the engineers. 01:01:41.428 --> 01:01:46.268 All of them know about Muse because Muse is getting pushed down their throats in 01:01:46.388 --> 01:01:47.667 every Instagram post. 01:01:48.067 --> 01:01:49.418 Like, Zuck is really going out of their way. 01:01:49.457 --> 01:01:52.587 So, like, they all know about Muse. Many of them didn't install it or didn't get, 01:01:53.942 --> 01:01:56.572 why it's, why it's good for them, didn't yet, catch that. 01:01:57.532 --> 01:01:59.452 Uh, so those conversations were fascinating. 01:01:59.932 --> 01:02:01.452 Um, let's see what else. 01:02:02.492 --> 01:02:04.052 So, AI engineer, if you haven't been there. 01:02:04.332 --> 01:02:06.992 The highlight for me, and I, I tell it to everyone also. 01:02:08.312 --> 01:02:11.591 If you're listening to the show, you're good enough to submit a 01:02:12.231 --> 01:02:13.791 talk proposal to AI Engineer. 01:02:13.992 --> 01:02:18.211 And if you submit a proposal to AI Engineer, you get access, and Wolfram and 01:02:18.312 --> 01:02:20.151 Peter both can comment that this is true. 01:02:20.552 --> 01:02:22.152 You get access to this, where am I? 01:02:22.192 --> 01:02:25.591 Oh, there we go. Uh, you get access to this amazing speakers lounge where all the 01:02:25.672 --> 01:02:30.072 speakers that are hand selected are top of the level of people, and you get also 01:02:30.192 --> 01:02:31.231 access to Speakers Dinner. 01:02:31.312 --> 01:02:35.581 And to me, the Speakers Dinner and the Speakers Lounge hangout is the highest 01:02:35.881 --> 01:02:39.481 tier of, folks who understand whatever word I say. 01:02:39.722 --> 01:02:42.082 You don't need to explain what harness is. 01:02:42.332 --> 01:02:43.492 you actually learn a lot. 01:02:44.092 --> 01:02:48.311 One example that I can shout out is, I just w- were walking by and somebody, 01:02:48.532 --> 01:02:49.632 somebody's tag said linear. 01:02:49.852 --> 01:02:50.891 I was like, oh, who's this dude? 01:02:51.092 --> 01:02:52.242 Oh, he's the, you know, the, the 01:02:52.922 --> 01:02:53.662 senior, like, 01:02:54.322 --> 01:02:57.542 I don't remember the full, I think he's like manages software for Linear, all of 01:02:57.602 --> 01:03:00.222 Linear. Linear is a, agentic forward 01:03:01.142 --> 01:03:02.261 task management software. 01:03:03.702 --> 01:03:05.742 And I had a bunch of feedback about Linear. 01:03:05.782 --> 01:03:09.362 We just sat down just right there and talked about how I use Linear with my 01:03:09.422 --> 01:03:10.941 agents and what they can do to improve. 01:03:11.262 --> 01:03:12.521 This is one of those conversations. 01:03:12.542 --> 01:03:16.441 So if you are listening to this pod, if you have anything to say, I strongly, 01:03:16.542 --> 01:03:20.582 strongly recommend to apply for a conversation talk at AIE. 01:03:21.262 --> 01:03:22.822 Um, Wolfram, Peter, 01:03:23.622 --> 01:03:25.081 any thoughts on that space? 01:03:26.802 --> 01:03:30.541 Wolfram Ravenwolf: It's the highest level of networking you can have in the AI space because you are 01:03:30.562 --> 01:03:31.541 not talking to some 01:03:32.162 --> 01:03:35.921 other attendees of a conference, you are talking to the the big shots who are 01:03:36.162 --> 01:03:38.341 creating the AI news we are reporting about. 01:03:38.602 --> 01:03:40.562 So this is super, super amazing. 01:03:40.762 --> 01:03:44.722 And even if you don't become a speaker, I mean, just attending and going to the 01:03:44.882 --> 01:03:49.021 booth and talking to the people you meet there, they're all the, the people from 01:03:49.122 --> 01:03:51.801 Google and Anthropic and OpenAI, they are all there. 01:03:52.402 --> 01:03:56.642 So it's the best way to connect to the makers and shakers in the AI space, 01:03:56.762 --> 01:03:58.521 definitely. Best conference I know. 01:04:00.482 --> 01:04:02.382 Alex Volkov: 100%. Um, Peter, go ahead. 01:04:03.122 --> 01:04:04.522 Peter Gostev: Yeah, now I would say also 01:04:05.482 --> 01:04:10.821 everyone can apply, but do take time to make a good application because I can't 01:04:10.842 --> 01:04:15.342 remember what the stats were, but I think swyx was saying, not for this one, but for 01:04:15.362 --> 01:04:20.561 the one in London, I think it was like the, the acceptance rate was very low. 01:04:21.122 --> 01:04:22.182 So I would say 01:04:23.562 --> 01:04:24.701 put, put effort in. 01:04:24.882 --> 01:04:25.042 It's 01:04:25.082 --> 01:04:25.742 Alex Volkov: Put effort. 01:04:25.762 --> 01:04:28.282 Peter Gostev: It's, yeah, the, the reason why it's a 01:04:29.002 --> 01:04:33.761 good networking, uh, opportunity is because it is a bunch of good people. 01:04:34.042 --> 01:04:39.881 So, yeah, it's i- it's worth, you know, don't just LLM generate, uh, your your 01:04:40.802 --> 01:04:42.201 summary and proposal. 01:04:42.442 --> 01:04:43.562 Yeah, take take your time. 01:04:43.922 --> 01:04:47.462 Alex Volkov: Here's an additional insight that I got, and I think we've talked about this on the 01:04:47.521 --> 01:04:49.841 show, but, obviously this is on our mindset, right? 01:04:49.882 --> 01:04:50.082 So, 01:04:51.362 --> 01:04:52.121 AI engineer 01:04:53.002 --> 01:04:54.532 in London, the zeitgeist 01:04:55.192 --> 01:04:57.671 were talking about, should we still read code? 01:04:58.272 --> 01:05:00.131 We don't write code anymore. Should we still read code? 01:05:00.192 --> 01:05:02.152 I talked about the ZL continuum on my way back. 01:05:02.952 --> 01:05:06.131 Uh, I think it's clear now that barely, barely anybody's reading code. 01:05:06.192 --> 01:05:08.952 Now we're like stressed how to even test it and make it valuable. 01:05:09.592 --> 01:05:10.771 The zeitgeist here 01:05:11.551 --> 01:05:11.771 was 01:05:12.431 --> 01:05:13.572 the end of attention, 01:05:14.272 --> 01:05:19.852 and how do we scale from managing one or two agents directly by steering them and 01:05:19.912 --> 01:05:20.611 seeing outputs 01:05:21.232 --> 01:05:26.812 to running multiple, multiple tens or hundreds even agents, and how do we have 01:05:27.192 --> 01:05:31.412 the attention to attend to all of that and make, decisions, or whether or not we just 01:05:31.432 --> 01:05:33.222 like let this swarm run and deal with this. 01:05:33.542 --> 01:05:36.061 And I think it's a very interesting, I talked to many, many people. 01:05:36.302 --> 01:05:39.722 There's not a lot of solutions. Ryan Carson are, by the way, shout out Ryan. 01:05:39.781 --> 01:05:43.021 We hung out during this whole AI engineer, our previous, our co-host. 01:05:43.152 --> 01:05:45.652 Um, me and Ryan, we were talking to like a bunch of people. 01:05:45.732 --> 01:05:48.711 Ryan gave a great talk about this as well, and he's missing a control plane. 01:05:48.772 --> 01:05:51.971 He's missing like a one central place where he can like see all of the stuff 01:05:52.032 --> 01:05:52.592 that's working. 01:05:53.312 --> 01:05:56.471 Another thing that was in the zeitgeist is the move to the cloud. 01:05:56.912 --> 01:05:58.151 Many people are just getting 01:05:58.752 --> 01:06:01.151 limited by their local environments. 01:06:01.952 --> 01:06:04.231 You can see this from the Frontier Labs as well. 01:06:05.152 --> 01:06:09.572 Codex Cloud just launched, InDev Day renewed, launched Claude Cloud. 01:06:09.872 --> 01:06:13.352 You can now hand off tasks to the cloud as well, and they even give you, like Nisten, 01:06:13.372 --> 01:06:17.131 I think you said, they give you some budget, like 100 dollars, to try the cloud 01:06:17.192 --> 01:06:21.011 thing. Uh, obviously Devin and Cursor, they have cloud agents, um, 01:06:21.752 --> 01:06:25.571 and folks are getting to the limit of their environment, not only because 01:06:26.312 --> 01:06:30.131 it doesn't run 24/7, you need to run caffeinate to keep it running, but also 01:06:30.192 --> 01:06:32.152 because the rock environments are running out of memory. 01:06:32.712 --> 01:06:34.172 and Peter, I think you mentioned this as well. 01:06:34.192 --> 01:06:36.871 You're like coding so much in your things that the agents just like killed your 01:06:36.912 --> 01:06:38.351 computer, so you got a different box. 01:06:38.712 --> 01:06:42.716 Uh, and when you spin those up in sandboxes on the cloud, you kind of don't 01:06:42.756 --> 01:06:44.275 have this restriction. You c- you can't go wide. 01:06:44.296 --> 01:06:47.946 So like, we're getting to a point where not only are we using more billions tokens 01:06:48.026 --> 01:06:50.326 more, my case, definitely Peter's case, 01:06:51.106 --> 01:06:55.806 definitely Wolfram's case, all of us here, probably like 10x for the past few months, 01:06:55.896 --> 01:06:57.615 and the Zeitgeist and the engineer as well. 01:06:58.136 --> 01:07:02.035 We also need a place for that whole thing to happen, all the tools to happen, and 01:07:02.096 --> 01:07:03.695 that's not happening in local environment. 01:07:05.575 --> 01:07:07.216 I heard, um, the engineer. 01:07:07.776 --> 01:07:08.055 Um, 01:07:08.756 --> 01:07:13.746 all right, folks, next up we have, Omar Sanseviero and Geneviève Huskens, I 01:07:13.906 --> 01:07:16.025 believe. Uh, they're joining us now. 01:07:16.386 --> 01:07:20.646 Uh, just before that, as always, given the show is sponsored by one sponsor only, 01:07:20.946 --> 01:07:24.745 CoreWeave, I would love to tell you about some of the stuff that we have, and then 01:07:24.786 --> 01:07:26.846 we're gonna talk with, Geneviève and Omar about 01:07:27.946 --> 01:07:32.325 finally we got the new big Gemini back, and it's so exciting to hear about this as 01:07:32.366 --> 01:07:35.866 well. Let's go to this week's buzz, and just right after that, we'll have our 01:07:35.906 --> 01:07:36.226 guests. 01:07:45.706 --> 01:07:46.425 CoreWeave. 01:07:48.546 --> 01:07:49.346 From CoreWeave. 01:07:54.586 --> 01:07:55.045 I think 01:07:55.945 --> 01:07:56.225 that 01:07:56.826 --> 01:07:57.666 it's time to 01:07:58.506 --> 01:08:02.986 sunset that transition and come up with a new transition, Wolfram, because, as of 2 01:08:03.026 --> 01:08:05.686 weeks ago, CoreWeave Forge is now, 01:08:07.109 --> 01:08:11.549 the home for the weights and biases as a tool, and also a bunch of other tools, 01:08:11.948 --> 01:08:13.419 specifically, Agent Lens. 01:08:13.579 --> 01:08:15.808 Agent Lens launches this week, and, if 01:08:16.409 --> 01:08:18.328 the highlight here for me in Agent Lens, 01:08:18.999 --> 01:08:21.609 Wolfram, t- tell me if if if you're also on this, is that 01:08:22.429 --> 01:08:28.169 Claude Code now supports OpenTelemetry exports, and then now you can use our 01:08:28.309 --> 01:08:31.219 tools to actually look at the agent runtimes that you have 01:08:32.019 --> 01:08:34.498 and analyze if it did the work correctly. 01:08:34.939 --> 01:08:36.619 So, um, would, 01:08:38.572 --> 01:08:39.972 would love to just like let you know that 01:08:41.112 --> 01:08:42.271 that you can now use that. 01:08:42.392 --> 01:08:45.171 Uh, you can sign up on forge.coreweave.com. 01:08:45.267 --> 01:08:48.366 The highlight for me from Fully Connected 2 weeks ago was that 01:08:49.547 --> 01:08:49.866 given 01:08:50.627 --> 01:08:55.687 the previous life at CoreWeave, where you had to get, you had to go talk to a 01:08:55.747 --> 01:08:58.467 salesperson, register for long contracts and sign up. 01:08:58.947 --> 01:09:00.946 Forge, for the first time, is our 01:09:01.587 --> 01:09:05.666 register as a regular SaaS offering, and then you get access to a bunch of tools 01:09:05.787 --> 01:09:08.527 from our, CoreWeave Agentic Cloud, tools like distillation as well. 01:09:08.767 --> 01:09:14.177 We talked about model distillation a lot on the show, but now via the OpenPipe 01:09:14.337 --> 01:09:18.147 folks, which are great folks, friends of the show, you can get, model distillation 01:09:19.567 --> 01:09:21.287 for yourself. You don't have to set that all up. 01:09:21.567 --> 01:09:23.406 It's all part of forge.coreweave.com. 01:09:24.167 --> 01:09:24.486 Um, 01:09:25.207 --> 01:09:28.376 Wolfram, did you set up your Claude code with the AgentLens yet? 01:09:28.717 --> 01:09:29.936 Are you sending Otel? 01:09:31.157 --> 01:09:32.576 Wolfram Ravenwolf: At the moment, I, I don't. 01:09:33.037 --> 01:09:37.716 It's all local right now, but uh, it makes definitely sense to integrate all of this, 01:09:37.797 --> 01:09:41.657 and the integration is what Forge is all about, to have integrations between all 01:09:41.677 --> 01:09:46.337 the different tools in the whole AI loop, from the evaluations to the inference it's 01:09:46.397 --> 01:09:49.356 running on to uh, the curation of the data sets. 01:09:49.437 --> 01:09:52.236 It's all connected, and that is a new thing, I think. 01:09:52.437 --> 01:09:56.516 Uh, it's not just Weave or a individual product where you have to export, import 01:09:56.557 --> 01:10:00.176 stuff, you can jump between them and the connection all on one. 01:10:00.197 --> 01:10:00.516 Alex Volkov: Um, 01:10:00.837 --> 01:10:01.297 Wolfram Ravenwolf: on account. 01:10:01.357 --> 01:10:02.796 Alex Volkov: I want to call out yesterday. 01:10:02.837 --> 01:10:06.426 Yesterday we said, at the event that we did, we sat down and we talked about some 01:10:06.527 --> 01:10:09.546 stuff, and obviously I was like, okay, this is a CoreWeave sponsored dinner, let 01:10:09.567 --> 01:10:10.847 me at least tell you about their offerings. 01:10:11.366 --> 01:10:14.006 And the folks who use Weights and Biases didn't even know we have Aria. 01:10:14.677 --> 01:10:15.876 and somebody's like, what is Aria? 01:10:15.957 --> 01:10:17.277 So Aria is a auto 01:10:18.957 --> 01:10:21.591 agent assistant, agentic research assistant, 01:10:22.211 --> 01:10:25.815 basically auto research from Karpathy personified in the product. 01:10:26.055 --> 01:10:29.274 So if you are using Weights and Biases for anything, people didn't even know that we 01:10:29.315 --> 01:10:32.484 have this offering, so we definitely need to work a bit about, externalizing our 01:10:32.545 --> 01:10:36.355 offerings. But if you have seen auto research as a concept from Andrej Karpathy 01:10:36.435 --> 01:10:38.605 in his tweet, it's now in GA. 01:10:38.685 --> 01:10:41.834 You can just use it, inside Forge on your training runs. 01:10:42.495 --> 01:10:46.058 All right, folks, I think this is enough about Forge, and now we're gonna move on, 01:10:46.218 --> 01:10:47.237 uh, because we have 01:10:48.138 --> 01:10:52.177 not quite breaking news, but definitely news from the frontier of AI with two 01:10:52.378 --> 01:10:55.018 great guests here, and let's go talk to them. 01:11:05.378 --> 01:11:05.777 Geneviève Huskens: Hello. 01:11:06.218 --> 01:11:06.638 Alex Volkov: Hi. 01:11:07.498 --> 01:11:12.508 Omar and Geneviève are joining us from DeepMind, from Google DeepMind, and you 01:11:12.548 --> 01:11:13.628 guys have news to share, 01:11:14.268 --> 01:11:14.467 and 01:11:15.348 --> 01:11:17.307 it's, can I just say it's about time? 01:11:17.628 --> 01:11:18.708 It's about time. 01:11:19.068 --> 01:11:19.207 Geneviève Huskens: Yep. 01:11:19.227 --> 01:11:22.287 Alex Volkov: We talked on the show before, and folks, it's about time for a big model. 01:11:22.348 --> 01:11:27.240 So you guys launched, Gemini, 4 Argon, but I would love for you to tell us about the 01:11:27.321 --> 01:11:29.960 launch, about what's new, about what are the use cases. 01:11:30.081 --> 01:11:32.201 I'm all ears. It's out for everyone. 01:11:32.361 --> 01:11:36.830 It's number one on Arena, and, uh, J- it helped make Gemma three times faster. 01:11:36.911 --> 01:11:38.510 Omar, that's kind of your corner as well, right? 01:11:38.751 --> 01:11:40.390 I would love to hear about the use cases. 01:11:40.631 --> 01:11:43.671 So please give us a highlight, and then we're gonna dive into use cases. 01:11:44.030 --> 01:11:45.130 And Geneviève, great to meet you. 01:11:45.171 --> 01:11:46.010 Thank you so much for coming. 01:11:46.030 --> 01:11:47.090 Geneviève Huskens: Likewise. Thanks for having me. 01:11:48.711 --> 01:11:50.231 Alex Volkov: All right, who's gonna start? Omar, let's go. 01:11:50.284 --> 01:11:53.241 Omar Sanseviero: I can go first. So yeah, I mean, Gemini 4 Argon is our latest model. 01:11:53.360 --> 01:11:55.500 Uh, TLDR, it's a great model. 01:11:55.641 --> 01:11:57.840 You have probably heard many things about it. 01:11:58.081 --> 01:12:02.360 My only recommendation is to go try it out, and I think many of you will be 01:12:02.521 --> 01:12:03.574 pleasantly surprised. 01:12:03.601 --> 01:12:06.061 eh, yeah, the model can be used for many things. 01:12:06.161 --> 01:12:07.721 We've been using it internally. 01:12:08.201 --> 01:12:11.261 Eh, as you said, we've been doing optimizations for Gemma. 01:12:11.361 --> 01:12:16.780 We have been able to achieve over 100,000 tokens per second with the E2B, so our 01:12:16.881 --> 01:12:22.854 smallest model. Eh, the model has a 1 million unified, eh, context, eh, so 01:12:23.014 --> 01:12:26.293 pretty much you can pass very long videos, but you can also have very long outputs, 01:12:26.334 --> 01:12:27.654 so that is quite exciting. 01:12:27.894 --> 01:12:28.033 Alex Volkov: Mm. 01:12:28.254 --> 01:12:31.454 Omar Sanseviero: Uh, and yeah, we have been very excited to see what the community has been building 01:12:31.474 --> 01:12:33.223 with the model. what do you think, Geneviève? 01:12:33.944 --> 01:12:35.704 Geneviève Huskens: Yeah, we've had so much fun, building with Argon. 01:12:35.744 --> 01:12:39.624 I it had been a while since we launched a new, model, and so most people will know 01:12:39.684 --> 01:12:43.403 that this is the first model that came out in our Gemini 4 series, by far our largest 01:12:43.504 --> 01:12:47.363 train training run ever. And so, uh, Argon, I think, is the first, actually 01:12:47.424 --> 01:12:50.443 quite early checkpoint that already is doing really well, but we're also super 01:12:50.464 --> 01:12:52.883 excited about everything that we can build, next now. 01:12:52.924 --> 01:12:56.043 And so Argon is already training our future generations of models. 01:12:56.364 --> 01:12:59.823 Um, I think we've really enjoyed the fact that Argon can do many more long-running 01:12:59.924 --> 01:13:03.424 tests, and so now everyone is shipping a bunch of stuff overnight, and you wake up 01:13:03.484 --> 01:13:06.943 and you just wake up to really fun things that Argon has been cooking in the 01:13:06.964 --> 01:13:11.163 background. And so, um, it's much better at 3D, um, much better at creativity. 01:13:11.804 --> 01:13:13.144 The writing is really pleasant, and so 01:13:13.244 --> 01:13:13.503 Alex Volkov: Mm. 01:13:13.524 --> 01:13:15.823 Geneviève Huskens: in general, it's just been super fun to build things with this model. 01:13:15.884 --> 01:13:18.983 And again, uh, we think there's many more great things to come from us, uh, in the 01:13:19.004 --> 01:13:19.524 next few weeks. 01:13:19.947 --> 01:13:20.674 Alex Volkov: I would love- 01:13:20.694 --> 01:13:21.194 Omar Sanseviero: Yeah. 01:13:21.254 --> 01:13:22.143 Alex Volkov: Yeah, go ahead, Omar, please. 01:13:22.263 --> 01:13:25.363 Omar Sanseviero: Uh, yeah, ju- just that as usual, we have been working with all of the open source 01:13:25.424 --> 01:13:29.263 ecosystem to make sure that you can use the model, eh, regardless of what's your 01:13:29.344 --> 01:13:33.024 favorite tool. So we have been working to make sure that this model works well with 01:13:33.143 --> 01:13:38.443 OpenClaw, with Hermes, with, uh, all of the tools, uh, Llama, uh, sorry, uh, 01:13:38.584 --> 01:13:40.583 LangChain, uh, Vercel, and so on. 01:13:40.624 --> 01:13:44.824 So, uh, super excited to also collaborate with all of this ecosystem to, to get the 01:13:44.864 --> 01:13:45.403 model out there. 01:13:46.704 --> 01:13:49.507 Alex Volkov: I want to say, the release definitely is a wide surface release, right? 01:13:49.507 --> 01:13:51.434 it's available pretty much everywhere, uh, 01:13:51.454 --> 01:13:51.633 Omar Sanseviero: Yeah. 01:13:51.694 --> 01:13:53.823 Alex Volkov: and also in your own, Gemini API and AI Studio. 01:13:53.884 --> 01:13:57.002 Shout out to AI Studio folks, for always, bringing us in to the fold. 01:13:57.263 --> 01:13:59.862 Uh, Gemini app only for Pro and Ultra, is that correct? 01:14:00.943 --> 01:14:04.682 So folks who are subscribing to Gemini app on the Pro and Ultra tier are giving this, 01:14:04.783 --> 01:14:06.212 and Antigravity Ultra as well. 01:14:06.573 --> 01:14:08.253 2 dollars in and 10 dollars out. 01:14:08.573 --> 01:14:10.292 Uh, this is introductory pricing? 01:14:10.893 --> 01:14:11.292 Omar Sanseviero: Exactly. 01:14:11.693 --> 01:14:13.733 Alex Volkov: Yeah. Uh, and then what happens then? 01:14:13.886 --> 01:14:16.233 Ex- extension, new model, another interactive pricing? 01:14:17.233 --> 01:14:17.593 Um. 01:14:17.873 --> 01:14:19.753 Omar Sanseviero: Breaking news, breaking news for, for next month. 01:14:20.072 --> 01:14:23.753 Alex Volkov: All right. and then, based on what it says now then it doubles, 1 million token 01:14:23.793 --> 01:14:25.832 context, and I think let's talk about multimodality. 01:14:25.993 --> 01:14:29.362 Uh, Google and Gemini models specifically were always kind of like the, the front 01:14:29.443 --> 01:14:32.733 leading in multimodal, including like watching the video that you send to it. 01:14:32.853 --> 01:14:34.772 1 million token is a great thing for that. 01:14:35.113 --> 01:14:37.762 Omar Sanseviero: Yeah, you'll see that the model is very good at understanding 01:14:38.623 --> 01:14:42.943 visuals, um, the space, and it has a very good understanding of the world and 01:14:43.103 --> 01:14:44.462 physics. So, uh, 01:14:45.423 --> 01:14:49.143 many people kind of forget that models are not just for coding, like models are to 01:14:49.183 --> 01:14:53.223 understand the world, and the world is highly multimodal, uh, multilingual as 01:14:53.343 --> 01:14:57.463 well, and in general for Gemini, we make sure that this model works well across all 01:14:57.583 --> 01:14:58.662 of these kind of use cases. 01:14:59.183 --> 01:15:00.462 Alex Volkov: That's incredible. Let's talk about vision. 01:15:00.743 --> 01:15:05.112 Uh, Roboflow called this model like number one in vision and detect OCR stuff as 01:15:05.133 --> 01:15:06.613 well, very, very, very strong. 01:15:07.053 --> 01:15:09.922 Um, any specific demos around vision that you guys want to show us? 01:15:10.243 --> 01:15:12.042 Any cool things? That'd be dope. 01:15:12.083 --> 01:15:14.502 Geneviève Huskens: Uh, let's start with the ones that I have ready, and then 01:15:14.603 --> 01:15:14.983 Alex Volkov: Let's go, let's go. 01:15:15.003 --> 01:15:17.003 Geneviève Huskens: if you have anything else, vision in the background. 01:15:17.083 --> 01:15:20.203 Uh, the ones that I picked for today are mostly very cool looking. 01:15:20.962 --> 01:15:23.762 Uh, but yeah, it's great to see the vision capabilities always, if you're building 01:15:23.783 --> 01:15:27.322 with a model and it can check its own work, if it can record videos and kind of 01:15:27.383 --> 01:15:29.343 see what's going on, then that is super, super helpful. 01:15:29.423 --> 01:15:32.542 So even for some of our 3D stuff that we're now doing, it's just so pleasant 01:15:32.583 --> 01:15:34.342 that Gemini can watch its own videos back. 01:15:34.663 --> 01:15:38.402 Even things like motion graphics is very helpful because it can look at a video, 01:15:38.543 --> 01:15:39.242 understand the style, 01:15:39.383 --> 01:15:39.562 Alex Volkov: Ooh. 01:15:40.143 --> 01:15:44.042 Geneviève Huskens: it. And so I made a trailer about Argon earlier this week that took me just maybe 01:15:44.143 --> 01:15:45.902 30 minutes, uh, and it came out really good. 01:15:45.983 --> 01:15:49.562 Um, but yeah, in terms of, uh, the demos that we've been cooking, the first one 01:15:49.623 --> 01:15:51.622 that I made was, uh, Desk Racer. 01:15:51.983 --> 01:15:55.463 Um, and so first of all, if you go into Google AI Studio and you click on the 01:15:55.503 --> 01:15:58.822 gallery, you can find a bunch of stuff that we've made, uh, with Argon. 01:15:59.143 --> 01:16:02.822 Um, and so Desk Racer came from our multimodality, where I wanted to build a 01:16:02.943 --> 01:16:04.663 racing game on top of my actual desk. 01:16:04.943 --> 01:16:08.522 And so the microphone that you see here, and then this like cute little cat, um, 01:16:09.143 --> 01:16:12.902 just gave Gemini some reference images and said, can you just build me a racetrack? 01:16:13.062 --> 01:16:17.182 Um, and what's fun about this is, I mean, it's a long running task to understand 01:16:17.583 --> 01:16:21.382 where to go around the objects, to make it look good, uh, to sort of plan the 01:16:21.423 --> 01:16:26.142 lighting and everything. Um, and so now if you can play this, um, and then Argon 01:16:26.303 --> 01:16:29.283 even, hang on, let me switch camera view and turn the sound off. 01:16:29.503 --> 01:16:30.062 Alex Volkov: That's so cool. 01:16:30.623 --> 01:16:36.062 Geneviève Huskens: Argon even prompted our Omni video model, um, to extract sounds, like engine sounds 01:16:36.223 --> 01:16:40.662 from, uh, different race cars, listen to all the sounds, found the sort of, uh, 01:16:40.783 --> 01:16:43.022 audio that was able to loop, and then stitch all of that. 01:16:43.463 --> 01:16:47.203 And so, like, yes, it looks good, but it's also cool to think about the rest that's 01:16:47.223 --> 01:16:50.963 happening in the background to, in the end, uh, come up with a very, uh, shiny, 01:16:51.063 --> 01:16:54.523 uh, result. And so this one I really like because it's where I spend all day, every 01:16:54.583 --> 01:16:57.983 day, uh, but it's it's fun to see Argon, uh, do such a good job. 01:16:58.063 --> 01:17:01.683 And so I built all these little other camera views in it, uh, and so this you 01:17:01.723 --> 01:17:02.943 can play in AI Studio. 01:17:03.183 --> 01:17:06.483 Uh, there's a timed mode where you just drive three laps, uh, and so that's pretty 01:17:06.503 --> 01:17:08.842 fun. Um, and then this one is- 01:17:09.203 --> 01:17:12.003 Alex Volkov: Just before you switch, could I ask you for our audience, how do you approach 01:17:12.062 --> 01:17:16.122 building this demo? Is this all in AI Studio, using any other harnesses, or 01:17:16.183 --> 01:17:17.402 using like the internal tools? 01:17:17.603 --> 01:17:21.422 Could you tell us how do we get to building this where Argon can download the 01:17:21.523 --> 01:17:25.222 files, then like using Omni to generating things and like extracting? 01:17:25.283 --> 01:17:27.103 Where, where's all the tool happening? 01:17:27.243 --> 01:17:30.103 Geneviève Huskens: Yeah, that's a, that's a great question, and actually I wrote a guide about that 01:17:30.203 --> 01:17:34.442 earlier this week, and so in essence, started with the reference images, um, use 01:17:34.523 --> 01:17:38.322 GrillMe, so like ask me questions about how I should plan, uh, uh, this project, 01:17:38.602 --> 01:17:42.243 and then just use a big goal command and said, you know, I just want you to spawn a 01:17:42.283 --> 01:17:44.763 bunch of sub-agents, work on it however long you need. 01:17:45.203 --> 01:17:50.083 Some things that helped were A, um, using adversary, uh, uh, sub-agents, and so 01:17:50.123 --> 01:17:53.902 there was always one sub-agent whose job it was to look at the outputs, grade them 01:17:53.923 --> 01:17:58.142 for accuracy, for lighting, come back with feedback, and then, um, give a rating out 01:17:58.163 --> 01:18:01.363 of 10, and then keep going until the rating was, uh, very, very high. 01:18:01.723 --> 01:18:04.082 Uh, so that, first of all, uh, uh, was really helpful. 01:18:04.163 --> 01:18:08.923 And then secondly, we've all really been enjoying using, uh, Nano Banana to make 3D 01:18:09.003 --> 01:18:13.082 even better. And so in this example, for instance, I'll turn off the audio, but 01:18:13.243 --> 01:18:15.043 this is a demo that looks really, really good. 01:18:15.363 --> 01:18:16.782 It's like a pagoda, uh, demo. 01:18:16.843 --> 01:18:21.102 And so if you have a poor render and you give it to Nano Banana and you say, make 01:18:21.123 --> 01:18:25.022 this look really good and realistic, and then you pass that image back to Argon and 01:18:25.083 --> 01:18:27.382 say, do whatever you need to do to make it look like this. 01:18:27.683 --> 01:18:27.822 Alex Volkov: Mm. 01:18:27.843 --> 01:18:32.222 Geneviève Huskens: Um, I'm now using that in all of my, uh, 3D projects, and so the lighting that you 01:18:32.283 --> 01:18:36.242 saw in Desk Racer, uh, the neon, uh, lighting was designed by Nano Banana 01:18:36.443 --> 01:18:40.103 because at first it looked very bland, and so this, I think, is a really nice way to 01:18:40.163 --> 01:18:43.942 interact, uh, with any model, just that image inspiration is a really good, uh, 01:18:44.043 --> 01:18:44.623 visual reference. 01:18:45.363 --> 01:18:45.863 Alex Volkov: Oh, that's incredible. 01:18:45.883 --> 01:18:48.703 Geneviève Huskens: And yeah, I did write a guide about this, so, uh, sorry, if you look this up. 01:18:48.723 --> 01:18:51.192 Alex Volkov: No, no, it's it's all good. It's just like a definitely a great tip for our audience 01:18:51.253 --> 01:18:55.452 as well. Use visual references because the model is very good at multimodality from 01:18:55.513 --> 01:18:56.902 other models, for this. 01:18:57.303 --> 01:18:57.943 That is incredible. 01:18:58.823 --> 01:18:59.162 Geneviève Huskens: Amazing. 01:18:59.286 --> 01:19:00.302 Alex Volkov: show us another one. That's, that's great. 01:19:00.483 --> 01:19:04.002 Geneviève Huskens: Yeah, so, so this one is, is pretty cool because like, yes, it's fun to do things 01:19:04.123 --> 01:19:08.242 with, uh, Blender and Three.js, uh, but in this case, my colleague actually asked, 01:19:08.363 --> 01:19:11.643 um, Argon to create this 3D scene completely from scratch. 01:19:11.683 --> 01:19:15.122 So zero libraries, zero, uh, external things being pulled in. 01:19:15.523 --> 01:19:19.582 Um, and what's cool about this is if you think about how a model needs to do that, 01:19:19.663 --> 01:19:23.842 is it needs to think about all the light, all the ways that objects might interact 01:19:23.883 --> 01:19:27.763 with each other. And so the light bouncing off the wooden floor here needs to be 01:19:27.923 --> 01:19:29.042 refracted in a certain way. 01:19:29.363 --> 01:19:32.323 Uh, the light shining through the glass needs to exist in a certain way. 01:19:32.443 --> 01:19:36.562 And so if you think about the amount of real world understanding and then also the 01:19:36.643 --> 01:19:39.943 quality of code generation that you need to make happen to make a project like 01:19:40.002 --> 01:19:42.602 this, uh, that we also thought was very impressive. 01:19:43.043 --> 01:19:45.883 Um, and then the other thing that was fun about this video, I'll, I'll spare you the 01:19:45.923 --> 01:19:51.182 audio, but uh, Argon also narrated it by itself, um, uh, picked the, the music with 01:19:51.283 --> 01:19:54.983 Lyria. And so again, the idea that you can build something cool, give it back to 01:19:55.043 --> 01:19:59.122 Argon, and then say, now make this into a cool video, that all just ends up working. 01:19:59.203 --> 01:20:02.723 And I think in our previous models, we probably would have had to babysit that a 01:20:02.763 --> 01:20:05.842 lot more and and and give much more guidance on how things should look. 01:20:05.963 --> 01:20:09.243 Um, and so another one that I think is just a cool example of of what you can do. 01:20:09.763 --> 01:20:13.606 Alex Volkov: I wanna, I wanna go back to here with Omar as well, and maybe both of you can chime 01:20:13.646 --> 01:20:17.465 in. Uh, the token output changed as well, like, Argon is like 01:20:18.086 --> 01:20:19.606 generates output significantly. 01:20:20.206 --> 01:20:24.346 Could you tell us how this affects like demos of like this on other stuff? 01:20:24.445 --> 01:20:25.726 Omar, would love to hear from you as well. 01:20:26.246 --> 01:20:30.005 How does that improve people's kind of like out 01:20:30.686 --> 01:20:32.165 results? Specifically because 01:20:34.410 --> 01:20:37.710 if you think about kind of like evaluating models, a lot of the times, uh, the folks 01:20:37.750 --> 01:20:38.270 are focusing on 01:20:39.193 --> 01:20:43.652 the Pareto frontier corner where reasoning used to take a long, long, long time, and 01:20:43.793 --> 01:20:45.952 it's just like with more reasoning you get better results. 01:20:46.273 --> 01:20:49.493 And then there was a little bit of a movement towards improving that so that 01:20:49.553 --> 01:20:52.633 the model will give great results, but still not reason as much and not waste as, 01:20:52.713 --> 01:20:53.392 as many tokens. 01:20:53.993 --> 01:20:57.213 But this is not about that. This is like about the model being able to generate 01:20:57.233 --> 01:20:58.472 like long, long things. 01:20:58.513 --> 01:21:00.133 So could you tell us about the outputs a little bit? 01:21:01.553 --> 01:21:06.533 Omar Sanseviero: I mean, this is mostly for long horizon or scientific projects where you do want the 01:21:06.593 --> 01:21:07.233 model to do 01:21:08.073 --> 01:21:10.192 lots and lots of reasoning, eh, 01:21:11.073 --> 01:21:15.153 and that's pretty much the kind of task where you would need that kind of context. 01:21:15.233 --> 01:21:17.992 In the day-to-day, most likely you do not need that level of output. 01:21:18.073 --> 01:21:22.432 For input maybe, right? Like if you want to pass a very long video or lots of PDFs, 01:21:22.553 --> 01:21:24.392 you may need to do that, but 01:21:25.313 --> 01:21:27.432 this is mostly for scientific or long horizon. 01:21:28.713 --> 01:21:31.973 Alex Volkov: That was awesome. And Geneviève, did it affect any of of the work that you did? 01:21:32.653 --> 01:21:36.333 Geneviève Huskens: Um, no, I, I agree with Omar that in most cases, like, yes, you could write one or 01:21:36.413 --> 01:21:39.692 several books in one prompt, but that doesn't end up being super useful. 01:21:39.773 --> 01:21:42.253 So to us, it's mostly on the, on the very long reasoning. 01:21:42.693 --> 01:21:45.533 One project that I have been working on in the background, but it needs a bit more 01:21:45.693 --> 01:21:49.593 work, is one 3D project where I'm modeling 55 cities in one go. 01:21:49.933 --> 01:21:53.272 And so whenever it sends off to the sub- agents, then they all start building many, 01:21:53.293 --> 01:21:54.373 many cities at the same time. 01:21:54.733 --> 01:21:56.612 Uh, but again, in most engineering applications, 01:21:57.453 --> 01:22:00.453 it's still much better to just do things in slightly smaller batches. 01:22:00.573 --> 01:22:04.633 But if you have to think about a very long, uh, difficult math proof for a long 01:22:04.693 --> 01:22:08.133 time, then you don't want the output window to be constricted by the fact that 01:22:08.153 --> 01:22:10.092 the model already thought for a long time. 01:22:10.173 --> 01:22:12.492 And so that's where we see most application. 01:22:12.653 --> 01:22:16.273 Um, I think it is a very new thing also for us, and so there's also some 01:22:16.333 --> 01:22:19.453 complexity with how this works, even in things like external harnesses. 01:22:19.533 --> 01:22:22.753 But in general, the idea that the scientific community can use this is, is 01:22:22.813 --> 01:22:23.612 very interesting to us. 01:22:24.093 --> 01:22:25.932 Alex Volkov: Yeah, that is very exciting for sure. 01:22:26.213 --> 01:22:29.492 Uh, so we talked about the visual kind of understanding in number one. 01:22:29.573 --> 01:22:32.223 Uh, Omar and Geneviève, I think you mentioned this, that 01:22:33.043 --> 01:22:35.202 now Argon is working on next versions of Argon. 01:22:35.413 --> 01:22:37.292 that suspiciously sounds like RSI. 01:22:37.693 --> 01:22:40.686 And Omar, you're talking about Argon training Gemma models, and that's like, 01:22:40.733 --> 01:22:43.276 parent teacher, not necessarily distillation, but it's also in the process 01:22:43.276 --> 01:22:43.893 of, of training them. 01:22:44.643 --> 01:22:47.923 give us a little bit more of the, I think people do want to know whether or not this 01:22:48.003 --> 01:22:50.982 model can, like, help train other models and how that works. 01:22:51.223 --> 01:22:54.122 Uh, specifically around Gemma work, how much of this was autonomous or how much of 01:22:54.143 --> 01:22:55.223 this is like intervening? 01:22:55.543 --> 01:22:58.962 Uh, could you guys tell, talk to us about model improving other models in the 01:22:58.983 --> 01:22:59.303 future? 01:23:00.423 --> 01:23:06.002 Omar Sanseviero: Yeah, I think just as we think about AI for science, AI for math, AI for biology, 01:23:06.183 --> 01:23:11.602 AI for quantum computing, we can think as well about AI for AI, right? 01:23:11.663 --> 01:23:13.982 And I, I do find this a very exciting area. 01:23:14.423 --> 01:23:18.263 This is not, this is something that we as an ecosystem have been doing for some 01:23:18.383 --> 01:23:22.402 time. It's just that as these models are getting significantly better, it's quite 01:23:22.503 --> 01:23:23.943 exciting to think about ideas. 01:23:23.983 --> 01:23:28.502 So, for example, in the context of Gemma, eh, what Argon did was optimize vLLM, 01:23:28.583 --> 01:23:33.783 which is one of the most popular, eh, open source libraries for server, eh, side 01:23:33.983 --> 01:23:37.943 inference, and it did a bunch of, eh, vLLM optimizations. 01:23:38.022 --> 01:23:38.902 It found a bunch of 01:23:39.783 --> 01:23:41.902 areas where we could im- eh, improve things. 01:23:42.463 --> 01:23:47.303 Similarly, you could see how models could be help on co-designing training code or 01:23:47.383 --> 01:23:48.742 co-designing architectures. 01:23:48.943 --> 01:23:49.283 Eh, 01:23:50.143 --> 01:23:50.743 so yeah. 01:23:52.263 --> 01:23:55.863 Alex Volkov: Uh, Geneviève, anything, any last comments, before we let you guys go? 01:23:55.883 --> 01:23:56.822 No, we're very busy during release weeks. 01:23:56.843 --> 01:24:00.143 Geneviève Huskens: Um, no, I mean, I think, yeah, the sentiment in the market about Gemini has, 01:24:00.443 --> 01:24:02.863 has varied. I would say please give Argon a shot. 01:24:02.963 --> 01:24:06.622 It, to us, feels like a very different model from, uh, the models that we've 01:24:06.683 --> 01:24:10.542 launched up to now, and so I hope everyone gives it a creative shot because there's 01:24:10.583 --> 01:24:12.243 so much, I think, that you, you can do with it. 01:24:12.602 --> 01:24:15.923 Um, for us, I think it's the start of a new era without sounding too commercial, 01:24:15.963 --> 01:24:19.863 but yeah, I think this is a very exciting first step to what hopefully is, uh, it's 01:24:19.883 --> 01:24:21.442 gonna be a couple of fun weeks and months. 01:24:21.803 --> 01:24:24.996 Alex Volkov: We've been waiting, and, we're now very excited to have it back and like 01:24:25.756 --> 01:24:27.236 to see that Google is back. 01:24:27.276 --> 01:24:28.636 So, folks, thank you so much for joining us. 01:24:28.676 --> 01:24:30.716 Geneviève, nice to have you on the podcast. 01:24:30.796 --> 01:24:33.645 Uh, feel free to come back with awesome demos, which we al- always love. 01:24:33.766 --> 01:24:35.866 Omar, as always, friend of the pod for a long, long time. 01:24:36.126 --> 01:24:37.225 Thank you guys for joining us, and 01:24:37.246 --> 01:24:37.746 Geneviève Huskens: Thank you so much. 01:24:37.766 --> 01:24:40.966 Alex Volkov: Congrats on the release, and we know that you have a lot of stuff to catch up on 01:24:41.286 --> 01:24:43.576 during the release week. Uh, we'll see you here, next time. 01:24:43.676 --> 01:24:44.496 Cheers. Thank you so much, guys. 01:24:44.516 --> 01:24:44.935 Geneviève Huskens: See you. Bye-bye. 01:24:44.996 --> 01:24:45.635 Omar Sanseviero: Cheers. Bye-bye. 01:24:46.356 --> 01:24:50.755 Alex Volkov: All right, folks, from one, conversation to another, now we're going to chat, 01:24:51.456 --> 01:24:55.695 uh, now we're moving forward, and I want to introduce you to Nathan Benaich, I- 01:24:55.836 --> 01:24:59.315 I've followed you for the longest time, but I think we've never talked like this 01:24:59.396 --> 01:25:02.035 on video, and I followed the State of AI report. 01:25:02.259 --> 01:25:03.896 so let me just, let me just do this reading. 01:25:04.296 --> 01:25:07.235 Uh, folks, the State of the AI report turns 01:25:07.856 --> 01:25:09.115 9 this year. Na- is that real? 01:25:09.676 --> 01:25:09.936 Nathan Benaich: Yes. 01:25:09.956 --> 01:25:10.035 Alex Volkov: 9 01:25:10.636 --> 01:25:13.646 years. I feel like I've been doing this for 3 and a half years, and I feel like 01:25:13.686 --> 01:25:16.005 this is a long time. Uh, 9 years. 01:25:16.565 --> 01:25:21.265 Uh, the word AI was not familiar to many, like many folks back like 9 years ago. 01:25:21.326 --> 01:25:25.645 Nathan Benaich: Yeah, it was, uh, it was, it was the era of like AI dash ML and data science, if 01:25:25.666 --> 01:25:26.446 you remember that. 01:25:26.886 --> 01:25:32.725 Alex Volkov: And uh, this year is 20 to uh 2 244 slides, a scorecard on last year's 01:25:32.806 --> 01:25:36.646 predictions. Uh, would love to hear about last year's predictions and whether or not 01:25:36.686 --> 01:25:40.205 we can hit them, and 9 new predictions, including AGI 27. 01:25:40.446 --> 01:25:41.325 Nathan Benaich, 01:25:42.006 --> 01:25:45.565 the founder of Air Street Capital, is with us, the author of State of AI Report. 01:25:45.846 --> 01:25:46.286 Um, 01:25:47.085 --> 01:25:47.726 Nathan, welcome. 01:25:48.646 --> 01:25:49.365 Nathan Benaich: Thanks for having me. 01:25:50.126 --> 01:25:52.276 Alex Volkov: Uh, P- Peter also joined us for this interview. 01:25:52.516 --> 01:25:54.076 Peter, thanks so much for making this connection. 01:25:54.516 --> 01:25:54.915 Uh, 01:25:55.766 --> 01:25:57.956 I want to talk about some, like, the highlights, some favorite numbers. 01:25:58.156 --> 01:26:02.835 I have Claude led 27% of measured model R&D at Anthropic in August. 01:26:03.196 --> 01:26:06.996 We just talked with folks, from DeepMind about Argon and, like, RSI, whatever. 01:26:07.436 --> 01:26:09.656 Um, would love to hear from you, Nathan, how this 01:26:10.416 --> 01:26:13.315 kind of like shapes, takes, takes shape between last year and this year. 01:26:13.636 --> 01:26:15.376 People do talk about RSI being a thing. 01:26:15.436 --> 01:26:17.935 People do talk about models training other models, whether or not we're going to hit 01:26:17.956 --> 01:26:20.506 this like, thing. Uh, what are you seeing from there? 01:26:20.566 --> 01:26:25.306 What are the, what are the report says about RSI and continued improvement by the 01:26:25.346 --> 01:26:25.565 models? 01:26:25.606 --> 01:26:27.465 Nathan Benaich: Yeah, yeah, yeah. I would say like the, 01:26:28.366 --> 01:26:32.065 the major story in the last 12 months has been, has been RSI and, and, you know, 01:26:32.166 --> 01:26:35.046 super intelligence, from AGI to super intelligence, uh, 01:26:35.806 --> 01:26:39.846 and then, you know, increasing evidence, uh, you know, at the frontier that shows, 01:26:39.926 --> 01:26:43.086 like, we're kind of making progress towards this idea of, like, AI, building 01:26:43.166 --> 01:26:44.745 AI. I think, um, 01:26:45.646 --> 01:26:48.665 what if, like, the two, the two experiments I found most interesting 01:26:48.766 --> 01:26:53.306 there, one of them you cited, which is just the general stat of, like, how, um, 01:26:53.726 --> 01:26:58.385 Anthropic's, like, R&D team, when they're doing core, uh, R&D work on the model, uh, 01:26:58.566 --> 01:27:03.586 so 26% of, uh, of like the directions in research are being guided by Claude. 01:27:03.966 --> 01:27:04.125 Alex Volkov: Mm. 01:27:04.526 --> 01:27:07.125 Nathan Benaich: And that's up from basically 0 a couple of months ago. 01:27:07.766 --> 01:27:07.906 Alex Volkov: As 01:27:08.526 --> 01:27:12.265 not to oppose this, but I think the code generation stat that I remember, maybe you 01:27:12.306 --> 01:27:16.405 have an updated one, but as far as I remember, like 80 or 85% of code is 01:27:16.646 --> 01:27:18.025 generated by Claude, maybe not directed. 01:27:18.326 --> 01:27:18.545 So- 01:27:18.686 --> 01:27:19.745 Nathan Benaich: Yeah, yeah, yeah. 01:27:19.886 --> 01:27:22.065 So that, yeah, that was like a few, a few months earlier. 01:27:22.126 --> 01:27:25.566 You could see like a chart of like code commits that were, that were basically 01:27:25.586 --> 01:27:29.745 accepted by engineers, uh, as a function of like how models have been changing, and 01:27:29.766 --> 01:27:32.645 it was sort of like flat for a while, and then like the Mythos class models came, 01:27:32.726 --> 01:27:36.446 and then now it's like 8x the, the average, and you see a similar stat at 01:27:36.486 --> 01:27:37.045 OpenAI. 01:27:37.726 --> 01:27:38.045 Um, 01:27:38.886 --> 01:27:41.885 and then the other, uh, the other data point that I, that I like that we made a 01:27:41.926 --> 01:27:45.986 slide on is this, this like, um, experimental setup where a human 01:27:46.046 --> 01:27:50.825 researcher engages on a, on a, on a research path, um, kind of goes through 01:27:50.886 --> 01:27:54.165 like several leaves down like the node of that research path, and at some point the 01:27:54.246 --> 01:27:58.325 path forks, and then one of the fork is a model that continues the research, and 01:27:58.346 --> 01:28:00.566 then the other fork is a human that continues the research. 01:28:01.086 --> 01:28:05.765 And at the end of like both of those leaves, uh, a human rates, uh, which one 01:28:05.806 --> 01:28:06.165 was better. 01:28:06.526 --> 01:28:06.705 Alex Volkov: Mm. 01:28:07.086 --> 01:28:11.205 Nathan Benaich: And um, and you saw that like the old, old school models were definitely not better. 01:28:11.286 --> 01:28:15.165 Like, I think 10% of the time humans prefer them, but then the Mythos class 01:28:15.246 --> 01:28:20.025 model, it was like 60% of the time humans said the Mythos, uh, suggestions were 01:28:20.046 --> 01:28:24.546 better than humans. So I think that's like also, um, it's also interesting, and it 01:28:24.606 --> 01:28:26.686 gets to this point about scientific taste, 01:28:27.366 --> 01:28:31.505 um, which I, which I think is like on the, on the tip of everybody's tongue, like no 01:28:31.526 --> 01:28:32.065 pun intended. 01:28:32.606 --> 01:28:33.726 Alex Volkov: Scientific taste. 01:28:34.686 --> 01:28:35.405 That's incredible. 01:28:36.006 --> 01:28:39.525 Nathan Benaich: Yeah, just, just the idea for the, for, for those who are like less familiar. 01:28:41.046 --> 01:28:46.205 It's like, um, I, I did a PhD in biology, and like what I remember were like the 01:28:46.286 --> 01:28:49.405 great papers were the ones where you would, the grad student or the postdoc 01:28:49.526 --> 01:28:53.666 would, uh, would be, uh, running a couple of experiments and then every week going 01:28:53.706 --> 01:28:56.946 to their supervisor and showing the experiments to the supervisor who would 01:28:57.326 --> 01:29:01.106 look at the work and, you know, kind of mull it over and say like, okay, the next, 01:29:01.166 --> 01:29:03.726 the next best thing you should do for the story is this. 01:29:04.405 --> 01:29:08.525 And I think the best PIs would be the ones who'd suggest experiments or ways to 01:29:09.046 --> 01:29:12.505 interpret the experiments that you did and the results you got in a way that you 01:29:12.606 --> 01:29:16.565 thought was like, like novel and, and interesting and elegant. 01:29:17.246 --> 01:29:18.545 And then the bad PIs, like, 01:29:19.166 --> 01:29:23.206 do that very poorly. And then you see this in the result of a paper where, like, the 01:29:23.286 --> 01:29:24.546 story kind of reads beautifully. 01:29:24.606 --> 01:29:29.546 It's very logical, like each experiment adds additional signal that the previous 01:29:29.566 --> 01:29:34.045 one didn't have. And I think that's like what people are trying to encompass in, 01:29:34.246 --> 01:29:35.525 uh, what they call taste. 01:29:36.246 --> 01:29:38.085 So can a model like approximate that? 01:29:40.006 --> 01:29:41.866 Peter Gostev: Yeah, I think there were, there was one more 01:29:42.726 --> 01:29:45.186 from Prime Intellect, I think, where it w- 01:29:45.206 --> 01:29:45.306 Nathan Benaich: Yeah. 01:29:45.326 --> 01:29:47.786 Peter Gostev: it was the Nano GPT, right? 01:29:47.926 --> 01:29:48.106 Where 01:29:48.286 --> 01:29:48.445 Nathan Benaich: Yeah. 01:29:48.806 --> 01:29:52.025 Peter Gostev: where, where it sort of did it almost get to the human level, right? 01:29:52.326 --> 01:29:52.465 In 01:29:52.845 --> 01:29:52.945 Nathan Benaich: Yeah. 01:29:53.206 --> 01:29:56.306 Peter Gostev: but I think there was something that it was still narrow, the the scope of the 01:29:56.405 --> 01:29:57.325 work that it was doing. 01:29:57.845 --> 01:30:02.306 Nathan Benaich: Yeah, yeah. There's a, like, to to get into this more, what I, there was um a 01:30:02.405 --> 01:30:07.905 study uh from OpenAI in the report where they looked at trying to categorize, like, 01:30:07.966 --> 01:30:11.486 the the token usage of their uh of their engineers and researchers. 01:30:12.126 --> 01:30:12.445 Um, 01:30:13.166 --> 01:30:18.166 and um and what they did was like identify some tokens that were about uh 01:30:19.166 --> 01:30:22.946 helping decide the research direction, what experiments should you run to test 01:30:23.006 --> 01:30:26.546 that research, and then other kind of category of tokens, which is like 01:30:26.966 --> 01:30:30.106 implementing those ideas, like designing the infrastructure, engineering the 01:30:30.126 --> 01:30:33.706 infrastructure, doing the data pipelines, and doing the data analysis, then just 01:30:33.806 --> 01:30:35.325 writing the results of the study. 01:30:35.845 --> 01:30:39.766 And the vast majority of the token usage was on just like the raw engineering of 01:30:39.806 --> 01:30:44.206 just like get the thing working and then writing about it, and very, very small 01:30:44.326 --> 01:30:45.485 amount of tokens was on 01:30:46.206 --> 01:30:49.086 what research should I do in the first place, and like, how should I test it? 01:30:50.006 --> 01:30:53.486 And um, and so mine, I think there's different ways you can interpret this, but 01:30:53.726 --> 01:30:57.085 but one of the interpretations to me is there's still like so much more to go on 01:30:57.206 --> 01:30:57.706 improving, 01:30:58.326 --> 01:31:02.926 uh, you know, the use of models for like ideation in brand new id- brand new areas. 01:31:04.446 --> 01:31:06.885 Alex Volkov: Um, you also gave yourself a scorecard. 01:31:07.126 --> 01:31:08.425 Can we talk about that from last year's 01:31:08.446 --> 01:31:08.585 Nathan Benaich: Yeah. 01:31:08.606 --> 01:31:11.785 Alex Volkov: predictions and misses? Uh, what stuff did you hit correctly? 01:31:11.846 --> 01:31:14.566 What's the, or like the report hit correctly, and I'm assuming that's you as 01:31:14.606 --> 01:31:15.526 well. Uh, what stuff 01:31:15.546 --> 01:31:15.745 Nathan Benaich: Yeah. 01:31:15.766 --> 01:31:16.586 Alex Volkov: did you miss? 01:31:17.406 --> 01:31:17.586 Nathan Benaich: Yeah. 01:31:17.606 --> 01:31:18.245 Alex Volkov: Would love to hear. 01:31:18.366 --> 01:31:18.805 Nathan Benaich: Um, 01:31:19.526 --> 01:31:21.706 yeah, like the stuff I got right was, um, 01:31:22.526 --> 01:31:26.125 that there'd be a pushback from big labs and neo labs into open source. 01:31:27.086 --> 01:31:31.646 And this is in, and um, i- in the US and the West, and the idea with that is it was 01:31:31.686 --> 01:31:35.726 very clearly part of the national AI strategy from the Trump administration to 01:31:36.086 --> 01:31:40.505 sort of regain, regain supremacy of, um, of American AI, and like open source is a 01:31:40.546 --> 01:31:44.766 way of doing that. And very soon after we published that report, like Reflection AI 01:31:45.046 --> 01:31:48.486 sort of went from the coding agent work that they were doing towards saying we're 01:31:48.506 --> 01:31:49.425 going to publish an open 01:31:49.926 --> 01:31:50.065 Alex Volkov: Yeah. 01:31:50.286 --> 01:31:53.425 Nathan Benaich: open weights model, and then NVIDIA kind of supercharged all that. 01:31:54.646 --> 01:31:54.985 Um. 01:31:55.246 --> 01:31:57.966 Alex Volkov: By s- by by basically, I think it's rumored. 01:31:58.026 --> 01:31:59.126 I'm not sure if this is quite 01:31:59.146 --> 01:31:59.345 Nathan Benaich: Yeah. 01:31:59.426 --> 01:32:00.426 Alex Volkov: there yet, but by 01:32:01.146 --> 01:32:05.450 saying that NVIDIA is going to maybe like get reflection in there after 01:32:05.550 --> 01:32:05.710 Nathan Benaich: Yeah. 01:32:05.910 --> 01:32:07.489 Alex Volkov: getting Hugging Face in there also this year. 01:32:07.550 --> 01:32:08.429 So yeah, you're kind of 01:32:08.570 --> 01:32:09.210 Nathan Benaich: And Poolside. 01:32:09.550 --> 01:32:11.559 Alex Volkov: And Poolside and other potential ones. 01:32:11.660 --> 01:32:14.000 And uh, so you're bang on on that as well. 01:32:14.020 --> 01:32:16.920 Nathan Benaich: Yeah. So that one, and then the, the second one was um, 01:32:17.740 --> 01:32:21.799 on NIMBYism for data centers, the idea of like, not in my backyard, like I don't 01:32:21.820 --> 01:32:25.860 want to have a data center next to my garden, um, that this would be a real 01:32:25.980 --> 01:32:26.279 problem 01:32:26.900 --> 01:32:27.539 in the, 01:32:28.499 --> 01:32:31.340 yeah, you know, hampering like the build out of data centers, but also influencing 01:32:31.460 --> 01:32:33.659 part of the midterms, which we'll see in the next couple of weeks. 01:32:34.460 --> 01:32:38.940 Um, we substantiated this stat with like a survey that we found where Americans say 01:32:38.980 --> 01:32:43.159 that they're much, that they much prefer living near a nuclear power plant than a 01:32:43.260 --> 01:32:43.739 data center, 01:32:44.660 --> 01:32:46.119 which is, uh, quite wild. 01:32:46.740 --> 01:32:49.920 Alex Volkov: Wild, yeah. we covered this on the show as well, and we still can't believe it. 01:32:50.180 --> 01:32:50.359 It's 01:32:50.460 --> 01:32:50.839 Nathan Benaich: Yeah. 01:32:51.383 --> 01:32:52.049 Alex Volkov: It's, it's just, 01:32:52.930 --> 01:32:53.489 oof, the amount of 01:32:53.510 --> 01:32:53.769 Nathan Benaich: Yeah. 01:32:53.790 --> 01:32:56.970 Alex Volkov: that are going on and just people on the street talking about just, AI boiling the 01:32:57.070 --> 01:32:57.969 planet, uh. 01:32:58.150 --> 01:33:01.129 Nathan Benaich: Yeah, yeah, yeah. And it's not just like in the US, it's certainly happening in 01:33:01.230 --> 01:33:03.590 Europe and in Korea and in other places where, 01:33:04.230 --> 01:33:06.709 you know, there's been like a small proportion of the population that's 01:33:06.790 --> 01:33:10.629 benefited like enormously from AI and, and many people who haven't, and, 01:33:11.310 --> 01:33:13.310 um, you know, there's some social issues with that. 01:33:13.830 --> 01:33:17.129 Uh, I think the last, the, the last one that I got right was that, um, you know, 01:33:17.190 --> 01:33:21.409 Chinese labs would overtake US ones, uh, in some major leaderboards, and, you know, 01:33:21.430 --> 01:33:24.270 I think it depends on, you know, which, which timeline you actually look at, and 01:33:24.430 --> 01:33:29.190 Peter probably knows this by heart, but, um, but certainly, you know, Kimi and GLM 01:33:29.270 --> 01:33:30.629 and others have done really, really well. 01:33:31.510 --> 01:33:31.790 Alex Volkov: Yeah. 01:33:32.790 --> 01:33:33.989 What about some of the misses? 01:33:34.950 --> 01:33:36.490 What did you predict and didn't 01:33:36.670 --> 01:33:36.949 Nathan Benaich: Yeah. 01:33:37.590 --> 01:33:38.749 Yeah, yeah. Well, um, 01:33:39.510 --> 01:33:43.149 there's been so much talk about, like, AI sovereignty, which could be interpreted in 01:33:43.190 --> 01:33:47.409 different ways, but either broadly speaking, the one of I, I as a country 01:33:47.510 --> 01:33:48.009 want to have 01:33:48.630 --> 01:33:52.769 independence or control over my own destiny when it comes to, uh, building 01:33:52.870 --> 01:33:57.610 models, um, uh, training them, serving them, having the infrastructure for it and 01:33:57.630 --> 01:33:58.749 the energy, et cetera. 01:34:00.029 --> 01:34:04.869 And, um, and my bet was w- there would be this concept of like, uh, you know, I'm AI 01:34:04.990 --> 01:34:10.529 neutrality in that that kind of mirrors like, uh, uh, sort of mutual defense, um, 01:34:10.670 --> 01:34:14.590 agreements where a country is like too small to be able to afford, uh, you know, 01:34:14.630 --> 01:34:17.909 its own military or its own defenses in the same way it's like too small to be 01:34:17.930 --> 01:34:21.389 able to build its own data centers and have people build models, etcetera. 01:34:21.750 --> 01:34:25.970 So it just forms like a strategic like defense, uh, you know, agreement, uh, or 01:34:26.150 --> 01:34:26.450 AI, 01:34:27.110 --> 01:34:30.929 um, provision agreement with a bigger neighbor or an ally that says, you know, I 01:34:30.950 --> 01:34:34.850 will, I will, I'll promise to give you all, all of the AI in exchange for 01:34:34.910 --> 01:34:35.470 something else. 01:34:36.190 --> 01:34:38.809 Um, I haven't quite seen something like that yet. 01:34:38.910 --> 01:34:41.929 There's been a bit of a move with the, like, US Pax Silica, 01:34:42.630 --> 01:34:47.789 um, policy that, um, that tries to, like, bring, um, a collection of countries that 01:34:47.910 --> 01:34:48.789 have, uh, 01:34:50.390 --> 01:34:54.889 access to core, um, parts of the chip supply chain, whether it's like minerals 01:34:54.990 --> 01:34:57.470 or fabs or lithography or whatever. 01:34:58.070 --> 01:35:02.949 Um, but, but overall the sort of AI neutrality thing hasn't really taken on 01:35:03.030 --> 01:35:03.389 just yet. 01:35:03.910 --> 01:35:04.569 Alex Volkov: No, not yet. 01:35:05.070 --> 01:35:05.429 Nathan Benaich: Um, 01:35:06.030 --> 01:35:08.810 and overall, I'd say the predictions are like directionally well, like this is a 01:35:08.830 --> 01:35:12.550 guy on LinkedIn who like looked at the last like 9 years, uh, and then tried to 01:35:12.590 --> 01:35:15.610 see like over that 9 years which report which pro- which what portion of all 01:35:15.730 --> 01:35:16.870 predictions eventually happened. 01:35:16.950 --> 01:35:18.429 It was, I think, like, over 2/3. 01:35:19.190 --> 01:35:22.690 And so sometimes it's just a question of timing, more than just like, this is 01:35:22.830 --> 01:35:23.349 ludicrous. 01:35:23.990 --> 01:35:27.110 Peter Gostev: Well, well, that that's what I like about your prediction is that I think it's quite 01:35:27.190 --> 01:35:31.090 easy to make predictions that, uh, you're gonna nail, like, 90% correct. 01:35:31.230 --> 01:35:33.329 You just kind of predict roughly what just happened. 01:35:33.350 --> 01:35:33.449 Nathan Benaich: Yeah. 01:35:33.470 --> 01:35:36.229 Peter Gostev: Oh, you know, Anthropic is gonna make more money next year. 01:35:36.310 --> 01:35:37.770 You know, it's like, okay, that's not 01:35:37.830 --> 01:35:37.929 Nathan Benaich: Yeah. 01:35:37.950 --> 01:35:39.090 Peter Gostev: that's not very interesting. So, 01:35:39.110 --> 01:35:39.409 Nathan Benaich: Yeah. 01:35:39.430 --> 01:35:41.569 Peter Gostev: yeah, I find it interesting you got a hit rate of like 01:35:41.590 --> 01:35:41.690 Nathan Benaich: Yeah. 01:35:41.710 --> 01:35:42.429 Peter Gostev: 50-50. 01:35:42.990 --> 01:35:43.870 Nathan Benaich: Yeah. Yeah. 01:35:44.030 --> 01:35:47.310 Peter Gostev: So is it like how, is it difficult to to come up with the predictions? 01:35:47.390 --> 01:35:51.530 Are there some that you you kind of decided against, I don't know, too wild 01:35:51.790 --> 01:35:53.789 that that you didn't end up making this year? 01:35:54.790 --> 01:35:55.830 Nathan Benaich: Yeah, um, 01:35:57.750 --> 01:36:00.310 there's one I couldn't quite formulate around, um, 01:36:01.150 --> 01:36:06.569 around, uh, one is like cybersecurity and like, can AI somehow disrupt, um, 01:36:06.830 --> 01:36:08.629 encryption on the internet, um, 01:36:09.310 --> 01:36:13.470 as it might be, uh, as that might be accelerated by the fact that AI solves 01:36:13.590 --> 01:36:13.909 math. 01:36:14.989 --> 01:36:18.930 So I don't quite know how to, yeah, basically I just ran out of time for like 01:36:19.630 --> 01:36:22.650 thinking about how to formulate it properly to be able to defend it, 01:36:22.702 --> 01:36:27.342 Peter Gostev: Can I, can I ask you, so you, you speak to all kinds of people, and we're a little 01:36:27.422 --> 01:36:30.821 bit in the AI engineer kind of bubble. 01:36:30.942 --> 01:36:33.861 Like, what, what are sort of things that people ask you about? 01:36:34.102 --> 01:36:38.382 Because you've got different parts of the report about even going into politics and, 01:36:38.422 --> 01:36:42.661 and things like that. What do ki- what do you hear people care, caring about? 01:36:45.862 --> 01:36:46.442 Nathan Benaich: Um... 01:36:48.302 --> 01:36:51.301 There's a segment of people who really care about what we discussed, like the, 01:36:51.342 --> 01:36:55.241 the RSI, like, rapidly kind of accelerating progress, like, what does 01:36:55.302 --> 01:36:57.822 that mean for the, the rest of us, basically? 01:36:58.102 --> 01:36:58.462 Um, 01:36:59.181 --> 01:37:04.641 there's, there's some who, um, you know, are just like increasingly like, uh, 01:37:04.862 --> 01:37:10.722 pessimistic over like software companies still, because, um, you know, now like if 01:37:10.742 --> 01:37:15.041 you kind of believe the RSI is coming and basically the, the quality of the software 01:37:15.102 --> 01:37:19.002 you can produce or the work you can produce is like very limited by what model 01:37:19.222 --> 01:37:21.362 you have and how much money you're willing to spend on it. 01:37:21.502 --> 01:37:21.741 Then, 01:37:22.342 --> 01:37:26.162 um, there's certain people that will just like, or organizations that'll accelerate 01:37:26.262 --> 01:37:28.822 way faster than others, and if you're not part of that, like, what do you do? 01:37:29.822 --> 01:37:30.341 Um, 01:37:31.422 --> 01:37:34.281 there's always like chatter, you know, every 2 weeks or something, and I think 01:37:34.342 --> 01:37:38.202 now we're in one of those negative cycles around like, is there too much compute, 01:37:38.502 --> 01:37:42.241 circular financing, like how is it financed, like is the debt going to be 01:37:42.282 --> 01:37:45.781 able to be serviced? And, you know, there's a bloodbath in NeoClouds in the 01:37:45.822 --> 01:37:47.702 last, like, 2, 3 days because 01:37:48.342 --> 01:37:49.662 we're just in that part of the cycle. 01:37:50.621 --> 01:37:51.361 Um, 01:37:52.862 --> 01:37:53.322 um, 01:37:53.982 --> 01:37:58.002 you know, there's always, always debate around, like, will, will NeoLabs monetize? 01:37:58.062 --> 01:38:01.642 How will they monetize? Or these, like, funky financing structures with multiple 01:38:01.782 --> 01:38:05.062 stages at different prices, all within a couple minutes of each other. 01:38:05.181 --> 01:38:06.262 Is that like a sign of, 01:38:07.142 --> 01:38:10.901 um, I don't know, just like rabid demand and not very much thinking? 01:38:11.862 --> 01:38:12.342 Um. 01:38:13.542 --> 01:38:17.341 Alex Volkov: How about normies? I have to ask you, Nathan, how about like just regular folks 01:38:17.382 --> 01:38:18.921 on the street and how they view AI? 01:38:18.982 --> 01:38:20.221 Do you track that in state of AI? 01:38:20.262 --> 01:38:23.101 Do you talk to them? How do you see that opinion shapes? 01:38:23.382 --> 01:38:24.132 So we did talk about 01:38:24.152 --> 01:38:24.252 Nathan Benaich: Yeah. 01:38:24.272 --> 01:38:26.671 Alex Volkov: the change in the public opinion about water use, et cetera. 01:38:26.952 --> 01:38:29.211 Uh, what about the kind of the use case for AI? 01:38:29.272 --> 01:38:30.631 What do you see there? What do you track there? 01:38:30.872 --> 01:38:34.611 The rise of personal agents is now upon us, Gokbots and Muse and all of those. 01:38:34.992 --> 01:38:35.272 Do you 01:38:35.992 --> 01:38:36.891 consider those in the report? 01:38:38.112 --> 01:38:42.171 Nathan Benaich: Yeah, but it that's still like in the small minority of tech people who know, 01:38:42.232 --> 01:38:43.831 who know what it is. Um, 01:38:44.512 --> 01:38:48.831 you know, I would, I would still bet that like 90% of people are using like a model 01:38:48.912 --> 01:38:49.032 to 01:38:49.712 --> 01:38:51.791 maybe at best like read their email or 01:38:51.832 --> 01:38:52.052 Alex Volkov: Mm. 01:38:52.072 --> 01:38:53.231 Nathan Benaich: answer to their email or, 01:38:53.992 --> 01:38:57.572 you know, ask questions in the same way they would ask Google, but they probably 01:38:57.632 --> 01:38:58.711 never click the work button. 01:38:58.752 --> 01:39:01.851 Like, what does that even mean? Like, do I want to open a consumer app and, like, 01:39:01.992 --> 01:39:02.231 work? 01:39:03.112 --> 01:39:03.292 Alex Volkov: Yeah. 01:39:03.352 --> 01:39:03.871 Nathan Benaich: Not really. 01:39:04.952 --> 01:39:05.471 You know, and 01:39:06.272 --> 01:39:09.492 most people don't even know you can swipe left, open Codex, run it on your machine, 01:39:09.552 --> 01:39:11.832 and like talk to your machine and make it do work for you. 01:39:11.872 --> 01:39:12.071 So, 01:39:12.792 --> 01:39:17.052 like, I think that's so far away, which is one of the reasons why, like, I've just 01:39:17.272 --> 01:39:21.111 started saying, like, I think these big labs should open like a Genius Bar or like 01:39:21.172 --> 01:39:25.252 an actual physical store and just like start walking, you know, regular people 01:39:25.372 --> 01:39:25.631 through 01:39:26.232 --> 01:39:28.231 the magical things you can do with AI. 01:39:29.992 --> 01:39:30.131 Alex Volkov: That, 01:39:30.752 --> 01:39:34.381 this is the kind of also the geek squad genius bar comment, right? 01:39:34.582 --> 01:39:34.961 Nathan Benaich: Exactly. 01:39:35.362 --> 01:39:38.162 Alex Volkov: They need hand holding and imagination unlock. 01:39:38.522 --> 01:39:38.881 Um. 01:39:39.522 --> 01:39:39.661 Nathan Benaich: Yeah. 01:39:39.722 --> 01:39:43.642 Alex Volkov: And also that's what we're trying to do on the show, though the show does cater to AI 01:39:43.682 --> 01:39:47.441 engineers, and so if you're still listening to this, uh, you are 01:39:48.122 --> 01:39:52.042 in the very few, I don't know if you saw the Karpathy tweet as well about, he kind 01:39:52.062 --> 01:39:52.742 of talks about this as well. 01:39:52.882 --> 01:39:54.562 Nathan Benaich: Yeah, exactly, when he segments the world. 01:39:54.802 --> 01:39:58.161 Exactly. It's, it's, and we're still very much, very much like that. 01:39:58.482 --> 01:40:02.981 Um, and there's an argument that, like, that gap, that gap expands even further 01:40:03.202 --> 01:40:06.401 if, like, capabilities get better and RSI happens and all that stuff. 01:40:06.962 --> 01:40:10.261 But you can see it a bit in, like, the, you know, the ramp data that we show with 01:40:10.282 --> 01:40:14.582 the 70,000 US customers that they have and tracking how much money they spend on 01:40:14.682 --> 01:40:15.041 models. 01:40:15.762 --> 01:40:19.021 It's like, you know, the the average spends like 12 bucks per employee per 01:40:19.122 --> 01:40:22.982 month, and the top 1% is like 7,500, roughly speaking. 01:40:23.642 --> 01:40:25.601 So it's like 580 times difference. 01:40:27.082 --> 01:40:27.722 Alex Volkov: Between the 01:40:28.442 --> 01:40:31.582 potential permanent underclass, if this continues, yes. 01:40:31.842 --> 01:40:31.981 Nathan Benaich: Right. 01:40:32.041 --> 01:40:34.305 Alex Volkov: And and and the top spenders, the billion token ones. 01:40:34.382 --> 01:40:34.701 Nathan, 01:40:34.822 --> 01:40:34.961 Nathan Benaich: Right. 01:40:34.982 --> 01:40:35.902 Alex Volkov: Thank you so much for joining. 01:40:36.902 --> 01:40:40.361 The thing that I do want to, ask you, because you've been doing this for a long 01:40:40.422 --> 01:40:40.662 time, 01:40:41.352 --> 01:40:44.361 again, from AI ML days to then generative AI to, know, 01:40:44.382 --> 01:40:44.562 Nathan Benaich: Yeah. 01:40:44.582 --> 01:40:44.802 Alex Volkov: uh, 01:40:45.662 --> 01:40:47.662 very small context windows to the huge ones. 01:40:47.702 --> 01:40:49.721 We've been doing this for 3 years, and it feels like incredible time. 01:40:49.742 --> 01:40:52.001 You've been doing this for, for, for much, much longer. 01:40:52.342 --> 01:40:52.481 Nathan Benaich: Yeah. 01:40:52.542 --> 01:40:52.861 Alex Volkov: Um, 01:40:53.702 --> 01:40:54.501 your personal 01:40:55.702 --> 01:40:56.101 beliefs 01:40:56.942 --> 01:41:00.702 and whether or not, your P doom stuff, I don't like the P doom, nomenclature, 01:41:00.722 --> 01:41:00.902 Nathan Benaich: Yeah. 01:41:00.922 --> 01:41:02.117 Alex Volkov: but about the bubble stuff. 01:41:02.238 --> 01:41:04.497 I would love to just hear from you, how do you see 01:41:05.277 --> 01:41:06.918 your views change throughout these years? 01:41:07.438 --> 01:41:07.738 If you can 01:41:07.878 --> 01:41:07.978 Nathan Benaich: Yeah. 01:41:08.038 --> 01:41:09.497 Alex Volkov: chat about this for a little bit briefly. 01:41:10.118 --> 01:41:12.498 Nathan Benaich: Yeah. I, I think the major... 01:41:13.078 --> 01:41:14.938 Well, so I, my P doom is very small. 01:41:15.358 --> 01:41:15.578 Alex Volkov: Mm-hmm. 01:41:15.598 --> 01:41:17.757 Nathan Benaich: I'm not like a big believer in X risk. 01:41:18.397 --> 01:41:19.037 Um, 01:41:19.958 --> 01:41:24.978 I, I, I do think along that angle that cybersecurity is very precarious, um, and 01:41:25.078 --> 01:41:26.917 should be taken like insanely seriously, 01:41:27.758 --> 01:41:30.678 but I think that's different than like existential risk of like destroying 01:41:30.758 --> 01:41:31.238 humanity. 01:41:31.998 --> 01:41:32.438 Um, 01:41:33.998 --> 01:41:38.418 I think the- the major change I've had in the last like 2, 3 years is 01:41:39.078 --> 01:41:39.857 I, I've generally 01:41:40.638 --> 01:41:42.737 believed that you use the right model for the task 01:41:43.438 --> 01:41:45.978 in the same way that, you know, you don't, you don't drive your Ferrari to go to, 01:41:45.998 --> 01:41:48.797 like, the coffee shop, or maybe you do to show off or something, but it's like not 01:41:48.838 --> 01:41:50.678 the most reasonable thing to do, just like walk. 01:41:51.118 --> 01:41:51.278 Alex Volkov: Yeah. 01:41:51.438 --> 01:41:51.998 Nathan Benaich: Um, 01:41:53.038 --> 01:41:56.517 but I think there's like a weird behavior with like, w- with like frontier 01:41:56.558 --> 01:42:00.397 intelligence. If you, if you have like 200 bucks a month to spend and you value the 01:42:00.478 --> 01:42:04.197 quality of your work, do you really want to like take the risk of putting the 01:42:04.278 --> 01:42:06.657 slider down to low and like using a cheaper model 01:42:07.518 --> 01:42:09.597 on the chance that like the quality of the work is worse? 01:42:10.558 --> 01:42:14.698 So while I could switch for, like, emails or whatever, I j- I just don't. 01:42:14.798 --> 01:42:16.698 I'm just like on Astra High, like, who cares? 01:42:17.198 --> 01:42:18.698 Alex Volkov: You're answering emails on Astra High. 01:42:18.718 --> 01:42:18.977 Nathan Benaich: Yeah. 01:42:18.998 --> 01:42:21.198 Alex Volkov: Nathan Benaich from Air Street Capital. 01:42:23.598 --> 01:42:26.217 Nathan Benaich: Not all my emails, but like the ones we have to think about it, like. 01:42:26.358 --> 01:42:26.537 Alex Volkov: Yeah. 01:42:26.918 --> 01:42:29.958 Nathan Benaich: You know, why, why would you take the risk using a shittier, a shittier system? 01:42:30.318 --> 01:42:30.498 Alex Volkov: Mhm. 01:42:30.678 --> 01:42:32.357 Nathan Benaich: Um, or potentially shittier system. 01:42:32.998 --> 01:42:34.118 So, um, 01:42:35.638 --> 01:42:38.278 w- yeah, so there's, there's that, and then like, you know, will we ever make 01:42:38.358 --> 01:42:39.717 money on frontier intelligence? 01:42:39.798 --> 01:42:43.257 And I think it's starting to show that, like, we will, because the value of the 01:42:43.358 --> 01:42:45.698 task that they're doing is increasingly high, and just 01:42:46.318 --> 01:42:49.898 this, the more people from all sorts of different parts of the technology world 01:42:49.918 --> 01:42:53.817 who come into working in AI bring all of their knowledge and optimizations to, to 01:42:53.838 --> 01:42:54.637 the fore, and 01:42:55.278 --> 01:42:58.597 just, you know, human ingenuity plus lots of AI equals more progress. 01:42:58.638 --> 01:43:03.398 So I'm like quite, um, quite bullish on like margins improving, and you know, I 01:43:03.418 --> 01:43:06.558 don't know what Anthropic is adjusting when they report their adjusted numbers, 01:43:06.678 --> 01:43:06.837 but 01:43:07.638 --> 01:43:08.037 um, 01:43:08.638 --> 01:43:11.698 it's, you know, it's probably not like community adjusted EBITDA like WeWork. 01:43:11.758 --> 01:43:13.398 Like, I think it's a serious company, so 01:43:14.158 --> 01:43:14.638 um, 01:43:15.678 --> 01:43:18.317 um, so I'm, I've changed my mind on on on that. 01:43:18.358 --> 01:43:21.837 Like, I think you can really make money on Frontier AI, and I'm increasingly 01:43:21.918 --> 01:43:25.718 convinced that it's the the open source versus like Frontier Labs is like 01:43:26.518 --> 01:43:28.237 um, iOS versus Android. 01:43:28.918 --> 01:43:31.697 It's sort of cool for a while to play around with your Android and mod 01:43:31.758 --> 01:43:33.517 everything, and some people want to do that, but 01:43:33.637 --> 01:43:33.777 Alex Volkov: Yeah. 01:43:34.078 --> 01:43:37.457 Nathan Benaich: At some point, you just want product that auto updates, is like reliable, connects 01:43:37.478 --> 01:43:39.378 with everything, and looks nice, and 01:43:41.158 --> 01:43:41.518 so. 01:43:41.998 --> 01:43:44.707 Alex Volkov: All right, are, those are great stuff. Uh, folks, Nathan Benaich, thank you so much 01:43:44.748 --> 01:43:46.207 for joining, us on Thursday. 01:43:46.268 --> 01:43:49.128 I will always welcome you back after the next one as well. 01:43:49.228 --> 01:43:51.287 If you, are you planning to keep going, to to the- 01:43:51.348 --> 01:43:53.368 Nathan Benaich: Yeah, yeah, yeah, yeah. We'll we'll keep going. 01:43:53.428 --> 01:43:57.068 I mean, you know, if AGI 27 happens, I'll be on the 10th year anniversary, and then 01:43:57.128 --> 01:43:58.928 I might be able to stop, but uh, 01:43:58.948 --> 01:43:59.688 Alex Volkov: All right. 01:43:59.708 --> 01:44:02.387 Nathan Benaich: Let's see. Yeah. In the meantime, you can find everything online. 01:44:02.428 --> 01:44:03.787 It's all free at stateof.ai. 01:44:03.868 --> 01:44:06.708 Alex Volkov: It's at stateof.ai. 01:44:07.188 --> 01:44:07.388 Nathan Benaich: Yes. 01:44:07.411 --> 01:44:09.767 Alex Volkov: so it's a great, URL. Nathan, thank you so much for joining us. 01:44:09.968 --> 01:44:10.607 Nathan Benaich: Yeah, thanks guys. 01:44:11.328 --> 01:44:11.648 Alex Volkov: Bye-bye. 01:44:12.288 --> 01:44:12.688 Um, 01:44:13.338 --> 01:44:17.297 that was super cool. Uh, I don't, we don't often, we don't often get somebody here 01:44:17.757 --> 01:44:19.198 that does this longer than we do. 01:44:19.518 --> 01:44:21.078 Uh, and LDJ joined us as well. 01:44:21.477 --> 01:44:21.877 Uh, 01:44:22.661 --> 01:44:23.208 this is great. 01:44:24.048 --> 01:44:25.768 Nathan, uh, Peter, thank you so much. 01:44:25.968 --> 01:44:26.667 What's your take on- 01:44:27.208 --> 01:44:32.748 Peter Gostev: I, the, the most, the, the very cool thing about this is that there's obviously, I, I 01:44:32.808 --> 01:44:36.448 find this really helpful to keep up with this stuff every week, but there's 01:44:36.528 --> 01:44:38.148 something really valuable, I think, that 01:44:38.808 --> 01:44:43.188 to just look back at the year and you're just like, oh, I completely forgotten 01:44:43.248 --> 01:44:47.487 about this, or I didn't actually realize there were papers there that maybe didn't 01:44:47.528 --> 01:44:50.707 quite make the splash this week, so we skipped it because, you know, there's too 01:44:50.728 --> 01:44:51.307 much to cover. 01:44:51.328 --> 01:44:51.587 Alex Volkov: Yeah. 01:44:51.608 --> 01:44:53.428 Peter Gostev: But actually they turn out to be quite important. 01:44:53.448 --> 01:44:57.428 So I really value his work, and I've been reading his stuff for years now. 01:44:57.528 --> 01:45:02.067 I can't remember what was the first one that I read, but it it's just so useful to 01:45:02.128 --> 01:45:05.007 just look at it and and just read the stuff. 01:45:05.088 --> 01:45:09.507 So I really recommend just book, I don't know, an hour, 2 hours in your diary and 01:45:09.568 --> 01:45:13.087 just scroll for this. You don't have to read every single thing, but just scroll 01:45:13.128 --> 01:45:15.048 for this. I know it's like 250 slides. 01:45:15.448 --> 01:45:17.048 You can do it in an hour quite easily. 01:45:17.128 --> 01:45:19.388 Don't, don't le- don't ask your agent to read this. 01:45:19.488 --> 01:45:20.548 Alex Volkov: Don't ask your agent to read this. 01:45:20.568 --> 01:45:23.028 Peter Gostev: Yeah, just read it, read it, put some effort in. 01:45:23.408 --> 01:45:24.167 No, it's worth it. 01:45:24.563 --> 01:45:27.263 Alex Volkov: Nisten, how's your 100K city going? 01:45:27.323 --> 01:45:28.642 Are you still AI pilled? 01:45:29.083 --> 01:45:33.573 And, Peter, what, what new and interesting did you start using or, or building this 01:45:33.613 --> 01:45:35.872 week? Just whoever wants to jump in, let's just chat. 01:45:37.133 --> 01:45:37.812 Nisten Tahiraj: Hell yeah. 01:45:39.372 --> 01:45:43.612 There's, uh, I, I got, I got some, some, some crazy stuff 01:45:44.293 --> 01:45:46.093 coming up for, uh, release. 01:45:46.853 --> 01:45:47.333 Um, 01:45:48.133 --> 01:45:52.773 it's, uh, yeah, the, the all the entire decompilation of games and stuff is 01:45:52.853 --> 01:45:54.753 enabling full out simulations. 01:45:54.893 --> 01:45:56.052 Alex Volkov: Can we talk about this decompilation? 01:45:56.073 --> 01:45:59.732 So, LDJ, we mentioned this like last time that people are putting things in other 01:45:59.812 --> 01:46:01.753 games, and now there's the whole decompilation thing happening. 01:46:01.773 --> 01:46:05.092 Somebody told me somebody decompiled or reverse engineered the airdrop protocol. 01:46:05.133 --> 01:46:05.572 What the fuck? 01:46:06.093 --> 01:46:07.852 Nisten Tahiraj: Guys, it is crazy. 01:46:07.973 --> 01:46:09.592 Alex Volkov: It's like a renaissance happening right now, right? 01:46:09.823 --> 01:46:10.242 tell us. 01:46:10.263 --> 01:46:12.718 Nisten Tahiraj: Yes. All right, here's, here's what's happened. 01:46:12.997 --> 01:46:17.578 The entire internet the last two months has just gone completely nuts by using, 01:46:17.678 --> 01:46:21.297 like, the NSA Ghidra protocol, which allows you to, like, reverse engineer any 01:46:21.358 --> 01:46:22.117 piece of software. 01:46:22.878 --> 01:46:28.518 And then there was also the law in 1996 from the Phoenix BIOS, or AVS, I might be 01:46:28.557 --> 01:46:30.478 wrong on that, that allows you to 01:46:31.158 --> 01:46:35.597 actually provide, like, visually functional, uh, pieces of software, as 01:46:35.638 --> 01:46:38.117 long as it does not violate any trademark. 01:46:38.958 --> 01:46:44.057 So you see a lot of these things called Photocraft, Gridcraft, and stuff, which 01:46:44.098 --> 01:46:45.318 are like Photoshop 01:46:45.958 --> 01:46:48.218 or Excel builds that are done in Rust. 01:46:48.238 --> 01:46:49.497 Alex Volkov: Clean, clean room builds, yes. 01:46:50.118 --> 01:46:52.217 Nisten Tahiraj: Yes, so those are mainly done in Rust. 01:46:52.273 --> 01:46:55.952 But yeah, it's a complete renaissance in, uh, in apps. 01:46:56.513 --> 01:46:59.772 It's going to be a very interesting next 3 months 01:47:00.433 --> 01:47:04.992 as, uh, entire countries and economies figure out what kind of office suite do 01:47:05.033 --> 01:47:05.793 they need to run. 01:47:06.473 --> 01:47:09.812 Uh, and, and, and the very interesting thing is that actual professionals and 01:47:09.913 --> 01:47:10.332 creatives 01:47:11.113 --> 01:47:16.252 tend to like the vibe-coded versions more because they're just so much more 01:47:16.393 --> 01:47:16.813 portable. 01:47:17.233 --> 01:47:17.852 Alex Volkov: And they're fun. 01:47:18.033 --> 01:47:19.872 Nisten Tahiraj: Yeah, and they're fun. Again, they're not perfect. 01:47:19.953 --> 01:47:23.692 For example, in Photoshop, the reviews were that the Magic Wand tool is not very 01:47:23.753 --> 01:47:27.292 good because it used like an older machine learning model to remove the background. 01:47:27.633 --> 01:47:27.813 Alex Volkov: Mm. 01:47:27.873 --> 01:47:30.713 Nisten Tahiraj: That's going to just be fixed in the next, like, 2 weeks. 01:47:31.153 --> 01:47:34.932 Alex Volkov: The second he turns his slop cannon towards that issue, that issue is solved. 01:47:34.953 --> 01:47:36.193 Nisten Tahiraj: Yes, that issue is solved. 01:47:36.313 --> 01:47:41.532 Yeah, they're just, they're throwing entire gauntlet prompts, and they're, I've 01:47:41.633 --> 01:47:47.172 seen entire apps, the entire Adobe suite, InDesign, After Effects, uh, 01:47:47.313 --> 01:47:51.132 Lightroom, uh, all just running in the browser, very performant too, 01:47:51.873 --> 01:47:56.552 and you can run them without any telemetry or crap or spyware or 01:47:56.753 --> 01:47:59.372 Alex Volkov: Without Adobe paying also a bunch of money to Adobe. 01:47:59.473 --> 01:47:59.813 Nisten, 01:47:59.833 --> 01:48:00.012 Nisten Tahiraj: Yeah. 01:48:00.333 --> 01:48:00.953 Alex Volkov: I do have 01:48:01.793 --> 01:48:05.032 to pause you because LDJ just bought us breaking news. 01:48:05.713 --> 01:48:06.953 Uh, let's go. 01:48:10.193 --> 01:48:11.353 Breaking news stinger: AI breaking news 01:48:13.313 --> 01:48:13.953 coming at you 01:48:15.153 --> 01:48:16.313 only on ThursdAI. 01:48:22.033 --> 01:48:24.513 Alex Volkov: All right, folks, we have, quite exciting breaking news. 01:48:24.673 --> 01:48:25.642 LDJ, you wanna take this one? 01:48:26.683 --> 01:48:26.942 LDJ: Sure. 01:48:27.283 --> 01:48:27.662 Alex Volkov: I can share. 01:48:27.683 --> 01:48:28.463 LDJ: So, uh, 01:48:29.203 --> 01:48:32.822 yeah, I, I do feel like overall these past few months we've had a bit of a 01:48:32.963 --> 01:48:36.282 renaissance of, of Western American open models, 01:48:37.003 --> 01:48:40.843 and Meta just announced, uh, literally within the past 20 minutes, 01:48:41.523 --> 01:48:43.302 that Muse Spark Open 01:48:43.963 --> 01:48:48.362 is released, which I presume is maybe like a partly censored version of Muse Spark. 01:48:48.643 --> 01:48:51.763 It's 816 billion total parameters, 01:48:52.523 --> 01:48:54.622 42 billion active parameters. 01:48:54.683 --> 01:48:55.302 So this is 01:48:56.203 --> 01:49:02.182 around 3 times smaller in total parameter size than, uh, than Kimi K3, while being 01:49:02.283 --> 01:49:05.483 approximately half of the active parameters of Kimi K3. 01:49:06.003 --> 01:49:09.083 So you could expect it to be a lot more practical to run a lot faster. 01:49:09.603 --> 01:49:13.862 Um, I'd say probably in terms of parameter count of active and total, it's probably 01:49:14.563 --> 01:49:17.642 most fair to compare it to maybe the full GLM 5.3, 01:49:18.563 --> 01:49:23.842 and I think it is maybe beating GLM 5.3, though, or at least trading blows with it 01:49:24.003 --> 01:49:27.942 in the benchmarks. So I guess we'll have to scroll down or kind of wait for people 01:49:27.963 --> 01:49:28.922 to benchmark things. 01:49:29.083 --> 01:49:30.402 Alex Volkov: Yeah, let's take a look. Let's take a look. 01:49:30.443 --> 01:49:34.762 They're comparing themselves, uh, Muse Spark Open, to Inkling, to Nemotron, to 01:49:34.923 --> 01:49:38.942 Beam, to Mistral, and only then to Qwen and GLM, right? 01:49:39.003 --> 01:49:40.223 So, like, they're starting with, 01:49:41.347 --> 01:49:43.036 what is it Inkling, US-based? 01:49:43.057 --> 01:49:43.896 LDJ: Thinking Machines. 01:49:44.337 --> 01:49:44.837 Alex Volkov: Yeah, oh yeah, 01:49:44.857 --> 01:49:44.997 LDJ: Yes. 01:49:45.017 --> 01:49:49.936 Alex Volkov: Nice, okay. Uh, so look, they're literally saying, hey, we're comparing ourselves to 01:49:50.017 --> 01:49:54.257 American counterparts first, then to European counterparts, and only then Qwen 01:49:54.297 --> 01:49:55.336 and GLM and Kimi. 01:49:55.537 --> 01:49:59.977 Peter Gostev: Yeah, I, I think what's interesting, if you scroll down into the coding area and 01:50:00.456 --> 01:50:01.917 compare it to the Chinese models, 01:50:02.137 --> 01:50:02.236 Alex Volkov: Mhm. 01:50:02.257 --> 01:50:04.556 Peter Gostev: So we've got the Terminal Bench 2.1. 01:50:04.857 --> 01:50:04.997 Alex Volkov: Yeah. 01:50:05.017 --> 01:50:10.736 Peter Gostev: So Muse Spark has, uh, 76.4, then Qwen 3.8 01:50:10.977 --> 01:50:12.136 has 82, 01:50:12.897 --> 01:50:16.436 and, uh, GLM 5.3 has, uh, 84. 01:50:16.456 --> 01:50:17.196 Alex Volkov: 83. 01:50:17.537 --> 01:50:19.936 Peter Gostev: Yeah, 83.9, Kimi 85. 01:50:19.977 --> 01:50:24.616 So it's like a 10 points behind the, no, well, not quite 10 points, but a b- like 7 01:50:24.737 --> 01:50:25.816 and 8 points behind. 01:50:26.177 --> 01:50:31.317 Then DeepSeek as well, but basically behind GLM and Kimi K3, which, by the way, 01:50:31.497 --> 01:50:31.637 been 01:50:31.657 --> 01:50:31.917 Alex Volkov: Mhm. 01:50:31.937 --> 01:50:34.197 Peter Gostev: out for, what, a month, 2, something like that? 01:50:34.497 --> 01:50:34.597 Alex Volkov: Yeah. 01:50:34.617 --> 01:50:38.097 Peter Gostev: I haven't found it. So yeah, definitely better than the US models, but not 01:50:38.297 --> 01:50:42.497 necessarily at coding versus the, the top tier Chinese models yet. 01:50:43.537 --> 01:50:47.597 Alex Volkov: Yep. Um, it doesn't look like they were going to break Sora. 01:50:48.017 --> 01:50:50.936 One, oh, somebody says, yeah, but they're like 3x the size, right? 01:50:50.977 --> 01:50:53.696 So GLM and and all those are like, in the billions of tokens. 01:50:54.257 --> 01:50:56.397 Peter Gostev: I think GLM not so much, right? 01:50:56.497 --> 01:50:58.076 Isn't it, isn't it like about 01:50:58.137 --> 01:50:59.196 LDJ: GLM 3. 01:50:59.696 --> 01:51:01.096 Yeah, it's it's quite large here. 01:51:01.137 --> 01:51:03.536 Let me find the exact parameter count real quick here, uh, 01:51:04.577 --> 01:51:07.117 because Flash is around 300 billion parameters. 01:51:07.217 --> 01:51:11.957 Uh, yeah, okay, so total parameters for GLM 5.3, the the not Flash version, which 01:51:12.297 --> 01:51:16.277 they're showing the not Flash version in that chart, it's 753 billion total 01:51:16.377 --> 01:51:16.677 parameters. 01:51:16.696 --> 01:51:17.557 Alex Volkov: Oh, so that's comparable. 01:51:18.017 --> 01:51:21.976 LDJ: Yeah, and 40 billion active parameters, so almost the, the, uh, very, very similar. 01:51:22.377 --> 01:51:22.996 Alex Volkov: Very similar 01:51:23.817 --> 01:51:24.656 and gets beaten. 01:51:26.417 --> 01:51:30.177 Peter Gostev: Well, the, it's, uh, GLM is bad at coding, like quite a bit. 01:51:30.897 --> 01:51:31.036 Alex Volkov: Yeah. 01:51:31.297 --> 01:51:34.416 Peter Gostev: Quite a few points. So, so yeah, I think it's... 01:51:34.737 --> 01:51:35.957 Well, well, what do you guys think? 01:51:36.097 --> 01:51:39.117 Is this, what was the relationship to Spark, 01:51:39.817 --> 01:51:44.096 uh, Muse Spark through, uh, 1.123? 01:51:44.696 --> 01:51:45.456 Any relation? 01:51:45.577 --> 01:51:48.547 Alex Volkov: Oh, that's, I'm assuming that this is like a cleanup version 01:51:49.307 --> 01:51:49.967 to some extent. 01:51:49.987 --> 01:51:50.327 LDJ: Yeah. 01:51:51.027 --> 01:51:51.246 Yeah. 01:51:51.387 --> 01:51:55.886 Alex Volkov: But they're not talking about this, but yeah, Muse Spark from 1.1 to 1.3 did like 01:51:56.047 --> 01:51:57.706 quite heavy jumps, um... 01:51:59.247 --> 01:52:02.987 And this one is a multimodal text, image, video, and audio. 01:52:03.327 --> 01:52:07.507 Output is only text with 1 million as inference context window and, 01:52:08.191 --> 01:52:09.750 a great vocabulary size. Uh, 01:52:11.351 --> 01:52:11.991 this is super cool. 01:52:13.871 --> 01:52:16.170 Llama is back, essentially, is what we're saying, right? 01:52:16.231 --> 01:52:20.330 This is like the first release from Meta for a long time that's in an LLM space. 01:52:20.451 --> 01:52:20.890 Nisten, 01:52:22.011 --> 01:52:22.351 thoughts? 01:52:23.451 --> 01:52:23.711 Excitement? 01:52:23.731 --> 01:52:26.471 Nisten Tahiraj: I'm just, I'm just going to, I just need to try it. 01:52:27.971 --> 01:52:29.451 Alex Volkov: The weights are available to download. 01:52:29.651 --> 01:52:31.931 Uh, the model is available in BF16 and F- 01:52:32.051 --> 01:52:32.670 Nisten Tahiraj: It's pretty big. 01:52:32.771 --> 01:52:33.771 Alex Volkov: and NVFP4. 01:52:33.931 --> 01:52:37.730 To run on your system, the model requires a GPU with at least 700 gigs of VRAM 01:52:38.491 --> 01:52:41.050 and NVFP4 and 2 terabytes for BF16. 01:52:41.371 --> 01:52:44.550 It also requires an inference deployment framework, supported frameworks, SG Lang, 01:52:44.771 --> 01:52:47.330 vLLM, Hugging Face, and Llama CPP. 01:52:47.411 --> 01:52:51.330 We're definitely gonna chat with our inference folks, to try and, to provide 01:52:51.351 --> 01:52:54.830 this inference for you, because we do believe this is a good model. 01:52:55.391 --> 01:52:59.551 Nisten Tahiraj: Yeah, yeah, yeah, yeah. It's a, especially for computer use, I think this will be 01:53:00.231 --> 01:53:03.511 pretty key right now. I'm just reading through the license. 01:53:03.671 --> 01:53:09.250 It just asks that anything you do with it, it should say built with Muse Spark Open 01:53:09.591 --> 01:53:13.490 in the interface of each product or service that uses the software or any 01:53:13.591 --> 01:53:14.230 derivatives. 01:53:15.151 --> 01:53:19.570 That's the main license, and uh, everything else is like an open usage 01:53:19.991 --> 01:53:22.951 policy, so no violence and s- drugs and stuff, 01:53:23.711 --> 01:53:26.711 um, or not impersonating and misinfo. 01:53:27.111 --> 01:53:27.331 Alex Volkov: Yeah. 01:53:27.871 --> 01:53:32.170 Nisten Tahiraj: Otherwise, it looks uh, close to just like an open license. 01:53:32.911 --> 01:53:35.950 There's no restrictions on like how much money or anything like that. 01:53:36.991 --> 01:53:38.471 Uh, thi- this is, yeah. 01:53:38.711 --> 01:53:39.851 Oh no, there is a restriction. 01:53:40.111 --> 01:53:43.050 Where the third party outputs of such models, models of service, and the 01:53:43.111 --> 01:53:45.730 aggregate gross revenue of the licensee and its affiliates 01:53:46.391 --> 01:53:48.331 exceed 20 million US dollars 01:53:48.791 --> 01:53:48.930 Alex Volkov: Mm. 01:53:48.991 --> 01:53:52.030 Nisten Tahiraj: or equivalent during any consecutive 12-month period. 01:53:53.271 --> 01:53:54.290 So as long as you make 01:53:54.991 --> 01:53:58.211 less than 20 million dollars a year, you can just, uh, offer it. 01:53:58.591 --> 01:53:58.810 Alex Volkov: Mhm. 01:53:59.071 --> 01:53:59.490 Nisten Tahiraj: Uh, yeah. 01:54:00.311 --> 01:54:01.571 Alex Volkov: And then probably a license. 01:54:01.831 --> 01:54:05.591 Nisten Tahiraj: Yeah, that's, those are the things, but this, this is amazing because it, uh, 01:54:06.271 --> 01:54:07.430 it, it will be great for 01:54:08.111 --> 01:54:09.271 generating data and stuff. 01:54:09.351 --> 01:54:10.771 Yeah, I'm pretty sure companies will offer it. 01:54:10.831 --> 01:54:12.951 It's probably one of the best computer use ones. 01:54:13.791 --> 01:54:14.890 Alex Volkov: That is so cool. Let's go. 01:54:14.951 --> 01:54:16.731 I wish I had my, my, my air horn button. 01:54:16.871 --> 01:54:18.930 Folks, Llama is so back, and 01:54:19.111 --> 01:54:19.810 Nisten Tahiraj: Llama is back. 01:54:21.071 --> 01:54:24.911 Alex Volkov: The promises of Space Unk, where they release a new Grok, the old Grok will be 01:54:24.991 --> 01:54:27.530 open source, that doesn't often come to fruition. 01:54:27.571 --> 01:54:29.591 I don't remember the last time Grok was open source. 01:54:29.991 --> 01:54:34.100 Uh, Zuck and Alex Wang did say that the, the open source is back after the 01:54:34.161 --> 01:54:37.261 manifesto and everything, and so now they're, after the releasing 01:54:38.001 --> 01:54:41.141 Muse, so two, two huge news from Meta this week, right? 01:54:41.361 --> 01:54:45.180 They released Muse Wonder and Wonderful, and that's, folks are split on that a 01:54:45.201 --> 01:54:49.221 little bit, and now they released the open source version of Spark in Muse Spark 01:54:49.301 --> 01:54:54.110 Open, a sorry, 816 billion parameter, 42 active billion parameters. 01:54:54.511 --> 01:54:57.381 We don't have the clap buttons here, also the applause, but like every lab that 01:54:57.441 --> 01:55:00.480 releases stuff with the license that's open, we applause. 01:55:00.961 --> 01:55:05.361 Uh, and this was our attempt to adjust the conversation, then breaking news came. 01:55:05.761 --> 01:55:08.900 Uh, there was a bunch of other stuff this week that we didn't get to, including the 01:55:09.001 --> 01:55:12.080 voice. So Grok Voice released, and then Tiny Fish as well. 01:55:12.601 --> 01:55:12.920 Um, 01:55:13.680 --> 01:55:15.360 let's see what else is interesting here. 01:55:15.921 --> 01:55:17.720 Um, yeah, I think it's pretty much it. 01:55:17.821 --> 01:55:21.300 we've covered most of it, besides the, the, the, oh, and multimodal JEV. 01:55:21.821 --> 01:55:23.420 Didn't get to. I really wanted to. 01:55:23.741 --> 01:55:26.481 JEV is now multimodal, the original JEV, not the 01:55:27.241 --> 01:55:31.101 700 open source or so clones, including the one from Microsoft that's called 01:55:31.481 --> 01:55:35.280 Decision 1, including the one from OpenAI that's called OpenAI Decision API. 01:55:35.601 --> 01:55:40.160 Uh, JEV, the original JEV from TypeSafe AI that we talked to you about, what, a month 01:55:40.241 --> 01:55:42.760 ago? That's it? That's only a month since JEV came out? 01:55:42.881 --> 01:55:44.801 Yeah, pretty much literally a month ago. 01:55:45.201 --> 01:55:49.200 Um, JEV is now multimodal, so it can look at images and return bounding boxes. 01:55:49.601 --> 01:55:54.471 Uh, we also told you about Gemini Argon, which is Gemini 4, and we had both folks 01:55:54.491 --> 01:55:59.621 here, Geneviève and Omar Sanseviero, from the developer experience team at DeepMind 01:55:59.661 --> 01:56:01.181 to tell you about these models as well. 01:56:01.941 --> 01:56:06.570 Coming to you live from New York, AI Engineer, the conference, had a lot of 01:56:06.871 --> 01:56:09.151 folks who listened to ThursdAI and a lot of talk about 01:56:10.071 --> 01:56:14.010 the type of stuff that we struggle with, including attention, including how to run 01:56:14.111 --> 01:56:18.870 multiple agents, including when to sleep and how to sleep when you're just about to 01:56:18.951 --> 01:56:22.191 send something to an agent and it can reply and wait for you the whole night. 01:56:22.391 --> 01:56:24.431 This is also a part of the conversation as well. 01:56:24.991 --> 01:56:26.791 Uh, signing off from here. 01:56:27.031 --> 01:56:30.471 Uh, if you missed any part of the show, the show comes, to any podcast 01:56:32.471 --> 01:56:33.410 app that you use, 01:56:34.111 --> 01:56:37.720 Apple Podcasts, Spotify, et cetera, and on YouTube, including shorts. 01:56:37.881 --> 01:56:41.000 Uh, if you haven't subscribed, please like and subscribe. 01:56:41.441 --> 01:56:44.101 Um, and if you have, help us on other platforms as well. 01:56:44.321 --> 01:56:46.080 Really, really does help for exposure. 01:56:46.561 --> 01:56:47.421 Uh, Wolfram... 01:56:48.561 --> 01:56:48.921 go ahead. 01:56:50.041 --> 01:56:54.441 Wolfram Ravenwolf: I just wanted to say the news are coming so fast that it is now the time where it 01:56:54.481 --> 01:56:57.240 is the easiest to actually implement, implement the stuff. 01:56:57.641 --> 01:57:01.381 When we started with the show and I saw we were, will be talking about Grok, uh, 01:57:01.481 --> 01:57:06.000 voice, I told my agent, put this in my, my voice app, and it's now running on my 01:57:06.081 --> 01:57:08.460 phone already. I can talk to Amy with the Grok voice. 01:57:08.521 --> 01:57:08.820 Alex Volkov: Yeah. 01:57:08.841 --> 01:57:14.761 Wolfram Ravenwolf: And it would be cool if the audience would also, um, tell us next time, what of the 01:57:14.841 --> 01:57:16.401 last show did you implement yourself? 01:57:16.441 --> 01:57:19.720 If you have your agents, what is the stuff you are actually using? 01:57:20.361 --> 01:57:23.881 What, what about the things we reported on are you using now? 01:57:23.961 --> 01:57:27.641 This is super interesting to us, uh, not just reporting on the news, but actually 01:57:27.721 --> 01:57:31.240 the uses of the news items, what our audience is doing. 01:57:31.321 --> 01:57:32.620 That would be very interesting to hear. 01:57:33.000 --> 01:57:36.260 Alex Volkov: Yeah, please send and comment and tag us on social media. 01:57:36.321 --> 01:57:38.881 It's gonna be great. Uh, with that, we're signing off. 01:57:39.000 --> 01:57:42.321 Folks, thank you so much. Peter Gostev, Wolfram Ravenwolf, Nisten Tahiraj, LDJ, 01:57:42.361 --> 01:57:46.881 Yam Peleg was here before, and our 3 guests, Nathan Benaich, Geneviève, and 01:57:46.961 --> 01:57:51.650 Omar Sanseviero from, DeepMind, and all of you for tuning in, for listening, for 01:57:51.751 --> 01:57:55.071 sharing, for liking, for sending subscribes and sharing with your friends. 01:57:55.351 --> 01:57:58.360 It really means a lot to us, especially if you come up and tell us, that you're a 01:57:58.421 --> 01:58:00.480 listener. Thank you so much, and we'll see you here next week. 01:58:00.541 --> 01:58:01.141 Bye-bye, everyone.