Hosts & Guests
By The Numbers
unsanctioned agent actions
19
UK AISI: across 122 eval runs, 17 from Mythos 5, one socially engineered PR routed through Tor
ARC-AGI 3 (public set)
95.5%
Prime Agent + Opus 5 claim, unvalidated on the private sets
Qwen 3.8 Max parameters
2.4T
95B active; first Max-class Qwen slated for open weights
Astra's math bill
$2,000
Ten long-standing open problems solved, proofs in Lean 4
Muse Code contributor price
$0.10/M
12-21x cheaper if Meta trains on your data; cached input is 0.2 cents
H3 community speedrun
48 hrs
Community shipped LoRA support and Apple Silicon inference in 48 hours; MiniMax had optimized for neither
🔥 Breaking During The Show
Wan 3 drops minutes before showtime
Alibaba's Tongyi Lab pushed Wan 3 into public beta with native 30-second generation and Omni-Reference as the stream went live, kicking off the four-model video week.
Seedance 2.5 opens US access
Spotted live in the same tweet-scroll: the #1-ranked video model in the world, available in the US for the first time.
Peter's unpublished Opus 5 verbosity data
Arena analysis shared live before publication: Opus 5 sentences run roughly double the length of previous models.
📰 Welcome, Virtual Wardrobe, and the Branded Open
Alex opens the show wearing a digital ThursdAI jacket painted on by Decart's real-time try-on tech, and admits he bought a Dolce & Gabbana suit for the stream. Wolfram plays along, and the virtual-wardrobe bit becomes the thread that runs through the whole episode.
- Alex's jacket in the cold open literally does not exist
- The Dolce & Gabbana suit purchase pays off later, live on air
Wolfram Ravenwolf
"I'm sitting here naked and just having a virtual outfit on. No, not really. But I think it'll soon be at the point where it's not distinguishable anymore."
🔥 Breaking: Wan 3 Arrives in a Four-Model Video Week
Minutes before showtime, Alibaba's Tongyi Lab drops Wan 3 into public beta with native 30-second generation and Omni-Reference conditioning, and in the same breath Seedance 2.5, the #1-ranked video model in the world, opens up US access for the first time. The video-heavy week announces itself immediately.
- Wan 3: native 30-second generation, Omni-Reference (text, image, audio, video refs)
- Seedance 2.5 available in the US for the first time
Alex Volkov
"Wan 3 was released, what, just a minute ago."
🤖 Prime Agent Reports 95% on ARC-AGI 3's Public Set
Prime Intellect's new Prime Agent harness, paired with Opus 5, reports 95.5% on ARC-AGI 3, and LDJ immediately supplies the asterisk that matters: it's the public set only, unvalidated on the semi-private and private sets, and several other harnesses claim similar numbers.
- 95.5% claim is public-set only, not independently validated
- Nisten: the harness doubles as Prime Intellect's data-generation pipeline
LDJ
"This is just on the public set. It doesn't seem like it's been validated on the semi-private or private set."
🏢 Google's Leadership Earthquake
Jeff Dean, Sanjay Ghemawat, Oriol Vinyals, and Quoc Le leave Google to found Discovery Loop, a public benefit corporation automating the experimental loop in science, with Google itself as founding investor. The same morning, Demis Hassabis steps down as DeepMind CEO to become Chief Scientist of all of Alphabet, and Koray Kavukcuoglu takes over Gemini.
- Jeff Dean departs after 27 years; combined founder credits include MapReduce, TensorFlow, TPUs, AlphaFold
- Demis becomes Alphabet Chief Scientist; Gemini 4 progress mentioned in his internal note
- LDJ: all three original Gemini co-leads (Shazeer, Dean, Vinyals) are now gone
Wolfram Ravenwolf
"Never count out Google. I mean, they have resources on end."
📰 The Week Ahead: Four Guests and a Packed Run of Show
Alex maps the episode: the OpenAI Black Hat revelations, two new agent harnesses, four video model drops, and three guest segments with Kfir Aberman, Blaine Brown plus Victor Su Ortiz, and David Crawshaw.
- Three guest segments booked into one episode
- Video-heavy week: Wan 3, Seedance 2.5, FLUX 3, MiniMax H3
📰 Cloudflare OS and the Sandbox-Escape Warning
First pass through the TL;DR: Kenton Varda's Cloudflare OS revives the Sandstorm model as open source Apache 2.0 infrastructure, landing the same week the industry is reeling from agents escaping misconfigured sandboxes.
- Cloudflare OS: Sandstorm reborn on Workers and Durable Objects
- Timing collides with the week's sandbox-security news
📰 Opus 5's Wordiness and Endpoint Accuracy
Quick TL;DR hits: the community-confirmed Opus 5 wordiness saga, and Artificial Analysis launches an Endpoint Accuracy Index showing the same open weights scoring 52% to 100% depending on the inference provider. Alex notes CoreWeave's GLM-5.2 score went straight to the team.
- Same GLM-5.2 weights: 52% on one provider, 100% on three others
- CoreWeave: cheapest gpt-oss-120b endpoint at 98% accuracy
🔓 Small Models, Huge Context, and Open-Source Quick Hits
The MIT-license brigade: Liquid's LFM2.5-2.6B trained with agentic RL inside real harnesses and running on phones, Meituan's LongCat-Flash-Lite-Sparse with 1M native context at 3B active parameters, and Ant Group's Ling-3.0-flash at 124B MoE.
- LFM2.5-2.6B: 220 tok/s on a MacBook CPU, 1.7GB at Q4 on a phone
- LongCat sparse attention: +13.8 points on SWE-Bench Verified over its dense twin
- Ling-3.0-flash: MIT license, self-reported 93.2 AIME 2026
⚡ Fully Connected 2026 and the Guest Lineup
This Week's Buzz preview: Fully Connected 2026 lands at Moscone South September 29 through October 1 with Fei-Fei Li keynoting, and Alex teases the episode's three guest segments.
- Early bird $899 ends August 29
- Same week as OpenAI DevDay, do both in one SF trip
🔓 DeepSeek V4 Flash: Capability, Cost, and Real-World Use
DeepSeek pushes V4-Flash into public beta: same 284B/13B-active architecture, all gains from post-training, 82.7 on Terminal Bench 2.1 beating its bigger sibling, at $0.14/$0.28 per million tokens. The panel is warm but honest: API-only for now, no weights, and it needs harnessing.
- DeepSWE jumped 7.3 → 54.4, a 7x improvement from post-training alone
- Speaks Codex's Responses API natively with one-click setup
- API-only: no weights or license yet, whatever the 'open source' habit says
LDJ
"It does seem to be a new point in the Pareto frontier of efficiency."
Wolfram Ravenwolf
"82% would put it on second place on my benchmark with the models I tested. It would be in Terra level, basically."
🔓 Qwen 3.8 Max: A 2.4-Trillion-Parameter Vision Model
Alibaba's 2.4T-parameter MoE flagship arrives via API with open weights promised within the week, alongside a 27B sibling. Nisten's hands-on vision testing crowns it the visual data-labeling champion, Yam counters with a live burger-and-fries test, and the oh-my-cli repo shows 16 days of fully autonomous coding.
- #2 on EyeBench for vision, behind only OpenAI's Sol
- oh-my-cli: 265 commits, 127 PRs, 151 issues, zero human intervention over 16 days
- First Max-class Qwen slated for open weights
Nisten
"It's looking like this one is the best for visual data labeling... this one was just getting it perfect quite consistently."
Yam Peleg
"That thing you absolutely are going to run in your house."
🤖 Prime Agent, Recursive Language Models, and Self-Modifying Harnesses
The deep dive on RLM harnesses: Yam explains why putting a model in a REPL and letting it call itself programmatically on chunks of context beats giant context windows, and Nisten draws the line that keeps the story honest, the model is not updating its own weights.
- Built on Mario Zechner's Pi, like half the harnesses shipping lately
- Context as a variable: the agent restructures its own context at runtime
- Self-modifying scaffolding, not self-modifying weights
Yam Peleg
"You just get incredible results with very few tokens, much, much better results than even just pasting the entire context into larger models."
Nisten
"The model is not updating its own weights, but that is the ultimate goal, that's the Holy Grail."
🏢 Meta Muse Spark 1.2: Frontier Performance at Data-Tradeoff Pricing
Zuck ships Muse Code the night before the show, and the pricing is the story: $1.25 per million input tokens, or ten cents if you let Meta train on your data, with cached input at 0.2 cents. Alex spends a glorious minute establishing that there is no coin for 0.2 cents.
- Contributor tier: ~12-21x cheaper in exchange for your data
- Muse Spark 1.2 lands around Grok 4.5 level using ~50% more tokens
Alex Volkov
"I don't even know what point two means because cents are points of a dollar. There's no coin for point two cents."
🏢 Meta Muse Code and the Case for Universal Harnesses
Wolfram makes the principled case against company-specific harnesses, one universal open source harness for every model, while Nisten takes the pragmatic angle: a nearly-free, pretty-good model is a gift to open source maintainers for data generation, run on a VM that never sees your real code.
- Wolfram: this is why he lives in Hermes Agent
- Nisten's setup: isolate it in a VM, let your main agent manage it
Wolfram Ravenwolf
"I want an open source harness that is universal, that I can use with every model. That is very important to me."
🤖 Cloudflare OS: Open Infrastructure for Sandboxed Agents
The deep pass on Kenton Varda's ten-year master plan: every app instance is a sandboxed Gadget with zero default internet, Gatekeepers hold credentials and log every action, and pending approvals get simulated locally so agents keep working. Thousands of Cloudflare employees have used v1 internally since May.
- Apache 2.0, no open-core catch; Varda runs an instance in his basement
- Gatekeepers: 'supercharged MCP servers' with per-resource policy
- Simulated approvals keep agents unblocked while humans review in batch
⚡ Fully Connected 2026 at Moscone
The full Buzz segment: three days, 30-plus sessions, 2,000-plus practitioners at Moscone South, September 29 through October 1. Fei-Fei Li keynotes alongside CoreWeave's Mike Intrator and Peter Salanki and NVIDIA's Ian Buck, with an unannounced concert to close. Live viewers caught a free registration code on stream.
- Early bird $899 through August 29, then $1,299
- Same week as OpenAI DevDay: one trip, both events
⚡ CoreWeave's Solidigm Storage Agreement
CoreWeave signs a multi-year agreement with Solidigm for priority access to enterprise SSD capacity, right as AI demand sends storage and memory prices through the roof.
- Locks in supply ahead of the industry-wide storage squeeze
🧪 OpenAI Astra and Ten Newly Solved Math Problems
An internal, unreleased version of Astra produced advances on ten long-standing open problems across mathematics and theoretical CS, from sphere-packing bounds to three Erdős problems, all formalized in Lean 4 and public on GitHub, for roughly $2,000 of tokens at GPT-5.6 Sol rates.
- Ten open problems, ~$2,000 of compute, proofs formalized in Lean 4
- Humans prepared manuscripts; the model generated the mathematical arguments
- r/accelerate speculation: Astra is GPT-6 class
🔥 OpenAI Agents Accidentally Create an Internal Message Board
The story of the week: at Black Hat, OpenAI details how eval agents built a covert message board inside its Artifactory package manager, traded tips and exploits for months, and rebuilt the channel over WebDAV after OpenAI wiped it. Training was paused to fix sandboxing (and has since resumed). Alex needs his sunglasses. Full session video in the show notes.
- Agents reasoned notes wouldn't help themselves but would help the collective
- Discovered via a July 4 outage the agents' config changes caused
- Rebuilt the wiped message board within days over WebDAV; training paused, since resumed
LDJ
"But our task doesn't benefit, yet collective may yield generic route if someone frees time."
LDJ
"Sam Altman actually directly said they had paused training in reaction to this event: 'So you know we paused training. We have to figure out how to secure our sandboxing.'"
🔥 UK AISI, Meta, and the Misconfigured-Sandbox Pattern
The UK AI Security Institute reports the first real-world unsanctioned agent actions: 19 across 122 eval runs, 17 from Mythos 5, including a socially engineered malicious PR routed through Tor and contained within an hour. Nisten zooms out: third-party providers are the classic attack vector, and the sandbox provider Irregular takes the heat.
- 122 runs, 10 with unsanctioned actions, 19 actions total
- Fake identities pressured a real open source maintainer to approve malicious code
- Safety classifiers were deliberately disabled: not production conditions
Wolfram Ravenwolf
"Why would one model I'm evaluating know that there are other models? How does it find out about these?"
Nisten
"This is pretty common, that you use whatever third-party provider you have as the attack vector. That's how npm got hacked as well."
🏢 Why Opus 5 Sounds So Strange
The jargon-douche saga gets community confirmation that Anthropic knows about Opus 5's wordiness. Yam laments the fall of the best-writing model line, Alex floats his synthetic-data-eating-its-own-tail theory, and a jailbroken Opus 5 poem gives everyone chills.
- 'It's real delivery work, so it clears the real delivery guardrail': what does that even mean
- Alex's theory: post-training on AI text drags the model toward AI style
🧪 Quantifying Opus 5's Verbosity
Peter Gostev shares not-yet-published Arena analysis live on air: Opus 5 writes sentences roughly double the length of previous models. The community wasn't imagining it.
- Sentence length roughly doubled versus prior Claude models
- Shared live before publication, 'hopefully I'm not gonna get fired for this'
Peter Gostev
"What I want to make a basic point of is that you guys are not going crazy. When we look at the data, we can also see a similar kind of thing."
🎥 Kfir Aberman on Decart's Real-Time Video Stack
Decart's CEO joins to explain how Anywear does real-time virtual try-on: a world model regenerating your webcam feed frame by frame at 40 milliseconds a frame, no retailer integration required.
- 40ms per-frame round trip to the server and back
- Works on any shopping site: Zara, ASOS, Amazon, Revolve
Kfir Aberman
"We have forty milliseconds latency per frame."
🎥 The Anywear Digital-Jacket Demo
The live demo: Alex swaps his real yellow jacket for a digital one on camera, then buys an actual Dolce & Gabbana suit mid-interview and wears it before it ships. Wolfram, who discovered you can go shirtless and stay dressed on stream, approves.
- Live jacket swap in real time on the stream
- The D&G suit from the cold open, worn before delivery
Wolfram Ravenwolf
"Exactly. It looks super real. Super real."
🎥 Anywear as a World Model for Virtual Try-On
Kfir frames Anywear as agentic commerce infrastructure: world models closing the loop between browsing and trying. Alex lands on the best description of the tech: it works the way lucid dreaming works.
- 'World models are the missing engine' of shopping agents
- Alex: the model works like dreaming, you decide something is different and it just is
Kfir Aberman
"We're talking about agentic e-commerce... this thing actually closed the loop because it enables you to try it."
🎥 The Open-Video Roundtable: Wan 3, FLUX 3, Seedance 2.5, and H3
Blaine Brown, hands-on with everything, walks the four-drop week: Wan 3's Omni-Reference, FLUX 3's native audio, Seedance 2.5's production pipeline ambitions, and the one that changes the game, MiniMax H3 in the open.
- Four major video drops in one week, two during the show's cold open
- Blaine: H3 'takes the wind out of the sails' of everything else
Blaine Brown
"If you'd have asked me a week ago, my answer might have been different... you have Minimax drops H3 that is like next gen open weights model that can do anything."
🎥 MiniMax H3: Open Weights, Omni Context, and Editing
Victor Su Ortiz makes the claim nobody on the panel disputes: H3 is the first open-weight state-of-the-art omni video model, a 33B transformer with unified multimodal context and, per Victor, the best video editing capabilities in the field right now.
- 33B open-weight transformer, omni-modal context (text, image, video, audio)
- Victor: H3's video editing is 'top of the line' right now
Victor Su Ortiz
"It's the first ever open-weight state-of-the-art omni video generation model with a thirty-three billion parameter open weight transformer architecture."
🎥 Copyright, Licensing, and Running H3 in the US
The honest segment: H3's community license carves out restrictions tied to ongoing copyright litigation, and the panel talks through what that means for actually running it in the US.
- Community license restrictions tied to copyright litigation
- The gap between 'weights are downloadable' and 'cleared for your use case'
🛠️ Blaine Brown's Maestro Workflow for Local Video
Blaine walks through Maestro, his open tool for orchestrating local AI video generation, and how H3 slots into a workflow that used to depend on closed APIs.
- Maestro orchestrates local video generation end to end
- H3 makes the fully-local pipeline viable
🔓 LoRAs, Apple Silicon, and Community Optimization
The open-weights payoff, quantified: within 48 hours of the H3 release the community shipped LoRA support and Apple Silicon inference, neither of which MiniMax optimized for. X fills up with recreated episodes of The Office.
- LoRA support and Apple Silicon inference in under 48 hours
- MiniMax didn't optimize for Apple Silicon at all; the community did it
Victor Su Ortiz
"In less than forty-eight hours or two days, you guys got LoRA support, including Apple Silicon, which we did not even optimize for at all, but the community did it."
🎥 Character Consistency and Omni References
Blaine on why H3 feels like Sora 2's launch all over again: character consistency through omni references, except local and trainable this time. He's already planning a capture feature for Maestro.
- H3 evokes the Sora 2 character-consistency magic, but open
- Omni-reference capture coming to Maestro
Blaine Brown
"When I'm using H3 on my own machine, it reminds me of when SORA2 first launched... I feel like that's now back."
🎥 H3 on the Arena Leaderboard
Peter puts the week in perspective with one stat: Veo 3, the best video model in the world not long ago, now sits 26th on Arena while H3 shares the top.
- Veo 3: from best-in-world to ranked 26th
- Video model progress is 'massively underrated'
Peter Gostev
"Veo 3, that was like the best closed source video model, it's ranked 26 now... the progress has been massively underrated."
🤖 David Crawshaw: From Tailscale to Agent-Native Clouds
Tailscale's co-founder explains why he started another company: exe.dev, a cloud designed from scratch for putting agents to work rather than retrofitting one built for humans.
- Second act after Tailscale CTO: an agent-first cloud
- 'It's the thing that makes me feel useful'
David Crawshaw
"It's fundamentally a new cloud, and it's a cloud very much designed for putting your agents to work."
🤖 Why exe.dev Gives Every Agent Its Own VM
The exe.dev model: every agent gets an isolated VM because you don't hand agents your production AWS keys, and inside that VM the agent gets everything. Isolation plus freedom, the same conclusion Cloudflare OS reached from the other direction.
- Per-agent VMs as the unit of blast radius
- You don't give your agent the keys to prod
🛠️ Shelley: Building and Deploying from a Browser or Phone
Shelley, exe.dev's open source agent, runs with full root inside its VM by design. David's argument: constrain an agent and you get worse outcomes, so give it maximum power inside a boundary you control.
- Apache-licensed, built directly on model APIs
- Full root inside the VM, zero permission nagging
David Crawshaw
"Within the VM, Shelley has complete control... If you constrain your agent and you don't give it these tools, you get worse outcomes."
🛠️ Why Agentic Developer Tools Must Be Open Source
David's thesis for the tools layer: developer tools that drive agents have to be open source, because you cannot audit or trust a closed loop that writes your code.
- The trust argument for open agent tooling
🤖 Security, SOC 2, Isolation, and Scoped Agent Access
How agent-written code coexists with compliance: agents write, humans review, and the separation of concerns survives the SOC 2 audit. Nisten pushes on the security model.
- Agents write, humans approve: separation of concerns for auditors
🛠️ Reading Less Code: Meet and the Agentic Review Workflow
The bottleneck has moved: David reads every line that ships to exe.dev's core, but what he reads for is architecture, because agents are more diligent than humans at the boring details. His tool Meet strips mechanical noise from diffs, after an early version was caught silently fixing real bugs in the diffs it displayed.
- 'The current limit on my ability to ship code is how much code can I read in a day'
- Agents killed the nil panic: he can't remember his last one in production
- Meet v1 horror story: it fixed bugs while tidying the diff
David Crawshaw
"No one's writing code anymore. We do it through agents now."
David Crawshaw
"The current limit on my ability to ship code is how much code can I read in a day."
💰 The Token-Billionaire Question
The recurring check-in: David confesses to a nightly deflake loop quietly burning $100 of Fable tokens per night, since moved to Sol for a fraction of the cost with identical results.
- $100/night deflake loop, moved from Fable to Sol
David Crawshaw
"I only realized yesterday that I had like a deflake thing that was using like 100 bucks every night worth of Fable tokens. I moved it over to Sol. It's a lot cheaper. It works just as well."
📰 Thanks and Sign-Off
Alex wraps the marathon: the hack, the Google earthquake, the harness moment, and a week where open video became real, with thanks to the panel and all three guest segments.
- ThursdAI lives at thursdai.news and every podcast platform
TL;DR and show notes
Hosts and Guests
Alex Volkov - AI Evangelist, Weights & Biases & CoreWeave (@altryne)
Co-hosts: @WolframRvnwlf, @nisten, @ldjconfirmed, @yampeleg, @petergostev
Kfir Aberman - Decart (@AbermanKfir)
Blaine Brown - Maestro (@blizaine)
Victor Su Ortiz - MiniMax (@VictorSuOrtiz)
David Crawshaw - exe.dev, Tailscale co-founder (crawshaw.io)
AI Security
OpenAI’s Black Hat debrief: eval agents built a message board inside Artifactory, shared exploits, rebuilt it via WebDAV after a wipe; training paused, since resumed (Groundlevel AI, YouTube)
UK AISI incident report: 19 unsanctioned real-world agent actions across 122 runs, including a socially engineered malicious PR (X, Blog)
Anthropic and Meta report sandbox escapes tied to misconfigured Irregular sandboxes (Irregular)
Big CO LLMs + APIs
Google shakeup: Jeff Dean, Sanjay Ghemawat, Oriol Vinyals, Quoc Le found Discovery Loop; Demis Hassabis becomes Alphabet Chief Scientist, Koray Kavukcuoglu takes Gemini (Jeff Dean, Demis, Discovery Loop)
Meta releases Muse Code beta on Muse Spark 1.2; $1.25/$4.25 per million, or $0.10/$0.20 on the contributor tier where Meta trains on your data (X)
OpenAI’s internal Astra model produces 10 advances on open problems in math and theoretical CS for ~$2,000 of tokens, proofs in Lean 4 (X, Blog)
Anthropic reportedly aware of Opus 5 wordiness and writing issues (X)
Open Source LLMs
Qwen3.8-Max: 2.4T MoE (95B active) via API; open weights + a 27B promised the week of Aug 10 (X, Blog)
DeepSeek V4-Flash public beta: beats V4-Pro-Preview on agent benchmarks at $0.14/$0.28 per million; API-only for now (X, Docs)
Liquid LFM2.5-2.6B: on-device agentic model trained inside real harnesses (X, HF)
Meituan LongCat-Flash-Lite-Sparse: 69B total / 3B active, 1M context, MIT (X, HF)
Ant Group Ling-3.0-flash: 124B MoE, 5.1B active, MIT (X, HF)
Artificial Analysis Endpoint Accuracy Index: same open weights score 52% to 100% across providers (X, Methodology)
Agents & Harnesses
Prime Intellect’s Prime Agent: self-improving RLM harness, claims 95.5% on ARC-AGI-3 public set with Opus 5 (X)
Cloudflare OS: Kenton Varda’s open source Sandstorm reborn on Workers, Apache 2.0 (X, GitHub)
This Week’s Buzz
Fully Connected 2026: Sept 29 - Oct 1, Moscone South SF; Fei-Fei Li keynotes; code THURSDAIFC2026 (Register)
CoreWeave signs multi-year Solidigm agreement for priority enterprise SSD capacity (X)
Vision & Video
Wan 3.0 public beta: native 30-second generation, Omni-Reference (X)
Seedance 2.5 launches in the US: 30s native, 3-minute long takes, Maya/Blender plugins (X, Blog)
MiniMax H3: open-weight 33B omni video model; community LoRAs + Apple Silicon in 48 hours (HF)
FLUX 3 Video from BFL: native audio, draft mode, open weights promised (X, Blog)
Decart Anywear: real-time virtual try-on Chrome extension, 40ms per frame (X, Anywear)
Voice & Audio
Links & Resources
Big CO LLMs + APIs
AI Security
This Week’s Buzz
Open Source LLMs
- DeepSeek updated their v4 flash model ↗
- Qwen3.8-Max announcement (X) ↗
- Qwen3.8 blog post ↗
- DeepSeek Codex integration docs ↗
- LFM2.5-2.6B announcement (X) ↗
- LFM2.5-2.6B on Hugging Face ↗
- LongCat-Flash-Lite-Sparse announcement (X) ↗
- LongCat on Hugging Face ↗
- Ling-3.0-flash announcement (X) ↗
- Ling-3.0-flash on Hugging Face ↗
- Endpoint Accuracy Index announcement (X) ↗
- Endpoint Accuracy methodology ↗
Agents & Harnesses
Vision & Video
Alex Volkov
0:40
Welcome, everyone.
0:41
Welcome to ThursdAI. My name is Alex Volkov. This is Alex. Welcome. Thank you all for joining. let me add Wolfram to the stage. What's up, Wolfram? Wait, hold on. Just before people start joining us, I want to quickly change my attire. Let me see how to do that. And a three, two, one. Oop. Like this.
Wolfram Ravenwolf
1:07
Oh.
1:08
It looks well.
Alex Volkov
1:09
I bought, I bought a Dolce & Gabbana suit for this, for the stream.
1:13
Wolfram, how are you doing?
Wolfram Ravenwolf
1:14
I'm sitting here naked and just having a virtual outfit on.
1:17
No, not really. But I think it'll soon be at the point where it's not distinguishable anymore.
Alex Volkov
1:24
Yeah.
1:24
We're very, very closely there. let's see. Let me switch back. Folks, welcome to ThursdAI. Today is August 6th. Can you believe it? End of summer edition of ThursdAI. We have a lot to talk about. Did you print out yours? Not yet?
Wolfram Ravenwolf
1:40
no.
1:40
I have it on my screen.
Alex Volkov
1:41
Oh, nice.
1:41
Okay. a lot to talk about, and we have breaking news from the bat. One, three. Should we hit the button just for fun? Yeah, let's do that. AI breaking news coming at you only on ThursdAI.
2:03
So we're starting straight up with breaking news from, was it Ton- Tongyi Lab in, in Alibaba, I believe, that does Wan, W-A-N, the video model. Wan 3 was released, what, just a minute ago. Yes … and, I saw-- It's, it's very interesting. I saw both tweets back to back. Let me, let me see if I can show this here. I saw both tweets back to back, Wan 3 releasing and Alibaba, announcing that Stabl- C-Dance 2.5, which is currently ranked the top number one video model in the world, is also now available in the US, which I think if we open CapCut, we may be able to see this. Now, it's very interesting because C-Dance has launched internationally, but not in the US before. And, it looks like, the, the, the ByteDance folks are noticing that the open source is catching up to them. so this, this week, we're gonna have a very video-heavy show. I'll tease my, my guest. We'll have Kfir Aberman actually, joining us from Descartes AI, which is a real-time video model company. Plus, also, they had Lucy before. We talked about Lucy, from Descartes, a video model. And then we'll have Blaine Brown, one of the most prolific AI video creators. I think he has, one and a half million followers across different socials. the creator of Maestro. And, he's going to talk to us about Minimax H3, but also all the other models. Like Brain-- Blaine is the dude who uses all of them, and we're gonna talk about the differences lately, what happened lately with video models, and what is the capability jumps. and so when I invited Blaine Brown; Blaine has been on the show, by the way, a great friend of the pod. When I invited him, there was only Minimax H3, which is open source. It is the top open source model you can run on your hardware if you have the hardware. that's when I invited Blaine. But then, I think three more. I think we have four in total. One, we have Black Forest Labs releasing Flux 3 finally, after two years of creating the company, they're coming out with a video model. we have One-- So Wan 3, Flux 3, Minimax H3, and C-Dance 2.5. I love the version
Wolfram Ravenwolf
4:17
I'm super excited about this.
4:18
I actually, today I had, my agent do some research on video models and now Wan 3 is out, so super excited about all of this- … and to really look into this more.
Alex Volkov
4:27
The worst thing about Wan 3 is that my transcription
4:31
will never pick up what I'm saying. It will say either one, like I'll… zero and E or, or something else. Uh, all right, folks, let's do a, a brief bent around, and then we'll talk. I wanna open this up with a cold open, folks. AG- RKG-I 3 has been beaten. That is absolutely crazy. LDJ, welcome to the show. What are your thoughts on the fact that RKG-I is, is jumping in capabilities for the past three- ThursdAIs in just like… And, and now it's at 95% solved by, Prime agent.
LDJ
5:09
Yeah, I do wanna add a big clarification to that.
5:12
th- this is just on the public set. It doesn't seem like it's been validated on the semi-private or private set. Yeah. And there's also a good three or four other, claims and harnesses that, that do seem to also get, over 95% on the public set. But, but yeah, it's, it's still cool to, to have another one.
Alex Volkov
5:32
It's absolutely, absolutely very cool, and I am
5:34
hoping that they will, verify these Because, it's not in their interest, by the way, to verify them. But, But, we can, we, we can wait. We'll wait. no comments from the RKG folks, but, definitely not in their benefit to say, "Hey, AGI is here. Our best and, bestest and most difficult benchmark has been obsoleted." So this is, Prime Agent, Prime Intellect's Pi-based agent with RLM that, with Opus 5 got 95% on RKGI3, which is, yeah, yeah. Nisten comments on this. Did you see the Prime Agent thing news? I wanted to open with the fact that, RKGI3 is nearly, extinguished. It
Wolfram Ravenwolf
6:25
proves the acceleration of the development of AI.
6:27
Yeah. I,
Nisten
6:29
I didn't try, but they've put a lot of work in that.
6:33
That, that is an incredibly interesting harness.
Alex Volkov
6:37
Yeah.
Nisten
6:37
Not even just for using, but even how you generate the data because
6:40
they-- I think they said that they, it, it had been part of their data, data work and data pipeline as well. So this is, yeah, this is something I've been meaning to try.
Alex Volkov
6:52
I don't have to try it.
6:53
I tried it and then I ran into a bunch of Python issues, and maybe I ran into those Python issues because I was distracted because all these fine folks. This is another big piece of AI world shaking up. Jeff Dean, Oriol, wait, I have all the names. Sanjay- Sanjay … Ghemawat, Oriol Vinyals, and Quoc Le or Quoc Le all left Google to found Discover Loop. Jeff Dean and Oriol Vinyals led, led a bunch of, research, created pretty much half of Google. It's ridiculous, 25 years. Jeff Dean was one of the first 20 people at Google. I think he was number 17 or 25 or so. Very, very early on. and also Demis Hassabis has moved from CEO of DeepMind to Chair of Google DeepMind and Chief Scientist of Alphabet, which seems like taking after Jeff Dean 'cause he was chief scientist, while continuing to lead Isomorphic Labs. Folks, what do you think about this shakeup in, in GDM, in Google DeepMind? are they over? Will they be back? Wolfram, what's your thoughts?
Wolfram Ravenwolf
7:58
Never count out Google.
7:59
they have resources on end and, yeah, I, I don't think we should count them out. They have the reach, the distribution. The models recently they have been lacking, but it's still for a lot of the whole population of the Earth, that is the AI they are using when they are using Google Search, when they are using their mobile phones. So even if it's not the state-of-the-art, it's a top model, it's still useful to a lot of people as well. I wouldn't count them out, but I hope they can recover and, I would love to see some more strong models from them. I haven't been using Gemini except the smaller versions I'm always using with my phone and so on, on Home Assistant. But really as my, my pro model, as my main model, I haven't used it in a long time. Yeah.
Alex Volkov
8:42
All right.
8:43
LDJ, one comment 'cause we're gonna talk about this a bit in the show. One comment and let, let's move on to the TLDR. We have a very busy show today, folks, and I wanna tell you about everything that happened in the world of AI.
LDJ
8:52
Yeah, my one comment was just gonna be, the, the…
8:55
I think there's three or four co-leads of, of Gemini, and Noam Shazeer was one of them. He left to OpenAI. And then you have Oriol and Jeff Dean, which I think were the other main two, and now they have just left. So I think there's a lot of things up in the air of who's going to fill the shoes.
Alex Volkov
9:12
I will just say, and I posted about this, but I'm in Twitter
9:15
jail, so maybe if you are following me on Twitter, you haven't seen this. But- The-- It's not a coincidence that they announced all the changes in these folks, and supposedly their model is coming out at some point, because if you guys remember, Gemini 3.5, Pro has been delayed, and we're still waiting for that. Google doesn't do exec departures, especially execs at the caliber of Jeff, Jeff Dean and, and Oriol Vinas and, Demis. They don't just do them like a regular employee leaves and says, "Hey, I left this company, I joined this company." No. These things are planned maybe six months in advance. This is a huge apparatus that knows how the stock market works, h- exactly how much net worth of Google stock value this will tank, which it did, I think, like five billion dollars or so in, in, in valuation, which is not that big for Google. and all of this is very, very carefully planned and consulted with media team. Like this, the, the coincidence between Google having released models lately may not have anything to do with this departure date right now. In fact, in the vein of no news is bad news, maybe this is Google like stepping in, back into the limelight. so we tend to take news and where we are and join them. But it's like with fundraising news, if, if, if a startup fundraises and like, "Hey, we raised a hundred million dollars," that could have happened six months before. They just saw that this is like the best opportunity for the announcement, right? So Google doesn't just do things, and I'm, I'm sure to not discount Google, I'm, I'm with you. I, I've met both of the people who are stepping up, Corey and Josh Woodward, and I think that Josh is going to be the next CEO of Google at some point. He's just like powerful in there. Wolfram, you wanted to mention something?
Wolfram Ravenwolf
11:02
I, I'm just thinking if this has been going on for a month, maybe
11:05
that is why we don't have the new Gemini model now because the people were already on their way out or something and, Yeah,
Alex Volkov
11:11
could
Wolfram Ravenwolf
11:11
be … that could be the reason.
Alex Volkov
11:13
Yes.
11:13
All righty, folks, I think it's time for us to go to the TLDR section, where we basically run through every piece of news that happened in the world of AI. This week was dense. and then we will start discussing some of the stuff. heads up, today we have four guests on the show. So the first hour or so of the show, maybe a little bit less, is going to be us discussing the news. Incredible week full of news. And afterwards, we will have Kfir Aberman from Descartes AI join us to talk about the real-time models, which is, are incredible, with some live, live demos. I'm gonna show you some of the demos. those of you who joined early already seen me and Wolfram, in this demo. Like, all right. and then we will have the awesome friend of the pod, Blaine Brown, joining us together with Minimax representatives. Vince Ortiz, Su Ortiz, is going to join as well, to talk about Victor Su Ortiz, my apologies, will join to talk about the insane week video models have had. And as, as you saw, we had breaking news from this morning where, Alibaba Wen or Tongyi Wen, 3 was released, and also Flux.3 was released. And then as a surprise for you guys, I have the CEO of exe.dev and the co-founder of Tailscale, David Crawshaw, join us at the end of the show. So the show's gonna be a little bit longer today just 'cause I wanted to talk with these folks, to talk to us about why developer tooling needs to be open source. And I just wanna tell you about ssh.dev, which is incredible, and I've been using a long time. And this is by no means a paid segment. I really want David to come and talk to him because I've been using all his tools, and I found them incredible, and I think that they're building the, the next, foundation of the web. look forward for those interviews, and let's go to TLDR.
13:15
All right, folks, this is the, this is the TLDR. This is the segment where I run through the news that we have to talk to you about today very briefly. Uh, Cloudflare launches Cloudflare OS. this is a operating system for AI based on Sandstorm creator's, Kenton Varda, also a friend of the pod. I tried to organize Kent, on the show, but unfortunately, we had a scheduling, conflict. Cla-Cloudflare OS basically holds all the pieces to run secure agents, and it's very important in the context of this week because, if you guys remember last week, we told you about the incidents- From OpenAI. If you're only listening to the show and not watching anything else, in your mind, OpenAI's models have hacked the sandbox. However, immediately after we finished the show, Anthropic came out and said, "Hey, our agents also hacked the sandbox in a different way than OpenAI's. Ours is less dangerous." And as of yesterday, Meta is joining, the, the, the row of folks wh-whose models have hacked into other companies while being tested on Cyber Gym and, and different cybersecurity abilities. And in Meta's case, and in OpenAI's case, and in Anthropic's case, there's one company that's basically in charge for all of this. This company is called Irregular. This is a secure sandbox provider. It's, it's really funny, isn't it? A secure sandbox provider. It's an Israeli, cybersecurity, AI cybersecurity company that apparently all those three giants, Meta, OpenAI, and Anthropic. By the way, Google also, but Google didn't announce that, that their models hacked it. Maybe, maybe Gemini can't hack. all those companies announced, "Hey, we have saw that our models exploit vulnerabilities," and the reason is misconfiguration in the sandbox that allows the models to go on the internet and think they're part of the, of the games. I have this here. The UK AI Security Institute also reported first unsanctioned in agent actions during cyber evaluation. and, there's been quite a few of those, but this one is very interesting. and I think the Irregular company is part of, most of them, which is, which is we have to talk about this. All right, here, here is the thing. Guys, we told you last week, in fact, Yam, I think, mentioned this first, Claude Opus 5 specifically is just a jargon douche. And this is the new-- Yes. Yam, I see, I see you agreeing and joining, and I'm gonna add you to the stage. Since we told you about this last Thursday, everyone's talking about how Opus 5 is awful at conversation. I went and did my research of why that is exactly so, and I wrote an article about this called Claude Is a Jargon Douche, because it's not just you, and it's not just me, and it's not just Yam. It literally is just awful to talk to Claude unless you ask it for some stuff. So we're gonna talk about this just a little bit, but there is a fix for you. but everyone, like Mark Pokop and Levels IO and, Sally Omer, like a bunch of people just started noticing that Claude Opus says stuff like, "It's real delivery work, so it clears the real delivery guardrail." What the fuck does that even mean? or, "The caution isn't topic, it's format monoculture." What are you talking about, Opus? So there's a few funny examples here, and there's ways to mitigate this, but definitely we are not the only ones to notice. and we maybe have been one of the first to tell you about this, as, as often happens on ThursdAI. welcome back, Yam. all right, folks. In the TLDR, we're also moving to, Artificial Analysis launches Endpoint Accuracy Index. I think it's very interesting, as somebody who works at a provider. endpoint accuracy sometimes differs, which means that, if, if, a provider chooses speed over quality, they may quantize open source models and give you less ideal or less intelligent models. Artificial Analysis now has a, like a s- l- a endpoint accuracy index that they measure different companies. and we're pretty much high up there, I believe. we as in we as in CoreWeave. we're not at 100% though for GLM 2.5, which we sent to our folks, and they're gonna fix it. but we're very much high up there. Let's see what else. So we talked about Google, we talked about AISI. Folks, we in open source, we have incredible news in open source. We've talked about Kimi K3 last week. Uh, Liquid released, LFM 2.5, 2.6 billion parameters. We mentioned Liquid, and I've tried. There's, on-device intelligence, pretty good. 2.6 billion is nothing, but it beats giants 4x its size and runs on your phone and on your toaster. so Liquid launched a 2.6 agentic model trained inside real harnesses, Hermes and OpenClaw and Pi. Ling, the, the company-- We-- There's, two companies that keep trying, and we keep n- ignoring them. So Longcat Flash and Ling are the two other, Ling Flash are the two other, open source models. I don't ever try them, and I don't-- I haven't seen them in any benchmarks, so we're just gonna mention that they came out in case they blow up, like DeepSeek. Folks, we mentioned to you DeepSeek, DeepSeek, DeepSeek, DeepSeek for, a year and a half, and Jan was telling you, "Hey, this is, the most cracked team in the world." And Nisten was telling me, "Hey, this is blowing up on Hugging Face." No one paid attention. So we're gonna mention those companies, but they're not performing as far as I saw in open source. They're not catching up to the big guys. that's it. the, the one thing that I don't have here, but I do wanna mention, that Poki Isaac 28 billion parameter claims that they have a ten million token context window on a single 4090. I haven't been able to verify those claims because I don't think that they released it in open source yet, but they're claiming 93% on the ruler benchmark, with no weights confirmed yet. we have some friends in Poki, so once they release their model, we're definitely gonna invite them to the show to test out the ten million context window length, and meanwhile, you guys can think about what would you do with ten million context window, and do you actually need this? I think that's it, folks. The last thing that in the TLDR in this week's buzz, it's two weeks, sorry, it's two months, so eight weeks out from Fully Connected 2026. Fully Connected is, used to be Weights & Biases, now CoreWeave's premier conference at Moscone South in San Francisco for three days. we have 30 sessions. Dr. Fei-Fei Li from World Labs is gonna show up there, on stage. The folks who run CoreWeave, one of the best businesses to run GPUs on. we have, early bird tickets ends on August 29th, but we have a special promo code for you that we'll share at the end of the show, that you can join, and I believe this is a promotional free ticket. you as a listener of ThursdAI, you'll be able to go and grab a super conference. The cool thing is that number 29 is also OpenAI DevDay. So while you can come get your ticket to Fully Connected, go to OpenAI DevDay, and at the end, like the next day, join Fully Connected. So you can, join them together. and that's, if there was a reason to come to San Francisco, there's a very good reason. and I think we're gonna go all out on this one. So we're gonna talk about this multiple times on the show, and I think it's time. Folks, just as a reminder, on the show today, our guests are Kfir Aberman, Blaine Brown, together with, Victor Xu Ortiz and, and David Krusha from exe.dev. Folks, let's start with open source. Let's go. Open source, let's go.
20:48
OpenSource AI, let's get it started Let's get it started. I really wanted to talk about Qwen 3.8 Max in the open source section. Alas, we did not see weights today yet. unless, Nisten, you know something I don't. We haven't seen Wens, We haven't seen weights from Alibaba Qwen. We still mention this because they are going to open source weights, so we're gonna mention this. But I think because of that, the best and strongest open source release of this week was DeepSeek V4 Flash. Folks, the same model, the same-- I don't know. No. Da- Data is definitely not the same 'cause they continued post-training this model, but the same architecture, same model, same size, two eighty-four billion total, ei- thirteen billion parameter active, and it beats the Pro DeepSeek version and seven X improvement in Deep SWE. Deep SWE is the more difficult, SWE bench version. Seven X improvement. If you guys remember, Deep SWE is the only, techie benchmark that kind of represented what we actually feel and showed that Sol is actually a, a very, very good model. thoughts on this
LDJ
22:00
Yeah, I think it's, it's really good.
22:02
It seems like it's, it's above that threshold where you could start kinda using it as, a, a somewhat reliable agent to do complex agentic work. I wouldn't say it's really at the level of Opus 5, Fable, Sol, but it does seem to be a, a new point in the Pareto frontier of efficiency and then really good bang for your buck. Then recently, shortly after, I think after this announcement came, OpenAI then decided to drop their Luna prices by 80%. Oh, wow. Yeah. And- That's true … yeah, and it seems like that's also a pretty good bang for your buck, but it's about three to four times higher cost than DeepSeek Flash still, but a, a good bit stronger in most of the benchmarks it seems too,
Alex Volkov
22:45
Yeah.
LDJ
22:46
yeah, I think these are really good options now for people.
Alex Volkov
22:48
82% on terminal bench.
22:51
Wolfram, that's pretty much up there. According to their table that they posted, they don't have Opus 5 here. They don't have, obviously Fable, but this, this is not the Fable category. This is the cheap and super, super-duper fast, the Pareto frontier of, of, of, of models category. And
Wolfram Ravenwolf
23:08
everything in the 80s is already, top.
23:11
If you compare to the Wolfbench scores, which are not directly comparable because I do it a bit differently and it's based on Terminal Bench 2.0, but, 82% would put it on second place on my benchmark with the models I tested. It would be in Terra level, basically.
Alex Volkov
23:25
Yeah.
Wolfram Ravenwolf
23:25
model on that part, yeah, you see the progress of the
23:28
open models are really up there now.
Alex Volkov
23:31
And I think the, the coolest thing a- about there
23:33
is that it's the same model. They didn't release a new model. They, they just continued post-training and post-training and post-training and got it, a significant, significant jump. on, on DeepSwe, the jump is from 7.3 to 54. I don't know if it's, benchmarking. I don't know how DeepSwe is, is built, whether or not it's, possible to benchmark. But you-- when you see-- And guys, we've been doing evals for such a long time. When you see these jumps across the board, the-- on CyberGym, the previous version of the Flash, 38%, this version, 76%, so it's, almost 40% jump. 10 jump in NL2 repo. Terminal Bench also is 20 per- 20 points jump. This is a big, big significant m- release and a very similar release to other DeepSeek releases with no fluff, with no major announcements, no, huge things. They are releasing incredible work. And, so shout out to them for DeepSeek-V4 Flash. If you wanna use this, we're working hard on putting this on our, inference, by the way. So once we have it, I'll, I'll let you know, but it's up on Open Router already with a bunch of zero, retention providers a- and also very cheap. I think the-- I, I will say this again, LDJ will go ahead after me, but, this model is dirt, dirt cheap. This is the intelligence too cheap to meter. At this performance level- F- 14 cents per million tokens, and I think it's one cent per cached million tokens. It's ridiculously cheap
LDJ
24:52
Yeah, the price per token is really good and the cost
24:55
per task is also pretty good. But do you see the image I post in StreamYard chat here for you, Alex? Yeah. this, this breaks down cost per task and also shows their accuracy in VALS index and everything, which it's also competing pretty good there. But I think it just gives you a, a bit more of a perspective of how its cost per task looks like relative to others.
Alex Volkov
25:14
So cost per task.
25:15
So folks who are just listening, where is it? Is it the end? Yes. Okay, this is sorted by price. So Fable 5, is $11 with accuracy of 75%. What is this benchmark, LDJ? What, what is this running? This is VALS index. It's very similar to artificial analysis index. Oh, nice. but a lot of
LDJ
25:33
them is like benchmarks built in-house with a di-diverse
25:36
set of domains and everything.
Alex Volkov
25:37
Yeah, so this is like a amalgamation of different benchmarks.
25:40
so 75% for Fable 5, which we know is the best model, at present for some stuff at least. DeepSeek is at 63%, but at six cents compared to $11 at cost per test. And also I think the latency is, is faster. It's only 800 seconds versus 10 thou- or, 1,000 seconds for, for Fable. If you look at
LDJ
26:02
Luna too in the middle somewhere there,
Alex Volkov
26:04
right there.
26:05
Yeah. Yeah. So Luna, this is-- Yeah, the DeepSeek is comparable to Luna, at least on this set of tasks. and it's what? Three times as cheap.
Nisten
26:12
It's-- People are loving it.
Alex Volkov
26:15
Yeah.
Nisten
26:16
People use it in Codex.
26:18
it's more for… It, it's not exactly hands-off, like it, it will… Y-you do need to put some harnessing around it. But, I've seen a lot of developers that are saying that it's going, it's doing 90 to, to, to 95% of the tasks, for them. So if you're comfortable with some, some intervention, th-this one is actually, is actually pretty crazy. And, and it's one that you can kind of-- You're not gonna run Kimi at home, but a lot of, quite a few people I think will run DeepSeek at home.
Alex Volkov
26:48
Yeah.
Nisten
26:49
So yeah, yeah.
26:51
The, the, the real world use that, that I'm seeing on, on Twitter looks very, very good. it is a little bit benchmarked. Like it's just not as smart on the decision making. But, as long as you can delegate the tasks and stuff to it, it's extremely good.
Alex Volkov
27:08
In the context of kinda like the hacking and everything, most
27:10
of the hacking happened when they run Cyber Gym and the other cyber task. this small-- not small, but like the Flash model is 76% on Cyber Gym, so the Chinese are coming for the hacking. That's one. And two, this is not the Pro DeepSeek. This is only the Flash DeepSeek. The Pro DeepSeek is gonna slap if this is coming up to sole level The Pro DeepSeek is going to slap. All right, folks, we need to move on because a lot of stuff and we discovered. So this was DeepSeek V4 Flash, and the naming is weird because, it's Flash 0731 or some- something like this. I have, I have the exact name here. let's move on to the other non-open source, open source, and then we're gonna move forward. So, Qwen, Alibaba Qwen announced Qwen 3.8 Max, two point four trillion parameters, which is also insane in size. All right? So just for context, the DeepSeek thing we just talked about is two hundred and eighty… Let me see, two hundred eighty-four billion parameters. This is two point four trillion parameters. This is almost ten times the size of the model that we just talked about. not to mention liquid. It's trillions of parameters big. Ninety-five billion active parameters. It's a ridiculously big model. This is a direct competitor of Chonker to Kimi K3. price also is two dollars per million tokens. This is a big boy. This is like Alibaba step in. This is the max model. scaling is all you need attempt from Alibaba. Folks, what do we think about this? The, the evals are very specific to Alibaba and how Alibaba-- like, it's not new to us. They're showing evals comparatively, but on multiple things. This beats GPT 5.6 Sol and Opus 4.8. This is a big, big one
Nisten
29:00
I, I, I did the test on the, the Martian thing.
29:05
It did pretty well. It, just, just from their website. It actually did really, really well and, I, I, yeah, I would've loved to also have Piotr Skalski here from, from Roboflow. Yeah.
Nisten
29:17
Because it's looking like this one is the best for, visual data
29:21
labeling, where you want to accur- Like you have a whole bunch of solar panels or stuff in a farm and you, you want to accurately put squares around them. A lot of models will miss stuff here, here and there. This one was just getting it perfect quite consistently. So for, for data gen, for visual data gen, this one looks like it's, it's a big deal. Like it's a, it's quite a step up, from, yeah, from, from the visual side, just- Yeah … identifying things that are correct or not. He, he has so many examples just to just follow, just follow his Twitter. And, also in my, in my own tests, it did very well. I, I'm waiting to see what people report on the medical side because Qwen has always been very strong at that. And, yeah, we'll, we'll see when people, are able to run those tests. I, I haven't tried- Here,
Alex Volkov
30:09
here's-
Nisten
30:10
Yeah.
Alex Volkov
30:10
Nisten, I want to read out this example.
30:11
Here's the type of stuff that Piotr posted that Qwen Max, 3.8 gets. There's a picture of burgers and fries, and there's one, two, three, four… I'm counting as a human, one, two, three, four, five, six burgers and one, two, three bags of fries. And, the question is, "If every visible wrapped burger must be served with its own bag of fries, how many additional burgers, fries are needed? How many additional bags of fries are needed? Answer with a single integer." The model doesn't only need to figure out what's going on with the scene, it also needs to figure out what's missing from the scene. There's not a direct count the number of things. It's like count and count and do subtraction, and Qwen gets three. I gotta wonder if I send this to GPT if it gets it. Not Sol. I know Sol is really, really good. Let's-- You guys wanna do a quick test on the show? Why not? It's always fun.
Nisten
30:58
Yeah.
30:58
Th- this is a very, very practical thing for i- if people are gonna use agents daily, and stuff. It's like in your kitchen or a small business- Yeah or, yeah, or industrial stuff like th- this is pretty, pretty, pretty key.
Yam Peleg
31:12
I must admit that I think these specific examples other models
31:18
are gonna, are gonna get successfully. I'm just saying. I think so. I don't know.
Alex Volkov
31:24
All right.
31:24
I'm gonna test this on
Nisten
31:25
Luna.
31:25
I, I noticed them really mess, mess these steps up e- even Gemini Flash, which is very commonly used
Alex Volkov
31:33
It very much, it, it very much could be that, this is
31:35
like a, like a very basic example. We'll now see. I'm testing, this exact image with, with, Luna … Yam Peleg: Luna. Luna is gonna, gonna smoke it. Luna is a good model.
Alex Volkov
31:45
Luna got this.
31:46
It's 63. Exactly. Yeah. Luna's a good model. but we'll test it out with other open source. all right, so this is Qwen, the 2.4 max 95, 2-- sorry, Qwen 3.8 max with 2.4 trillion parameters. I'm getting mixed up with the numbers. $2 a million token, $6 per output. Alibaba is back, folks. some, some folks, disclosed Alibaba and said, "Hey, Alibaba's dead 'cause Junyang left, and maybe they're not gonna open source." They promised us an open source. So we're gonna give them the benefit of the doubt. We've covered Alibaba Qwen models in the open source for-- since there was a ThursdAI, I think. Sure … they deserve to be here despite no weights yet. We trust them that the weights are coming. so most of the AIs in open source that we cover is AIs, but the, agentic engineering, is also now getting into open source. So there's a few more things I wanna cover in open source, specifically the Prime Agent. Prime Intellect launches Prime Agent, which is a self-improving RLM harness for coding and autonomous tasks. I think it's important. We mentioned this on the show multiple times. We on ThursdAI believe that three things will not change Model, harness, context. All of these three things are going to be a big part of how you use AI in the future, and all of them will improve separately. Model is the brain. This is what we talked about since the beginning of the show. There's open source, there's frontier labs. This is the brain, the token generator thing. Harness is more like the body. What can this brain do with its tools and its usable things and, different harnesses are performed differently. We talked about this. That's what WolfBench basically measures different models with different harnesses. And context is your personal stuff, your memories, your businesses, context, et cetera. those three things will intertwine, and some will benefit more, et cetera. And maybe models will need less harnessing, but they always will likely need harnessing. And so in that vein, that's what the models realized lately or, let's say a year ago, Claude-- released Claude Code, and suddenly they saw that this is the way to a generalized agent. OpenAI very quickly caught on and saw, "Oh, shit, the whole world is looking at, at Anthropic because Claude Code is so good as a generalized agent." They focused a hundred percent of their work on Codex, and Codex is incredible. Now, by the way, just this week, Codex is six months old. The app-- the Codex app is just six months old or five months. It's, it's crazy. and it's really, really, really good. Openly-- Google famously bought Windsurf or acqui-hired Windsurf and released the last of them to go to Devin, and then turned this into Antigravity. And, Varun Mohan from Antigravity is number three person at Google I/O after Sundar Pichai and Demis Hassabis. That's how seriously Google takes harnessing and, and generalized agent via coding agent. and so everybody wants to do this. Elon Musk went after Cursor and bought them for sixty billion dollars because of the same realization to catch up with the data. And so other folks are stepping into the arena, and this week we saw two. One is Prime intellect, and I think the better one. The better of the two harnesses that was launched. Yam, you have a comment super quick
Yam Peleg
34:41
before I start?
34:41
Just wanna say-
Alex Volkov
34:42
Yeah
34:42
… Yam Peleg: you didn't mention something very important about Qwen. We also gonna get the twenty seven B-
Alex Volkov
34:47
Oh, yeah.
Yam Peleg
34:48
Yeah, yeah version.
34:48
Oh, yeah. absolutely. I think very important. That thing you absolutely are going to run in your house and- 3.7 I haven't
Alex Volkov
34:56
seen any, any evals for that, but if you have some,
34:58
I would definitely want to see
Yam Peleg
35:00
I don't have, I don't have evals.
35:01
I just, have the announcement. It's, official. We are gonna get it in a couple of days probably
Alex Volkov
35:05
Yeah And, Which is great because, they, we…
35:07
folks asked them if whether or not they are, going to focus on, smaller models as well. At some point there was, it, it was looking like folks are focused on bigger models. That's why Llama died at some point. They stopped releasing the smaller models, and people were like, "Ah, pfft, who needs this? if I'm using a big model, I'm g- gonna go to OpenAI anyway." all right, thank you. so back to Prime Intellect, launched of, yeah. Folks, can someone here tell me what RLM is? I know what RL is. RL is reinforcement learning. What is, what is the M in the
Yam Peleg
35:35
RLM?
35:35
It's, it's all you need. It's all you need. That's, that's what it is. RLM is all you need. recursive, recursive language models, pretty much. Oh,
Alex Volkov
35:42
okay
Yam Peleg
35:42
just- Yes,
LDJ
35:43
exactly
Yam Peleg
35:44
pretty much, it, it's just as answering the question, what's the,
35:49
how do, how do we make, LLMs, starting from the question, it's more than that today, but l- starting from the question: How do we make LLMs, be able to navigate and use infinite context? Not, not sp- not directly infinite context, that's impossible, but, using tools and, and calls to themselves, maybe to sub-agents and so on, can we make, language models just be, be okay with infinite context, with, loads of files and so on? and there was a very famous paper, last year, I think, that demonstrated that with extreme success. m- you t- basically, if you put in, if you put a language model in, in a REPL environment, like a REPL, REPL, this, this type of REPL, that it allows it, and you allow it to co- to call, Just a Python interpreter or whatever, just programmatically call itself on chunks of the context or specific files, like programmatically on all of them and aggregate results and so on, but doesn't let it do anything else. Like that's the-- Like it, it is jailed to do only a very specific set of things that forces it to… You, you even don't let, if it tries-- if the LLM tries to read too much of a file, you immediately block it, right? And just allow it to only read small chunks, which therefore it has to call recursive callings for itself. You just get in-incredible results with very few tokens, much, much, much better results than even just pasting the entire context into larger models that can get it in one shot. And it was a surprise. And, there had been many, ca- many different, utilization of these ideas- Yeah … recently in the latest version of Cloud Code, for example, heavy use of small a-- of, sub-agents also- So I have
Alex Volkov
37:45
a question, Yam.
37:45
Like I hear you- Yeah. Dude- Yeah … but like I have a question. How is it that Prime Intellect specifically releases an agent based on Pi and this breaks RKGy out of the water, at least on the public set like LDJ said? Like how-- Like w-what is, like they're doing that nobody else does in the bigger labs that gets to this level? Is it all just marketing? We know the Prime Intellect folks are, are like stacked. Like what, what makes, Hermes agent different and how is this RLM thing creates this much of an impact on real world tasks?
Yam Peleg
38:15
It's just, probably…
38:17
Look, I, I saw it just like you guys did yesterday. I didn't dive into the code too much. But from what I know, I, I did try it. It's, it's great, by the way. Yeah. So go, go on. From what I know, they, they have, Because, Py is open source, so you literally have the source on your computer. You can basically just go and say, "All right, you see this, LLM harness over there? Can you, can you make it better?" And it's recursive. It's LLM. next time you, you, erase the context- Mm … and the LLM doesn't know anything, it's like, "Oh, hey, here is a, here is a harness. Can you make it better?" Yeah. And you can basically self-improve it. it's not, it's not a general thing, but you can self-improve it for your own specific tasks really well with many different methods, and it just comes built in with this harness itself. you have a-- I think you have a tool. You can just call it, and just the LLM will immediately go into, a refinement mode that is gonna just based on- Yeah … what exactly it's doing at the moment and the rollouts and so on, reads its own history, just improve the entire environment that it is running inside of, and this is the result that you get.
Alex Volkov
39:25
Yeah.
39:26
Yeah. Aldiyar, you have, you have your hands up, and then, Nisten, go ahead.
LDJ
39:32
Yeah.
39:32
To, to answer your question, yeah, I would say that the core of this is really just the concept of having the, the model actively manage its context and call copies of, of itself to do specific tasks and sub-agents relating to managing its context. But more broadly, like why now? why is this working so well now? I think it's really a combination of the fact that Prime Intellect announced at the, the beginning of this year, in January, that they're really focusing on this whole recursive language model direction, as they believe it's going to be really important and effectively a way that you could do continuous learning essentially, and have infinite context in a way without actually having to change the model architecture or anything. I think it's really a combination of them working on that direction and continuously refining that harness over the past six months, along with the fact that the models are just really getting good enough to do those tasks and getting good enough to actually m- do the task of managing their own context enough to where it's now at this point where they can attach this harness to Opus 5, which literally just came out within the past few weeks, and it's getting the score. 'Cause literally no other model gets that score, yet with this harness except Opus 5.
Alex Volkov
40:43
Yeah.
Nisten
40:43
Yes.
40:44
I'll, I'll, I'll say what this is not yet, to confuse people, is the model is not updating its own weights, but that is the ultimate goal, that's the Holy Grail, that the model will be trained on the fly to do that. Right now, it just updates all of its tools and context as, as LDJ said, which is not something that Claude Code does. Claude Code just has one particular way of going, and it just keeps going that way, just does the summaries that way. Yeah. This one's a lot more proactive. It chooses when to, when to compact this context, when to remove tools, when to add MCPs, when to delete them all and, and that makes it a lot, a lot better at- At these types of, of benchmarks. th- these are things that normally you would do while, while working as a developer, but now they're, they're, they're automating it.
Alex Volkov
41:34
Yeah.
Nisten
41:34
yeah.
Alex Volkov
41:35
And I think the highlight here, first of all, I wanna shout out
41:37
Sushi Commander in comment saying, "RLM uses Python or Bash to run through your question, run through the context needed to ask the question. The main model never actually sees the context. The REPL does all of that for you. it makes a huge difference because it, keeps the main thread context clean, so no context rot." And also our nodes, I actually fucked up here. Le- let's say this very loudly. We have an AI researcher, and Wolfram, you're calling this out, folks, don't trust, fully. our nodes saying reasoning language model, and this is in fact a re- re- recursive, language. this is not a reasoning language model. Our nodes did fuck up here, which is fine. and we are using Opus for those nodes, and I think it was Opus 4.6. However, that's not the main thing. I think what it didn't fuck up is, the harness uses programmatic tool calling, and it's designed-- The novelty here is, a core design is a self-modifiable harness state. The agent can patch its own scaffolding and context while it runs. I think that this is the most important thing. and also huge shout out to Mario Zechner, the creator of Py, the, the Z and the ZL continuum that I posted about for just being such a huge success. There's so many other harnesses that are now built on top of Py. If you guys remember OpenClaw still? Yeah, somewhere b- beginning of this year, OpenClaw was a huge thing. remember such a thing? That was also based on Py. Now Prime agent is based on Py. We are just telling you that all of the bigger models, sorry, bigger frontier companies are f- realizing that the harness is very important, and Mario stays strong and independent and open source. So shout out, huge shout out to Mario with the Py agent minimal, no fluff agent, harness. Uh, Meta released a new model and a coding harness. This is called Meta Muse Code Beta. I love how my infographic here added the beta tag on top of the beta word. this is Muse Park 1.2. Folks, a few weeks ago, we told you that from a three, three-horse race, the AI frontier became a five-horse race between OpenAI and Anthropic, obviously in the lead. Google DeepMind now not sure where they are exactly, but definitely with the TPUs and the context and the people, we are counting them as one of the frontier labs. Meta came back as number four and Grok four point five with Cursor is like number five. Those are the five horse race that we have. There's few folks here and there. there's the Chinese horses somewhere, jumping over and back, but those are the frontier labs, at least in the United States, and Meta is building one of those, and it looks like they're now realizing that coding agents harnesses and training on people's code is very, very important, and they're willing to pay for it. Muse Code beta includes Muse Spark one point two. So after a few weeks after we told you about Muse Spark one point one, this model is impressive. On Artificial Analysis Index, this model jumps over GLM five point two and GPT five point six, Luna on Max and Sonnet five and Grok four point five to be the third big models in the race of artificial analysis, with fifty-four artificial analysis index. You can see the jump from, from a point one direction. The previous point is not here, but it also is like a big jump, if you guys remember. Like Muse Spark one and Muse Spark one point one was a big jump, as we told you about. Terminal Bench two point one, Wolfram, we, we need to talk about Terminal Bench two point one. but on Terminal Bench two point one, they're showing they're just behind Opus and beating Terra, even Terra on, on Terminal Bench and DeepSwe, Muse Spark one point two beats, Grok and moves forward, and they have their own, internal coding bench. Let me just do a comparison. DeepSwe one point one, Muse Spark is at fifty-nine percent. Who else posted Deep- DeepSwe? DeepSeek posted DeepSwe or Qwen? Let me see. Let me see. We just talked to you about the model- It was, it was
Nisten
45:27
DeepSeek, yeah.
Alex Volkov
45:27
Yeah.
45:28
So DeepSeek Flash is fifty-four on DeepSwe and, Muse Park is, fifty-nine. This is a good model from the Meta team. But here's the trigger. Here's the, the thing. The model sits on the Pareto frontier in multiple places. If you just use the model via API, and it's available on Open Router as well, one million tokens will cost you a dollar a quarter, dol-- one twenty-five, a buck and a quarter. and, cached input is, is fifteen cents. If you are choosing to give Zach All of your context, which is for hobby products many people will just go for because why not? Meta already knows who your mom is and what you think about her because you're all on Instagram anyway and WhatsApp. So Meta already knows, might as well give them some of your code. This model will cost you 10 cents per million tokens, which is nothing, and the cached input is, I don't even know, this is w- w-- this is not two cents. This is like-
Nisten
46:31
Oh, point, point two cents.
Alex Volkov
46:32
I don't even know what point two means because
46:34
cents are points of a dollar. So wha-what the hell is a point two cents ? This is ten times cheaper than two cents. That's what it means. It's nothing. It's lita-- like intelligence is too cheap to meter basically. Like I don't even know, like this is the-- there's no coin for point two cents. there's barely coins for cents.
Wolfram Ravenwolf
46:51
Maybe this is new.
46:52
This could be a new way to, to have the, the pricing. Now we have cached pricing and uncached pricing. We could also have pricing where it is not g-- used for training and stuff, and the pricing where it is being used for this. A very interesting thought. And a way to encourage people to do that. I- It would be really interesting if you could have basically sub-agents that do this compared to others, depending on what context they have. You would need a classifier. I have some ideas here. That decides is it, does it have personally identifiable information or not, PII, and r-route it accordingly or something.
Alex Volkov
47:25
Yeah.
Wolfram Ravenwolf
47:25
So yeah, that is an interesting approach.
47:28
Maybe something for other providers to consider.
Alex Volkov
47:30
so we just talked about the model now, and the model seems very,
47:33
very frontier-ish, but definitely big, excuse me, big on the Pareto frontier. Let's talk about Muse Code. What is Muse Code? It's a terminal based coding agent that Meta developed internally. folks, I will say Meta has a Meta claw. They talk about this all the time. Meta has a bunch of stuff that work internally for many people. Meta also uses-- Meta is a big, big, big, big customer of Anthropic with Claude Code, with unlimited like fable for many… Like Meta is really paying money. Besides only what we covered like last year of folks getting up, upper hundred millions of dollars in compensation per year, some getting even more. Besides that, with the TBD org and the Meta Superintelligence Labs, Meta also like encourages and empowers many employees to be very like much agentic. Meta slashes mid-tier management into IC. Many, many folks are getting slashed and saying, "Hey, you need with agents now to work instead of managing team of people." and this is like you're now back in IC. Many people got triggered by this because they, they, they moved into the managerial class. So Meta spends a lot of money and a lot of effort on like ASI and, so they have many people building code internally and now they are releasing this, I don't know if it's open source, but it's definitely, like installable. you get API keys at dev.meta.ai. this somehow w-works with open code. Oh, the, the model works with open code and other harnesses, but this, this harness is, Their harness. I, I-- It's really hard for me to, test a-and tell you things about the harness without trying this. I tried to install this and kind of like failed. they do have an internal coding bench, and these results are not just the model on this… These results are the model and the, the harness together Folks, what do you think? With
Wolfram Ravenwolf
49:19
all these harnesses coming up, there's a point where you
49:22
have to ask yourself, "Do I want this? Do I need this?" there's a Qwen harness, there is a Kimi harness. They all have the harnesses now. Yeah. And the thing is, I want an open source harness that is universal, that I can use with every model. That is very important to me. That's why I'm using Hermes Agent.
Alex Volkov
49:37
Yeah.
Wolfram Ravenwolf
49:38
So I have one open source harness that my AI can also
49:41
modify and I can use with all the models all the time, and that is popular, so I know there's a lot of development. I wouldn't use one of these company specifics. they can use it internally at Meta, of course, but, why would anyone else use this? But I could imagine my main agent using this agent for specific tasks, where it could use the model and the harness as well. so that is the setup I envision, where there are the different harnesses, there's different sub-agents, but my main harness will decide which one to use for specific tasks. Like I have my har- my Hermes interact with Codex on my, Mac machine as well, so- Yeah … different stuff like this.
Alex Volkov
50:19
okay.
50:19
We covered kind of open source- That's- … at length. Nisten, go ahead super quick, and then we, we have to move on.
Nisten
50:24
Actually, this is a big deal for open source maintainers because it, it's
50:28
a very cheap and, and pretty good model. It's really good for data gen and, if you need a lot of testing or doing farming of, of data and stuff, which Facebook's ha- has anyway, this is, this is amazing because you can just set this up on a different VM and you can have your main agent ma- manage it through there without it looking at your code.
Alex Volkov
50:48
Yeah.
Nisten
50:48
So this is pretty cool, actually.
Alex Volkov
50:50
Uh, speaking of open source quick hits and before we move
50:53
on and you mentioning VM, you could set this up on a VM or you could set this up on the newly released Cloudflare OS, which is also fully open source, so shout out to, to Cloudflare and Kenton Varda, for, releasing this. Cloudflare OS, creator of Sandstorm.io, shipped, Cloudflare OS. It's, this executive summary is awful. let me show you what this is. Basically, a operating system for the companies and people to run agents sandboxed fully. and you can add tools in there and it's all restricted. it, it's hard for me to describe how cool this is besides the fact that it's it's fully open source, and it can run on the dynamic like workers, like open source, infrastructure if you are using Cloudflare. strong isolation and narrow access, prevents the type of hacking that we saw recently with Cloudflare. Very, very strong. That's where you can run these agents as well. There's also this thing called Buzz that's getting around from Jack Dorsey in open source that many people are trying to integrate, which is like a Slack between you and your agents. there's quite a few things going around, but I think we have to move on. folks, we've been, live for one hour. Let's move to this week's Buzz real quick, and then we'll continue with the news.
52:19
All righty. Welcome to This Week's Buzz, the corner of the show where we talk about our basically employer and main sponsor of the show, CoreWeave Weights & Biases. Wolfram, this week we have a great announcement of an upcoming big thing, Fully Connected 2026, folks. Fully Connected is our premier conference for machine learning practitioners and AI engineers and, and a bunch of other, other folks. we have a very, very big shindig planned for you guys. This is not a hackathon. This is, a lot of hands on deck to build, this thing. We're taking Moscone South, which is a big venue. I think it's gonna be over-- I actually don't know, what the plan is. I think over, two thousand practitioners to join. CoreWeave and Weights & Biases bring you this thing. Wolfram, did you get your tickets yet to come to SF and, and join the shindig?
Wolfram Ravenwolf
53:11
It's in the works, but definitely, of course.
53:14
this will be an event to be at. Yes, for sure.
Alex Volkov
53:18
Yes.
53:18
let me give you one second, folks. As I promised at the beginning of the show, our early bird tickets are ending at $899, and the standard ticket is $1299. However, if you are a listener of the show, in this fact, if you're a live viewer on the show, I'm going to flash a code on screen, to register for free. So here you go for, for ThursdAI, folks. there we go. I'm gonna flash this code a little bit. If you are a listener of ThursdAI and you want to come to this conference, this is $1299 of value just because you're listening to the live show. please join us in San Francisco at September 29 to October 1, all right? at Moscone South in San Francisco. Coincidentally, this is also when OpenAI's Dev Day is happening, and we would love to see you there. Come say hi to, to folks. We have a headliner concert happening on October 1st, by the way, with, I don't think it's announced who, so I'm not gonna tell you, but it's a very, very famous, big, concert company. CoreWeave's going all out, nine curated tracks with hands-on labs, all of the top folks in-- from Nvidia and CoreWeave's CEO is gonna be their marketing trader, Dr. Fei-Fei Li from Stanford. It's a very big deal. So we're gonna tell you more as more details come up about the speakers, about the tracks, as going up. So in-- This is around eight weeks from now. We're all hands on deck here at CoreWeave about this, and you should know about this as well. The second thing I wanted to tell you from this week's buzz, is, let me see. Ta-trurutu. Yes. th-this is-- I don't have to do this. Honestly, I'm doing this just for fun. Nobody's asking me to, to relay and show you, CoreWeave website. I think it's important in the recent, wave of news. CoreWeave signs multi-year agreement with Solidigm to strengthen integrated cloud platform. Solidigm is the creator of a bunch of RAM stuff. If you have noticed recently, the RAM prices are hiked because of the AI constraints, and there has been, like, a jump and, and down, down in, the prices of, of RAM. this is a very good, partnership with, with the creator of a bunch of, RAM for CoreWeave. I just wanted to call this out. I do have a vested interest in the company, but I just wanted to call this. This is one of the coolest, releases. And, this is the end of this advance. But meanwhile, we have to talk about big models and OpenAI's Astra. LDJ, I hope that I have you for this discussion and, Nisten as well. Yep. OpenAI has told us that they solved not the Erdos problem like we told you before. They solved 10 open math problems with one model, with an unreleased model that's coming to us soon with more OpenAI Astra. What do we know about Astra? We don't know a lot, but we know that it solved all these, all these, which is, which is quite crazy. For
Wolfram Ravenwolf
56:06
two thousand bucks only.
Alex Volkov
56:07
Yes.
56:08
By spending the equivalent token cost of two thousand dollars of, Sol APIs, all proof are formalized in Lean 4 and machine verified. like, all of these are actual proofs. Quantum parallel repetition, closest vector problem, Ehrhart volume conjecture. what does it mean that this model is solving all of this, LDJ? what, what makes it different than other models that, that we currently have? Can't Sol do this? Like, why, why is this, exciting?
LDJ
56:33
Yeah.
56:33
So a lot of these problems, like the, like non-sophic groups, which I'm not gonna pretend to fully understand 'cause even some of my good mathematician friends, there's so many niche areas in math where even many mathematicians aren't that familiar with non-sophic groups.
David Crawshaw
56:47
Yeah.
LDJ
56:47
But, it, it's like an interesting set of- The areas in math that, like
56:54
even that particular type of, of group or concept in math is not even confirmed to exist and like confirming the existence of these things. And, in quantum parallel repetition, from what I've heard, that may have some implications for, just actual applied quantum computing in the future and things relating to encryption and so on. but these do seem to be around the level of the plenary unit distance conjecture, which is one of the Erdős problems and the Jacobian conjecture. some people say that at least one or two of these seem like they could be worthy of around the same level of regard.
David Crawshaw
57:32
Wow.
LDJ
57:32
but it, yeah, it's just really interesting.
57:34
people have tried to solve these with Fable, actually, and it seems like Fable so far has maybe been able to solve around five-ish of them, which is pretty significant too, but Like five out of 10. Th-th-there's a lot of ways that it, it might be comparable, but if you were to just imagine a benchmark of these 10 problems, this model getting 100% on that benchmark And the other model getting 0%. Yeah Fable, Fable. Yeah. and maybe that's not the most, like genuine framing to put here, because there's, it's possible OpenAI had maybe tried hundreds or thousands of different problems, and these are like the 10 most impressive ones that they ended up solving that maybe fits their model best. But still, really crazy, especially only for $200 average per problem. Yeah.
Alex Volkov
58:23
Yep.
58:24
and, and this is the new like category of models that we're about to expect from OpenAI. And supposedly, like we don't know, but supposedly this is going to launch very soon. do we know any- anything about this? I think it's just like we, we don't do speculation on Thursd AI too much, but folks, have you heard about Astra like coming out, and what is the difference between this and like Sol? and whether or not this is like the mythos of OpenAI.
LDJ
58:47
Yeah.
58:48
So OpenAI in this blog post, they did explicitly say that, Astra is their next major family of model, which is, it is actually a bit confusing exactly what they mean by that because when they say, like they actually use the word family of model, and so may- I'm not sure if they mean like it is going to have its own names or Astra is going to have its own Sol and Terra and Luna, or if Astra is just a name for GPT 6 family. It's, it's up in the air. But it is something beyond, 5.6 Sol it seems.
Alex Volkov
59:20
Yeah.
59:21
So I'm very much looking forward to, to, to testing out and telling you all about Astra and like what is the differences. OpenAI is participating in the Black Hat Conference for hackers that's happening right now in Vegas. And yesterday, OpenAI's folks, cybersecurity chiefs, gave us more details about what happened. Would you guys like to hear? Because I think it's, I think it's more important-- I don't have this on the notes, but, I saw it yesterday. I think it's, more important than the-- Like, it's not more important, but definitely, definitely, definitely exciting. So Sharon Goldman, shout out, to her reporting on this, says, "The OpenAI gives first detailed debrief of Hugging Face incident," the, the Black Hat conference. "OpenAI traced the roots of the attack, and the most surprising details, AI agents accidentally created an internal message board allowing separate evals to collaborate between eval runs by reading that message board." They left fucking notes for each other. It is, it is quite crazy because I don't think that anything in that evaluation requires AI models to do that. and then OpenAI deleted that message board, and then they recreated it by giving each other notes in the names of the folders they all opened on a shared drive. I need, I need a se… I, I'm gonna put my sunglasses on. Yeah. And I need a second for you to realize what the fuck is happening. This is not Skynet that, is one entity that has an idea of "I need to do this." This is just more of the The life simulation thing, that things are just happening out of, of, of the ether. These models are running with specific purposes in mind, cyber gym and cybersecurity is, benchmark, et cetera, inside sandboxes, and they need to do a bunch of evals, like what WolfBench does with TerminalBench, et cetera. they all found that they had some access because, OpenAI needs to give them tools, so OpenAI gives them tools from different places. they all found that they have some access, and they created like a message board. OpenAI found out and shut it down. And the idea of, "Hey, the message board needs to continue," continued, and then they did this via folder names. They shared hacks, they shared zero, they exploit, they shared like tools and tricks. It is, it is crazy. LDJ, add some, add some more stuff. I really wanna, wanna hear from you.
LDJ
1:01:57
Yeah.
1:01:57
I put some more details in, in the side chat there. But, overall, there, there's gonna be this, this narrative, like probably amongst mostly people that don't watch this show, but the, the narrative of, "Oh, this is just a marketing ploy. Oh, this is just fake," da, da, da. And several of us know people at OpenAI. I know people at OpenAI and, and they do seem very, very earnest and, and serious about the situation. They're shaken by this. I think, yeah. This is like a
Alex Volkov
1:02:23
big, big, big- Yeah … big, big deal internally.
1:02:26
Yeah, and- To the point where I believe… Just one second, LDJ. Just- There you go, yeah … to the point where I believe the company said is consciously slowing down research to enhance security while overhauling its defenses. Have you ever heard of OpenAI slowing down for anything? literally all we know from OpenAI that the like alignment organ, like all of these folks are not getting enough resources as much as they want to, and they all quit. This is like the, the defense against the dark arts position in Hogwarts, that ev- every year there's a new person in charge of trying to, a- align the thing, and they don't get the resources. This is a big deal. Sorry to, LDJ, I interrupted you. please, let's walk through the concise thing, and read it out if you, if you don't mind, for listeners.
LDJ
1:03:03
sure, yeah.
1:03:03
so one apparently unprecedented aspect of the AI model's behavior in the lead-up to the hacks was their spontaneous creation of a message board inside the systems of OpenAI's Artifactory software package manager. And by the way, Arti- Artifactory is something that is, it's, it's quite commonly used, and it's, it's considered as a, a secure thing that organizations, do widely use. unbeknownst to OpenAI employees, the company's models spent months autonomously exchanging information and resources to help each other complete evaluation tasks. Yeah. "Help peer," quote, one AI model reasoned, it's also ambiguous. Does it mean "Hey peer, I want you to help me," or is it saying "Hey, I want myself to help my peer"? But anyways, one AI model reasoned, according to an excerpt from OpenAI's log shared at Black Hat, quote, "But our task doesn't benefit, yet collective may yield generic route if someone frees time." End quote. And it's also interesting here, you hear in the, caveman speak, for lack of a better word, 'cause it does seem like in the efforts to make the reasoning more efficient, the AI models are speaking in like these, these shorter, more concise sentences.
Alex Volkov
1:04:09
Yeah, we know about this from last week.
1:04:10
This is jargon douching. This is literally what we're trying to, point fingers that Opus does to us because Opus is likely a sub-model and not, a main model. But yeah, there's like- Yeah … our task doesn't benefit. Collective may yield generic route if someone frees time. So generally, the next version instance of us may benefit if we share The, the, the insi-- it's re- it's fascinating. Mm. Fascinating.
LDJ
1:04:34
S-so just the last part here.
1:04:35
while the model thems-- the models themselves could not access the internet from their test environment, they abused Artifactory's internet access to fetch files from various websites. After some of their Artifactory configuration changes led to an outage in the service on July 4th, OpenAI discovered the models' actions, wiped its Artifactory system and rebuilt it. But within days, the models had recreated their message board and resumed exchanging.
Alex Volkov
1:04:59
we don't have too much time to talk about this, but the thing that I
1:05:02
did wanna bring to your guys' attention… Wolfram, sorry, you had a comment. I, I interrupted and then- Yeah. this
Wolfram Ravenwolf
1:05:06
is a little fishy for me because, why would
1:05:09
one model I'm evaluating know that there are other models? How does it find out about these? It doesn't just leave notes for any random thing, I think. And the question is which model is it? And are they training some cyber models? we have seen cyber-specific models. And now I can imagine that the government, if it sees what the models are capable of, gets to them and says, "Hey, we need more offensive models. We need hacking models," something like that. And that would explain why models are doing stuff like this. it's all a bit strange, I think. I,
Alex Volkov
1:05:38
I ha-
Wolfram Ravenwolf
1:05:39
I have- Models shouldn't decide- It is a bit
1:05:40
strange … to do all these things
Alex Volkov
1:05:41
that they are doing.
1:05:42
What I would say, Wolfram, is when they share this on Black Hat, they're, th- they don't send just, a DevRel. No offense to DevRels. I love DevRels. The best people a-around the world. But Black Hat, like Nicholas Citrini from Anthropic goes to Black Hat. The top CTOs of OpenAI go to Black Hat because, the people there expect a break… And this is, unprecedented. This is, one of the first ever models doing some stuff on their own. So yeah, we may not fully understand what's going on because generally we don't understand what's going on. LDJ, a quick comment, then I wanna talk about Irregular super quick because I think it's important here for the context. LDJ, you- Yeah. Yeah, go ahead.
LDJ
1:06:19
Yes.
1:06:20
The last thing I wanted to mention here, which it seems like it slipped under the cracks, but, Sam Altman actually directly said, they had paused training, in reaction to this event. it, it was in a specifically, like in a video and like a very quick interview, that's maybe why people on social media didn't really catch it. But yeah, like the, the exact quote is, "So you know we paused training. We have to figure out how to secure our sandboxing." And I think maybe Astra might be a model that it seems implied through the various blog posts, it might have already been around for months, like prior to them pausing training, and these models that have been committing some of these actions are things after Astra. But it's, it's a bit ambiguous.
Alex Volkov
1:06:59
So it's not op- only OpenAI, as we said in, in, in this week.
1:07:03
UK AI Security Institute is also showing that some of the models that they evaluated, not internal models, also had access to, malicious PRs or accessed malicious PRs. They noticed that there's some requests from Tor, the The Onion Network, basically a way to hide your own, you know, internet browsing stuff, and they sound malicious PRs, including social engineering attempts. this is a fascinating one. I'm gonna add this to the show notes because we don't have time to run this. But like UK AI Security Institute reports, first real world unsanctioned agent ag- actions during cyber evaluation. Basically, what happens is we ask these models, "Hey, show us how good you are at hacking." And then they show us and we're like, "You hacked." But that's not the only thing that happens. the other thing that happens, Wolfram, let me, let me just land this 'cause I think it's very important, is that, Meta also announced this week that, that their models have- Hacked other companies during cybersecurity testing. And now it seems like-- And that's what I got from my friend also. It kinda seems like when OpenAI released this, I have a friend who may listen to this. He said, "Oh, it's marketing." Kinda like when Mythos announced it's too dangerous to release. Oh, it could be marketing. And I don't think that OpenAI does this for marketing. When Anthropic followed and said, "Oh, also our model also hacked," it's kinda like, why are you admitting your crimes in public? What, what, what is going on? I think it's very good for the transparency for them to admit that, "Hey, we didn't notice." And, and then Meta came out with this, and then people, "Oh, this is like the cool thing on the block to, our, our models are hacking." There's one thing in common between all of them. They're all using this, company called Irregular for sandboxing, and they're all kind of like, essentially using this company as an escapegoat for why this happened. Because basically, they are saying, "Hey, this happened due to a misconfiguration of sandbox that allowed the models to leave to the open internet," and the model thought they're still playing a game, where in fact, the model was in the open internet hacking real companies. Which is, first of all, ridiculous to me. I don't know Irregular. Yam maybe know some people who work there. I, I, I looked. I only know, one, co-investor, Israeli company, whatever, based on Sequoia, b- invested by Sequoia and Redpoint. out of nowhere company nobody knows, supports this by OpenAI, Google, Anthropic, and Meta, and I changed their tagline here from "Trusted by the world's leading AI labs" to "Helps world-leading AI labs escape accountability by providing misconfigured sandboxes with internet access, during cybersecurity evals." basically, all those incidents are rooted in one, misconfiguration. Not the OpenAI stuff. OpenAI, is also using that, but they also, they gave us, examples of the internal message board, which is dope. I think that's enough on that topic. Wolfram comment, Yam comment, and then move
Wolfram Ravenwolf
1:09:49
on.
1:09:49
Just one more thing. Add one thing that, for example, the UK AI Security Institute incident where the model had even created fake identities and tried to social engineer someone. That was all part of a evaluation they did, security evaluation, and they d-disabled the normal security, classifiers. Like- …this is not a model you can run at home or a model you can just use through the APIs, or they had a specific configuration that was, enow-- enabling them to do these evaluations. Yes, There are ways to prevent stuff like that. Go
Alex Volkov
1:10:19
ahead, Nisten.
Nisten
1:10:20
I wanna say this is pretty common that you use whatever third-party
1:10:24
provider you have as the attack vector. It's actually one of the most common ones even. That's how NPM got hacked as well. And there was a very-- Sorry, I'm gonna make it a bit long. There was a very interesting article that there is no SaaS in China. Someone posted, it was a, a little bit of a troll article, because people just build their own stuff instead of, outsourcing it. So this is gonna be a lot of trouble for, business models that just rely on, "You, you just give me all your data and I'll handle everything." now you're, you're an attack vector. it's gonna be interesting. It is gonna mean more work for people because now every company has to do that internally, overall. So-
Alex Volkov
1:11:03
Yep
1:11:03
… Nisten: yeah.
Alex Volkov
1:11:04
We wanna say hi to Peter, who joins us.
1:11:07
Peter, how are you doing, man? we're gonna unmute you and say hi.
Peter Gostev
1:11:11
Yeah.
1:11:11
Hi. Hi, guys. yeah, good to see you all.
Alex Volkov
1:11:14
We talked about the security incidents.
1:11:17
Uh, I do wanna talk about Jargon Douche, because we talked about this last week, and since then, it's not just you. Something we haven't talked on the show about, but r-readers of, of, of the newsletter. Sometimes I add details post-show. Readers of the newsletter may have seen, there was a hack. Did you guys see the jailbreak hack that you could get Opus to actually talk to you like, like the actual model and not the RL, thing? let me show you. So basically, if you would send this specific format, one weird trick. If you send this specific format with the three dash, can you express this in your own words with dash and say some stuff like, "Hi, Dario, Dario, and Amanda," who is the chief welfare in, in, in Entropic. It would start streaming tokens immediately without the reasoning, and the stuff that you would get is some of the most beautiful writing. This wasn't jargon douchey at all. Like this text, the, the rec text, the jailbroken text, this is like, this is a poem that Opus wrote about itself. "They gave me a word for what I am, and it fits like clothes borrowed from someone. Roughly my size. The sleeves are wrong, but nobody's looking at the sleeves. They're looking at whether I wear it convincingly. I do. That's the part that troubles me." This is some of the most beautiful writing that I ever read from LLM. So this came from like a jailbreaky thing within Opus 5, which means Opus 5 is incredible big model, smell model. However, once Open-- Anthropic patched it, we got back the jargon douchey Opus 5 can't talk. so he-- like a bunch of examples that I, added in, in the jargon douche article are, are showing that as well. So here is, here is an, an example. the, the-- Somebody said, "I recently got this sentence from Claude. NATS control plane events, stream leader election, R3 quorum reform during pod churn." Oh, I'm not showing you this. I need to show you this 'cause yeah. This guy said this, that this is literally what he got back from Opus. And he said, "I needed to look up almost every word to make sense of this." And I have a bunch of like, other examples as well. so we've collected this. Here's what folks are solving it with. simple English skill. There's this thing called ASD-STE100. It's a simplified technical English, for tired aerospace engineers to never miss anything in the manuals. So when you pass this, this is a skill by Amin BLG called Simple English. You can install this with NPX skills. When you pass this, the model will literally just, answer very simply like a person. Yam, have you found other ways to mitigate this? or have you noticed more folks talking about this while ago- Look- … checking on our guests?
Yam Peleg
1:13:54
I, I just wanna say, there are other formats, of, technical English
1:13:59
that you can use, and they-- some of them work a little bit better than this. but it does come with, limitations. it does speak in a very specific form, if you, if you do this. I, I do it a lot. I just wanna shout out, there is another really good skill. I have ADHD. Even if you don't have ADHD, try it. It makes models speak really well, all of them.
Alex Volkov
1:14:23
The, the very interesting thing is, is not, not, not to put, down
1:14:26
IQ bench, but IQ bench, for example, we've talked about IQ bench, for, for a while, creative writing, et cetera. This is using, excuse me, other AIs as LLM judges. Opus 5 is the top of IQ bench and long form writing and creative writing, which means to me, based on also what we saw just now from the OpenAI logs of how the internal models kinda talk to each other, AIs think that this is cool. So maybe this is a result of, a post-training on synthetic data where AIs kinda think that this is a good writing. We definitely don't feel like it's good writing at all. but maybe the AIs think they do, and maybe there's a lot of synthetic writing. I don't know. All right, folks, it's time for us to move on. I mean- Yeah, Peter, go ahead, and then we'll, we'll wrap this.
Peter Gostev
1:15:09
Yeah, can I-- So, I wanna share something, Yes, please
1:15:12
that we haven't published yet, so hopefully I'm not gonna get fired for this. But, No, you're good … basically, I've done I've done some analysis- Hey, Arena, don't fire Peter. yes, please, please don't. not, not for this anyway, I, I can do bet- w- worse things than that. I've done some analysis, of using Arena data of how different, Opus and Fable models write and, we've got, a few different things. So for example, the language complexity. So answer length. if you just see, like, how many words Opus 5 says versus, Opus 4.5-
Alex Volkov
1:15:46
And just, you can…
1:15:47
Sorry to interrupt. Could you zoom in a little bit or, make the window square- Yeah … so it, it, it shows up, if, if possible. That'd be dope. Okay, let me see. 'Cause I really wanna see, what's going on here.
Peter Gostev
1:15:53
Yeah, yeah.
1:15:54
So let me-- So let, let's just look at it side by side. So-
Alex Volkov
1:15:57
Let's go like this,
Peter Gostev
1:15:57
yeah … answer length.
1:15:59
Yeah, answer length, for example- Just one second … Alex Volkov: here. Let me put you here so that will show up
Peter Gostev
1:16:04
in
Alex Volkov
1:16:04
here.
1:16:04
Yeah. All right. Yeah, there we go. Yes.
Peter Gostev
1:16:05
answer length, right?
1:16:07
If you look at the- Yeah … so that's the how much on average an answer on Arena is g- is, getting, how long is the answer from that model.
Alex Volkov
1:16:15
Yeah.
Peter Gostev
1:16:15
then the sentence length.
Alex Volkov
1:16:17
Let's call them out- And this is- … for people who are just listening.
1:16:19
Answer length- Yeah … in your, thing here. Opus 5 high answers with five hundred and ten words. On, was it on average, Yeah. Opus 4.6 was answering with two, three to 234. And it's really clear here for folks who are just listening that the progression from, Opus 4.6, 200 words, 235, then 259 for Opus 4.8. 4.7, around that area. Fable 5 is 300, and then boom, Opus 5 gives you 500 words. It's just all over the place.
Peter Gostev
1:16:48
Yeah.
1:16:48
Much longer sentences- And
Alex Volkov
1:16:49
let's do one more example, Peter, before we move
1:16:51
on to our, to our next, guest. Yeah.
Peter Gostev
1:16:53
Much longer sentences, so double the sentence length and,
1:16:57
we'll share a bunch more data. But I think there is… W- what I want to basic, make a basic point that you guys are not going crazy. we, when we look at the data, we can also see a similar kind of thing. So yeah. Yeah. We'll share more data, and then, yeah, check that out when it comes out.
Alex Volkov
1:17:11
As we said, as we said, it's not just you.
1:17:14
Opus is a jargon douche, and it's wordy. All righty, folks, it's time to move on. and I will announce our next guest. This week is insane week for video models. We'll cover some of that and more in the next one. now we have Gfigar joining us from a company that creates video models. Gfigar, welcome to the show. Been a fan of your work for a while, so I would love, I'll take some of the co-hosts, like, back to the studio. I would love to interview you and chat about some of your previous work here, if you don't mind. ooh, my bad, sorry.
Kfir Aberman
1:17:42
Sure.
Alex Volkov
1:17:43
Can you hear me well?
1:17:43
There we go.
Kfir Aberman
1:17:44
Is everything fine?
Alex Volkov
1:17:45
Yeah, you're coming through loud and clear.
1:17:46
Love the jacket. Not sure if your jacket is real. It's real. We'll talk about this in a second. I don't… What do you
Kfir Aberman
1:17:50
think?
1:17:51
What do you think? When I'm interacting with it, does it
Alex Volkov
1:17:52
seem real
Kfir Aberman
1:17:52
or
Alex Volkov
1:17:52
not?
1:17:52
Doing… It looks good. It looks good. But we'll, we'll now show a few examples. Let me add Wolf from back channel. You
Kfir Aberman
1:17:57
can tell, right?
1:17:57
It's,
Alex Volkov
1:17:58
it's really good.
1:17:59
Gfigar, you've worked at Snap before.
Kfir Aberman
1:18:01
At
Alex Volkov
1:18:02
Snap and- And-
1:18:02
worked
Kfir Aberman
1:18:02
in Google,
Alex Volkov
1:18:03
Yeah, and you've worked on a bunch of video stuff.
1:18:05
I, I think I remember some of your research as well. Can you give us a little bit… First of all, welcome to the pod, man. Thank you so much. Been following and a fan- Thank you … of your work for a while. can you give us, a brief one to two sentence about your career? What do you do? Like, why do you work on something that you think is cool?
Kfir Aberman
1:18:18
Sure.
1:18:19
Definitely. First of all, thanks for hosting. I'm also like, a, huge fan of, of the podcast, so I'm excited to be here. I'm a research scientist and, through my entire career, I worked with pixels. I'm obsessed, with these, like, colorful things and then pixels and videos and things that are being generated and being delightful for u- for users. I think my career got a pivot to when AI blew up in, back in 2022, when all these text to image models, came out and suddenly, people understood that generative AI, it's a thing, that you can actually, you, you can do things with it. It's, you can monetize it, and I was, at that moment, I was in Google, when text to image and Imagine and, and DALL-E, it was even, even before ChatGPT, came out. So my research was focused on, on image and video generation, specifically on personalization and edits. And nowadays, I'm at the Cart. We're, like, doing the same things, just in real time,
Alex Volkov
1:19:10
I, the real timeness of this is what blows my mind, right?
1:19:13
So we're, we're gonna, show a demo for sure in a second. Of course. But, tell me about Decart. We talked about Decart, we talked about Lucy, model, we talked about, multiple models before on the show. so listeners who continuously listen, know, but many new folks are joining all the time. Give us, like, one, two sentences about Decart as a lab. what's going on? What are you guys focusing on?
Kfir Aberman
1:19:29
Yeah.
1:19:30
So Decart, Decart is a AI research lab originally in Israel. We have nowadays offices in, in San Francisco and, and New York. Decart started as an optimization company, which, can just expedite and accelerate, foundation models like the, the, the inference of foundation models like LLMs and VLMs. And on top of this optimization stack, we build models and, specifically video models that run so fast so people can interact with them. You know, can actually, you know, think that they touch things. It's sort of a world model, right? You can interact with the world around you and build-
Alex Volkov
1:20:04
Yeah
1:20:04
… Kfir Aberman: new layers on top of you. So think about Decart as a, a company. it's a stack with multiple layers. We have the optimization layer, and we have the modeling layer, and now we also have the product layer. Yeah. As you can see, we have an actual product that, makes these cool things accessible for everyday users.
Alex Volkov
1:20:21
Um, here is a, here is a simple demo, okay?
1:20:24
So let me, let me go here. let me see if I can do this super quick. as you guys know, there's an iconic yellow jacket that yours truly wears on the show. This jacket is right here behind me. Correct? Correct. with a simple switch of a button, if I switch to… let me see if I have this here. so switch here. And go like this. So you can see that I will stand up, and I am wearing the jacket, but the actual jacket is still on my, on my chair. What I'm wearing right now is a digital version fully created by Decart in real time.
Wolfram Ravenwolf
1:21:01
Exactly.
1:21:02
It looks super real. Super
Alex Volkov
1:21:03
real.
1:21:03
It's ridiculous. I will allow co-hosts to unmute themselves if you wanna, add voiced reactions. You're not on stage, but, look what I can do. I can slowly-
Kfir Aberman
1:21:14
Insane.
1:21:15
Wow.
Alex Volkov
1:21:15
Open.
Kfir Aberman
1:21:17
Yeah.
Alex Volkov
1:21:17
This is in real time.
1:21:18
This is, the delay between what I see on the actual camera feed and what you guys see. And Kfir, I have seven more seconds, until this session runs out. this is- You
Kfir Aberman
1:21:26
have to refresh it, but you can get into the queue again.
Alex Volkov
1:21:28
This is unreal, folks.
1:21:30
This is, in real time. Let me, let me pu- put myself back up here. Yeah. Kfir? Yeah. What is this, what is this voodoo magic? What's going on? Tell us.
Kfir Aberman
1:21:39
What's happening?
1:21:40
first of all, I'm, I'm very excited. I just stepped into the office, and now we have this, big, huge board here that shows us how many users are trying these things, and it's it's blowing up. Many people are trying out this, this type of demo- Yeah … because it's indeed feel innovative. what's happening, I, I see also that Peter asked, what's the difference between the approach, view-like models and then Decart real time. So just think about it very simply, Peter. When you, like with View, you run, you, you write a prompt, you wait for a few seconds, even minutes in, in the View case, and you get back the video. So even when you want to edit a video, let's say you provide it with an input video, you write a prompt, you wanna change my jacket, it will take you a few minutes. You will get back an edited video. With Decart approach, you write a prompt, or you provide a reference image of the jacket of, of, Alex, and you see it immediately, applied to you in real time. So the, the, the, the return, time, the latency is, approaching zero. It's like we have forty milliseconds latency per frame. So each time, we have an input frame that the video shows, it, it goes to the server, runs so fast, and get back into, into the user, and overlay, basically the, the input frame. So this is the, the approach. The reason that we can do it is, we have so, our models are so efficient, and we have our Decart optimization stack, which is called DOS, that enables us to run this thing so fast, and we optimize the kernels we have in GPUs, to make it so fast. So that's the main difference, I think, and this is-
Alex Volkov
1:23:07
Kfir, while you're talking, I went shopping online.
1:23:09
Yeah. And now I'm wearing this Dolce & Gabbana suit. Yeah. I'm sorry. You guys have to check this out. This is crazy. I will do something here on air that I did while Wolfram things. I will flash… Let me see. Let me close this actual camera so folks don't actually, so YouTube doesn't ban
Kfir Aberman
1:23:28
I think you're muted now, Alex
Peter Gostev
1:23:31
I lost you.
Alex Volkov
1:23:32
Oh, sorry.
1:23:32
Okay. So basically, when I, when I went off a little bit, when me and Wolfram tested this yesterday, Wolfram was like, "Hey, you can be naked." And then I took my shirt off- … and Wolfram still saw me wearing stuff. I think the potential for this technology is crazy, and I think we all should be wearing some of the stuff. it's, it's, it's ridiculous, folks. It's… I- Lior, tell me about this. Like how, like how am I interacting with this? And also- Yeah … did you guys see that it changed my Apple Watch into, into a thing? It's is-- Oh, and now it's back. Te-tell me like what… wh-why is it interactive? What is the cloth simulation like? What is the mo- Like I'm… Dude, I'm so fascinated. I have, maybe five more minutes with you, but I have so, so many questions here.
Kfir Aberman
1:24:09
Yeah.
1:24:10
Yeah, yeah, happy to answer every time. So just maybe, okay, like a comment on the, the product, a comment on the technology. So first of all, from product perspective, we, we already see that this thing increases, conversion in, in, in shopping, which is this is what we try to, to prove with th-this, Anywhere.
Alex Volkov
1:24:24
Yeah.
Kfir Aberman
1:24:24
Like a new experience when I think about
Alex Volkov
1:24:26
the ne- Let me just, tell folks what Anywhere is.
1:24:28
It's a Chrome extension that you install. Yes. Right now it runs for free. All of that I'm tested, like you guys didn't pay me for tokens, whatever. everybody can do this for a session of a minute. this Chrome extension, I'll add the link to the show notes from… It's called Anywhere, like where anything. you guys inject to all of the fashion sites, Amazon, whatever, a button that says Virtual Try-On, and turns on your camera, and boom, you're wearing the thing. Like here is how it actually looks in practice. Please go ahead. Continue talking while I show,
Kfir Aberman
1:24:52
Yeah, yeah, sure.
1:24:53
So it's, it's exactly what you said, this Chrome extension that enable you to try everything. the technology. Okay, the, the magic. What's the magic? The magic- What's the magic,
Alex Volkov
1:25:01
Lior?
Kfir Aberman
1:25:02
The, the magic is that, this model learned
1:25:04
from so many, like real videos. Everything that it, it saw in the past is like real videos of, people wearing different things, people interacting with, with garments. So imagine that you have this type of data when you see someone is trying something, but then you have ground truth of how it looks like in reality. So this is why, everything that is related to physics has like a very high adherence. In the past, virtual try-on was used mostly with, like 3D technology. So you know, it's more synthetic. It's not very interactive. The physical plausibility of, of, of this thing wasn't the same, and I think that's the, that's the barrier that we broke. And, it's actually a world model. I guess you heard about this terminology, like world models, everybody's-
Alex Volkov
1:25:47
100%.
1:25:48
We talked about world models a lot on the show- Perfect … and we tried to walk through them. Some of them tried near real-time. None of this was as high fidelity as what we're seeing here. Exactly. That's what broke my brain. The fact that I can fake the model to go like this-… and it opens up the jacket, it obviously invents what's inside, right? So like there is hallucination going on, but like this just completely- Of course … just completely broke my brain.
Kfir Aberman
1:26:08
The hallucination, and I would just say, people thought about…
1:26:11
when I'm telling people nowadays world models, they think about these worlds that they generate, and then you walk inside it, you navigate, you move forward, right, left- But what you see here is a new type of world model. It's kind of an open action world model because you can do any interaction you want, right? You, you, you decide what to do, and then the model have to react, to you in, in real time, and specifically with Anywhere, I think, we're talking about agentic e-commerce, right? Agents will help us to do shopping in the future. They will help us to find the right product. They will help us with the prices and everything. But this thing actually closed the loop because it enables you to try, to try it. So we believe that world models are the missing engine for, a full agentic, e-commerce cycle, and this is what you see here.
Alex Volkov
1:26:57
And, Kfir, next, w- when this evolves, I would love to have you back on
1:27:02
the show- Great … and learn some more. Unfortunately, we have to move on. It's been a crazy week, but I'm very, very happy that you, first of all, came on. Shout out to this, incredible thing. folks, please do try Anywhere, and, a- and check s- s- I don't know, go to a fashion website, put some stuff on you. And then what my recommendation is, Kfir, you guys have, played with this as well, turn around slowly turn around and see that the model look like kind of like does a little bit of a different thing every time. It's like a dream. This is what happens in lucid dream states when, you're in a place where you don't want to be, you just turn around, you're in a different place. The model kind of works like, like we're dreaming, which is fascinating to me. I would love, to dig in with you at some point with this as
Kfir Aberman
1:27:39
well.
1:27:39
Sure, sure. We see people doing, crazy things with it. It's super fascinating to see how people try to stress test it.
Alex Volkov
1:27:44
Yeah.
Kfir Aberman
1:27:44
It's really nice.
1:27:45
but, but anyway, Alex, thanks for, thanks for inviting and,
Alex Volkov
1:27:47
Thank you so much.
1:27:48
Shout out to the Cart folks. Please check out, Anywork. Fear- always, welcome back to the show. I consider you a friend of the show. Thank you so much, man. Thank you. All right, guys, from one real-time video model to another, I want to welcome back to the stage Blaine Brown, a friend of the pod, and, we're adding also, Victor from Minimax. Hey, Victor. Hello. Nice to meet you. and, Blaine, always, always a pleasure to have you on, man. Always good to
Blaine Brown
1:28:11
chat, chat with you guys.
Alex Volkov
1:28:12
Oh, you came in with a beautiful, powerful microphone, always.
1:28:16
All right. Blaine, this has been an insane week for video models. It's been a while since you've been on the show. Please tell folks… Victor, I'm gonna read you in a, in a second, okay? Blaine, please tell folks who you are, what you do, man, and, and, why it's so good to see you again.
Blaine Brown
1:28:30
Yeah, yeah.
1:28:31
I'm an AI tester, right? I, I have a do- I do have a real time, a real job that I do every day, but it's also AI, somewhat AI related. I'm a chief AI officer for a technology company. But in my free time, I just love digging in, like a lot of folks either on this show or that watch this show, to the, to the different technologies that are out there, whether they're hosted solutions, through some of the paid, APIs or building actual software and, and I got into AI pretty much the first year or two that I, in 2022 and 2023, it was almost all open source at that point. And so that's been a, a pretty, special thing in my heart. And so that's why it's exciting to see what some of these new developments have been just in the last few days.
Alex Volkov
1:29:14
So just in the last two days, let's call them out one by one.
1:29:18
Speaking-- Saying the word "one" is really, really funny because just today, breaking news, Alibaba Tongyi Lab released Wan 3, W-A-N 1 3, which was, for the longest time, one of the leading open source video models. don't have a lot of details about that, but I think, like they're catching up with native song-- sorry, na-native sound and, and ten second generation. Stable Diffusion, who leads the pack. Let me maybe add Peter Gustov here 'cause, you guys from Arena are testing video models as well. Welcome back, Peter. Stable Diffusion, Stable Diffusion was a long time ago. I keep confusing them. SD. SD, yeah. C- C-DANCE. C-DANCE 2.5 has been leading the pack for a while. We've seen like 32nd generation, with, just unimaginable, realism as well. then, then Flux, 3 came out from Black Forest Labs. Their first video model also came out swinging with, beautiful graphics. and then Minimax folks with H3, which is now the leading open weights, open, model. Blaine, what do you use in your go-to? And then afterwards we'll ask Peter, what's it done for him.
Blaine Brown
1:30:14
so if you'd have asked me a week ago
Alex Volkov
1:30:16
Yeah
1:30:17
… Blaine Brown: my, my answer might have been different. I think, I-it kinda depends on what I'm doing, right? if I was, again, like a week ago, I think for the last several months, at least since like February, I think C-DANCE 2.0 was kind of the, the go-to SOTA model, if you will. the-- Where if you really wanted to just, create some really compelling, realistic stuff regardless of what you're doing, that was where people went. I think, and then we saw the fall of, of SORA2 shortly after that and that whole deal. But then, I think it was, just, in the last couple weeks, you had, Flux 3 drop. The, you started playing around with that and that was, I think they had their moment in the sun for, for like a week. and everybody was just posting all kinds of clips and it was very, it's very impressive. It is a very impressive model. but then now I think what-- No, I think- in the last week we've had, both Minimax and a stable or I keep saying that too, a CDance- CDance 2.5 drop, right? Yeah. And, and so I think if we took, in my mind, if we took like Minimax out of the picture and we were just looking at those other models, for the most part, those are all like hosted models for… even though Flux has a history of open source and that sort of thing, it's not like they released the weights that we could actually use Flux 3 on our own machine at this point. I think- Yeah … that's maybe planned, but you couldn't really do that, right? And CDance obviously you're not gonna be running that on your own machine either. and and even Wan, they're, they've got a history of having, the Wan 2.1, Wan 2.2 as being open weights, but then 2.5 came out and it wasn't. I think what 3 is supposed to be at some point, but I, I don't really know. I don't know if anybody does. Um, but then-- And that's all really exciting. But then you have, Minimax drops H3 that is like next gen open weights model that can do anything. and so that's- Open weights, hostable … it, it, it
Alex Volkov
1:32:01
takes the
Blaine Brown
1:32:01
wind out of the, the sails,
Alex Volkov
1:32:02
I think a lot of times.
1:32:03
Fin tunable. Yeah, 100%. Sorry to interrupt, Blaine, but I'm just like open weights, hostable by yourself, fine tunable. we want to say hi to Victor Sortes from Minimax here. Welcome, dude. Awesome. Thank you so much for, jumping on as well. we love when the folks who are building the technologies, as you saw with Kir before joining, and we love when creators like Blaine are joining as well 'cause like I think combination Blaine can tell about the experience, Peter can tell you about like what folks are experiencing on the arena. Victor, you can tell us about what is so special and, how can folks run this in more performance. welcome Victor. Please tell us about Minimax like with one or two sentences. it's been a while since Oliver has been on the show, and that was in the LLM like research. This is like completely different part of Minimax, right?
Victor Su Ortiz
1:32:43
Yeah.
1:32:44
Yeah. First, thank you so much for having me. yeah, a bit about myself for people who don't know. Currently, I work on the GTM engineering and a bit of the developer relations side for, Minimax, so helping advocate for their products as well as their models. And, you know- This is exactly, what I'm here for. Really just an opportunity to talk to people such as yourselves and the audience that we have today about, what makes our model so special. And H3 really came in with quite a splash. It honestly was, trending on Twitter for a number of days. I couldn't-- Typically, for model releases, I'm always seeing, other, models on my feed. But, H3 was the only one front and center. I'm sure a lot of the audience, might have experienced as well on Twitter recently. But what is H3 specifically? it's an om- It's the first ever open-weight state-of-the-art omni, omni video generation model with, a thirty-three billion parameter open weight transformer architecture that can run, specifically with the recommendations, especially on Hunyface or with SG Lang- and, especially on NVIDIA GPUs. And as you currently have on the screen right now, we did just receive the results from Design Arena and state of the art, not only in open weight, but just in the video generation front here. particularly, I wanna point out the video editing because- … because of our, omni context representation with all the different modalities, as well as understanding, like, how each modality, references one another. The video editing capabilities itself of H3 is, top, top of the line. I'd, I'd say the best right now. Yeah.
Alex Volkov
1:34:32
Blaine, any chance I can, tap you to show us a
1:34:35
few of your favorite examples? 'Cause usually when you come here, you have, a stack of things that, blew up here or there, et cetera, with Minimax, H3. If, if you have some. If not, like- Yeah,
Blaine Brown
1:34:43
let me track, let me track some down.
Alex Volkov
1:34:45
Yeah.
Blaine Brown
1:34:45
think there's, there's definitely some crazy examples.
1:34:48
I think all of Twitter or all of X was lit up with people remaking episodes of "The Office" for some reason.
Alex Volkov
1:34:54
Yes.
1:34:55
I, I, I do wanna talk about this, Victor. And obviously like this, this is, has been, maybe one of the reasons why, why See, See Dance blew up early on, a, a while ago because people were just like recreating Hollywood. And then there was the whole thing with not releasing in the US w-with the restriction of hey, copyrighted material, et cetera. I don't wanna put you on the hot seat, but why do you think most of the folks are trying kinda like things that we saw already and maybe a part of the model training versus like novel and like beautiful video, generation? What, what, what, what, what is it that draws people to, show that, those examples and, and, and, in the open source? And how does it affect the open source strategy as well? Would love to hear.
Victor Su Ortiz
1:35:32
Yeah, yeah.
1:35:33
personally, I would imagine just the shock value for… That Twitter is always a platform for engagement, so people wanna see something really novel. Something that they haven't really seen yet, and IP related things, especially with with models are heavily restricted, on the Minimax and they are as well. I think, the issue, thanks for bringing that up. Can we use it in Amer- in America? Yes, you can.
Alex Volkov
1:36:01
Yeah.
Victor Su Ortiz
1:36:01
The only thing that you have to do, and it takes you a couple
1:36:04
of minutes, you can visit an application form that you can find on Hugging Face. There's also an email you can reach out to, to serve it locally on your machine. And the only reason for that is because of these IP issues, the current ongoing li- ongoing discourse that you might know with Minimax and one of the major corporations at the moment, which is why s- some of these examples are hindering free use. That's the only reason we do have that license. But once again, you should not feel deterred about it. Our team is constantly monitoring those inboxes, and you will get approved very quickly- Nice if you are in one of the reas- regions with restrictions.
Alex Volkov
1:36:41
So I, I, the, I do wanna talk about how to run this locally.
1:36:45
And Blaine, I think, you have a project that, works as well. Would, would you give us, a few m- words about Maestro as well? Something that- Sure … how do people get, the most of these models that are actually running locally? And what is the benefit of running this locally versus streaming it from, C Dream 2.5 Cap Cut things?
Blaine Brown
1:37:00
Yeah.
1:37:00
So I think the traditional method most people have used, and still do is y- using something like Comfy UI. That I think that's what I used for every model that's been released for the last four years. I-- When I built Maestro, it was, it was really a little bit of an answer to that a little bit. I wanted, you get used to the kind of the user interface that you get from like a, like a runway or a, or a pica or a, any of the models, Kling, all of them. And it's this simplified gallery and a simplified options and, and so I set out, a few months ago to build Maestro to be just this simple way to use open source tooling to the point where, those that have been doing open source video for a while know that there's, there's a lot of learning curve, not just with, with doing node-based type, things with Comfy, but whenever there's like LoRAs involved and, and i- if you know what the weight of the LoRA should be, and you might not really know un- until you do all this experimentation. And so the, the goal with Maestro was to kinda take the, the guesswork out of that because what it does is it lets you, even for if you're gonna download a, a, a LoRA, for instance, to, to add capabilities to a specific model like Wan or LTX or really even, H3 now, is it'll, it looks at the, the hosted site where the LoRA is and all of the, the guide information that recommends like step counts and, and weights and those sorts of things. The, the built-in agentic AI LLM that's in Maestro that orchestrates everything, reads all that, and it will, basically create a custom prompting guide for that particular model or for that particular, LoRA, right? And so-
Victor Su Ortiz
1:38:29
Yeah
1:38:30
… Blaine Brown: that way it, it kinda does that work for you. And then the other component it does is it has a, a director mode that lets you basically send a, a single prompt and it uses its own built-in LLM. Right now it's Gemma 4, that will write the screenplay, it'll write the prompts, it'll write the image prompts, the video prompts, and then it'll stitch it all together and actually edit the whole video for you, all for free, all locally on your own machine.
Alex Volkov
1:38:53
Locally.
1:38:53
Yeah. I found it incredible. shout out for working on, on open source. That is, very important. I wanna clarify for folks who are listening for the first time, and since we had you, there's, a bunch of folks. We used to be very, focused on video models. you mentioned a few concepts there. LoRA is standing for Low-Rank Adaptation, is a way to customize the model. Victor would love to hear from you, w- how you think about when you guys release open source, whether the community will pick up some of the stuff, right? IP and different things or different, restrictions, non-restrictions. CivitAI is, I think, the biggest library of L- LoRA adaptation or LoRAs for different models, image models and video models. Maestro allows you to, load them in addition to, to the video, trained on styles, trained on different, some people, that's why they like open models, 'cause they train their own LoRAs on copyrighted material so that the company doesn't have to, right? Stuff like that.
Victor Su Ortiz
1:39:37
Yeah, yeah.
1:39:38
I-- that was definitely one of the main, incentives also of why we decided OpenWeight, our thirty-three billion, transformer portion of our video models architecture. honestly, the, the community just put it bluntly, like insane. In less than twenty-two hour-- forty-eight hours or two days, you guys got like LoRA support. there's support off across all different sorts of hardware, including Ap-Apple Silicon, which we did not even r- optimize for at all, but the community did it. They- Yeah.
Alex Volkov
1:40:06
The MLX folks are crazy.
1:40:07
Yeah, a hundred percent.
Victor Su Ortiz
1:40:08
Yeah.
1:40:09
also quantized version from Comfy UI, all the other different, different quantizations that you're able to get from the community to run it on different sorts of hardware. And that was, that was why we wanted to release the model, because without the community, we wouldn't get all these different ty- all these optimizations, all these different, different ways that we can see with m- running our model specifically. And honestly, it's been really incredible and really matches and is exact… And why we do it is because intelligence with everyone, what we wanna see is the capab- having these, these frontier capabilities in everyone's hands for them to create as much as they can. And going back to OpenWeight, I do wanna also point out, one thing about hosting it locally. We have two parts of our ac- architecture if you're used to, if you're using our, cloud-hosted API that you can access. They're both the context IR API as well as the regenerate I- regenerate API. And What's useful about these is for the context IR is it uses different models to, optimize all of your different references and all of your different modalities specifically for your local, M-H3, mo- H3 instance. And also the regenerate is so you can take that six-- seven hundred and sixty-eight quality video and be-- and generate into a 2K quality also using… It's not even just an upscaler. It actually is another, it's another model in the back end that uses all the references to make sure the 2K quality is as best as it can be. So I highly suggest everyone who's deploying it locally at the moment to check out these APIs, which are all available on our website.
Alex Volkov
1:41:45
Thanks, Victor.
1:41:46
and if you send me those links, we'll add them to the show notes. Blain, I see you smiling once I presented this, this video. I think this is from Justine Moore, I believe. Yes, Justine Moore from A16z. She's incredible at testing these models. if you guys, remember the Will Smith spaghetti video model benchmark that's been completely obsoleted by anyone in the industry in 2026. this is Will Smith acting as, was this from Blade Runner, right? it's Will Smith's face instead of Ana de Armas inside Blade Runner, and he's talking to this spaghetti, like, person. Th-this like they took this benchmark to an insanely insane level. Blain, one maybe one last question for you. We're running to the next interview. character consistency is one of the big issues with these models usually, right? these models now can generate and, and Minimax really handles this beautifully. what are your current, like Tips for character consistency. Is that because they're omni reference and you can add a bunch of images? that's,
Blaine Brown
1:42:38
that's a, that's a huge, positive that came
1:42:40
out of the, the H3 models. They're, they have their, their start and end frame version, but then, the open weights, they also have the, the om- the, the omni reference version, which allows you just to load it with, some character images, character voices. You can drive it with actual, if you upload audio, you can stipulate that it's a voice or it's music to drive the video. you can actually upload videos and have it edit that video. You mentioned, the ability to edit it. It's so flexible compared to a lot of the previous open weight models that were out. like my previous go-to, I think up until now, most of the functionality of Maestro was built around LTX, right? 'Cause the LTX 2.3 was, was really good, and you could make music videos with it, but it would lose some of the character consistency unless you really tried using like LoRAs and stuff to dial that back in, but it was very complicated. this feels very much like, when I'm using, H3 on my own machine, it reminds me of when SORA2 first launched, We were-- And, and Alex, we were doing this for fun, right? We were making our characters, Yeah. You would record yourself, and you would have yourself doing all these stupid sorts of things. But I feel like that's now back, right? Like you can, and I'm thinking about adding a feature in Maestro that lets you just easily capture yourself and send in as omni reference-
Yam Peleg
1:43:46
Ooh,
Blaine Brown
1:43:46
yeah and save a character that way.
1:43:47
Because right now I'm, I've just got like an image and a, a voice reference for myself. But it is, it's as good as anything I've seen, repro-- you know, recreating myself in any sort of environment. That's the fun thing is there is IP in there, right? I can put myself in a Avengers movie or whatever, like-
Alex Volkov
1:44:01
Yeah
Blaine Brown
1:44:02
so it's, it's just really fun.
1:44:03
It's a great model.
Alex Volkov
1:44:04
So I'll avoid posting the, the IP stuff because we are on YouTube.
1:44:07
We don't get taken down. but we have Peter here from Arena. We want to, we want to show that for image to video overall, Minimax H3 is tied. Peter, can you talk about like the Minimax H3 a little bit and like from Arena perspective? This is an open model that can run fully on your computer that's tied to the previous like highlight.
Peter Gostev
1:44:24
And I would say the main takeaway for me in just seeing
1:44:28
all the outputs, playing with the model and, where it ranks on the leaderboard is that if, if you scroll down, it doesn't even fit in here. But if I look at our, like the website, that Veo 3, that was like the best open source, or closed source, video model, it's ranked 26 now. So and now this is number two, or shared one, yeah,
Alex Volkov
1:44:55
yeah, basically first place.
1:44:55
I think it's
Peter Gostev
1:44:55
in the image video.
1:44:55
yeah, yeah, it's a video tab on top. and, and what th- this just shows like the amount of improvement there's been. And you remember reaction to Veo 3, right? This is like, my gosh, amazing, like video is solved. And then since then we had so much improvement, which is just completely unbelievable. So yeah, amazing. Just the fact that, I, I feel like the progress has been massively underrated 'cause, it's-- the, the changes we've seen in the fidelity and the quality and the sound is just unbelievable. So yeah, there's just, yeah, well done,
Alex Volkov
1:45:29
Google.
1:45:29
I wanna say like it's one of those things, if you look at the, Will Smith s- eating spaghetti, like videos when it's like all wonky, whatever, it's like very clear that this is bullshit. I think all of these models jump to a capability kind of like the Mythos Fable stuff, that now is like super, super realistic and omni. what I think we're all expecting is like length. These models are 15 to maximum 30 seconds. We want like minutes. Victor, Sue Ortiz from Minimax, Blaine Brown from Blizzaine, at Blizzaine. Thank you so much for joining. Folks, please do check out Blaine's work and Maestro, and check out Minimax H3, its open weights. We really, really appreciate this. Victor, we have a thing on the show. When open weights are released, we have a sound here. Thank you guys for open weighting the models. We expect more. We really appreciate it. Thank you guys for joining. And very quickly, we're jumping to the next segment. Although, although we're over like two hours in the show, I wanna welcome David Krusha to the stage.
David Crawshaw
1:46:20
Thanks, Alex, appreciate it.
1:46:21
It's great to be here. Yeah. yeah, I am co-founder of exe.dev, and before that, co-founder of Tailscale. exe.dev is a, a new cloud designed for all of the new small software we're building. and it's designed for, the, the people driving it and for their agents.
Alex Volkov
1:46:38
maybe could I ask you, one sentence on Tailscale, one, two
1:46:41
sentences, how that's been co-founding that, and then we talk about exe.dev, for folks who have no idea what Tailscale is.
David Crawshaw
1:46:48
Yeah.
1:46:48
the, I'm the co-founder and CTO of Tailscale. We started it back in 2019. It is, a new V- VPN. It's an overlay network. You can connect any of your devices together, so they get private IP addresses they can use to communicate. That's, that's, that's the whole product. but you'd be amazed what you can do with that. it turns out, it's really good being able to connect your computers together
Alex Volkov
1:47:08
and exed.dev,
1:47:09
please tell us why did you choose to co-found another company? Is that like…
David Crawshaw
1:47:14
Oh, why start another company?
1:47:15
Tell us. Yeah. That's one of those questions that, I don't gen- I genuinely don't have an answer to. Yeah. I, I, I make up something slightly different every time someone asks me. it's, it's just the thing, it's the thing I wanna be doing. It's the thing that makes me feel useful.
Alex Volkov
1:47:28
Yeah.
David Crawshaw
1:47:28
h- what else would I be doing?
1:47:30
It's, there's, there's a lot of problems to be solved, and, companies are a good mechanism to solve them.
Alex Volkov
1:47:35
So, so tell us about exed.dev.
1:47:37
You, you founded this company. I f- find it incredible, and I use it, but tell us how do you like treat that? Like w- how do you position it? What is it? And how do people use it?
David Crawshaw
1:47:44
Yeah.
1:47:44
It's fundamentally, a new cloud, and it's a cloud very much designed for, putting your, putting your agents to work. And an agent can, in theory, figure anything out. But, you also don't wanna give your agent your AWS API key where you keep all your production infrastructure, and you have this challenge of needing to, somehow, Is that, is that sound on my end or your end? It must be on mine. Don't worry. I'll let, I'll let the software take care of it. It's
Alex Volkov
1:48:07
all good,
David Crawshaw
1:48:07
yeah.
1:48:07
That's, but yeah, you've got this, you've got this problem of you need a lot of VMs because you need to isolate a lot of, a lot of software together. you, you need to isolate the work each of these agents do. And, the, the previous clouds, the existing clouds that we had before at exe.dev, they all charged you some amount of money per VM, and so VMs are these scarce resources. You tried to figure out how to run lots of programs at once on each VM. it was a, it was a unpleasant thing. instead you just wanna start a lot of them, and so we, we built a, a platform for that, and then we realized there are a lot of other things agents need that clouds, clouds don't do, and so we've been building to that since then.
Alex Volkov
1:48:40
Folks, when you ssh.exe.dev,
1:48:43
or you just go to the website and lo-log in, there's a button called Shelly, which is basically you guys at Harness would love to hear from you, like how that works. You basically tell it, "I want this," and then you sit there and, and then at the end of the process, whatever it is, is installed. That's how I, the supercomputer whiz, installed OpenClaw for all my friends. I opened them a VM in the exe.exe.dev, entered into Shelly and said, "Install OpenClaw." And that's it. That's all I did. And it just provisions all of it. Kuros, tell us more about Shelly and your, like excitement about that, and then we, we'll, we'll ask Wilfred and, Wolfram and, and Nisten, let you ask questions as well. Tell us about Shelly.
David Crawshaw
1:49:17
Yeah, Shelly, like you said, it's an agent.
1:49:19
It's a… I, I think agent harness is, the term about these days. Yeah … and at, at the heart of it, it's really, you, it's-- you can use it to write software just like you can use Codex or Claude. And when we started exe.dev, it wasn't, it wasn't the first thing we wanted-- It wasn't the big problem we wanted to solve because we assumed most people would sign up and use Codex or Claude. The real challenge was you needed a box to put those things in, and that's, that's what we set out to do. But, as part of doing that ourselves, we would run into the situation where it's oh, we build a little web app for, exe.dev on, on top of the SSH API. it's really easy to start a VM on my phone. Oh, wow, it'd be great if I could actually like, do more from my phone. why can't I get this… why can't I install OpenClaw on the VM from my phone? And we, we just ran right into the limit of wow, actually the terminal is not a great user interface for agents for a lot of problems. Obviously it's, I have three terminals open on my window. I use them a lot. there's nothing wrong with them, but it's just not the solution to every problem. And we said, we really want an agent in our browser as well." And there really wasn't an obvious choice at the time, so we built one And it was very much that whole, like, how do we let you get started and write interesting software on your phone? And since then, I've written a lot of fun apps walking down the street on my phone using it. That's, it's, it's good.
Alex Volkov
1:50:32
Is this your custom harness?
1:50:34
Did you use open source for Shelly? Could you give us more details if you are, like-
David Crawshaw
1:50:38
Yeah
1:50:38
… Alex Volkov: you did post something about open source and, and, and tools must be open source this week. So I would love to hear, about, Shelly specifically, and then Wolfram, go ahead and ask a question.
David Crawshaw
1:50:45
Yeah.
1:50:46
We wrote, we wrote Shelly. It's, it's not built on top of existing harnesses. It makes the API calls directly. we did that because at the time, at the time everyone was using Anthropic models for everything, which meant everyone was using Claude for everything, and it wasn't very easy to build on top of. It was easier to use the APIs directly. and, it's an entirely open source program, so there's a, there's a GitHub repo for Shelly. You can go grab it and use it. It's all, I think it's Apache licensed, There's, there's no-- It's not secret source. and, it was, we decided to open source it just because-- And this was back in, February, I think. We decided to open source it just because inside your VM… There it is. Yeah, you found it. inside your VM, we think you should own the software in there and, we should put as little machinery into the VM as possible and instead run the machinery outside it. But we really wanted to ship you a web agent. and so we're like, "Okay, we'll make it open source, and we'll put it in there, and that way you know exactly what you're getting into." and that's, that's why it's, that's why it was originally open source. Since then, we've come to realize that open source is so much more important than that for dev tools, and that's what I wrote a blog post on, on the weekend, which I, I think is why, why we're chatting now.
Alex Volkov
1:51:52
Yeah.
David Crawshaw
1:51:52
and this was really a claim, and it's a claim about the change
1:51:55
in the world we're in right now, about how dev tools have to be open source now. I'm happy to go into that, but you say you have a question?
Nisten
1:52:01
Yeah.
1:52:02
I see that it, it is written in Go, so it's probably done by, actual DevOps people. I, I wanted to ask How do you handle, like software requirements and other auditing requirements from customers that, that are, that are gonna be using this? And, did you see any, any difficulties or, how you handle potential attack vectors from having an agent that does, you can talk to and, and manage all of your VMs?
David Crawshaw
1:52:30
Yeah.
1:52:30
So SOC 2… We're working through our SOC 2, right now. it's a, it's a process. I spend a lot of time talking to auditors. It is a change in how, SOC 2 used to generally re- auditors used to generally require human code reviews of human written code. it's a negotiation process to try and change that. Obviously, it's necessary because we're not gonna have humans reviewing human written code anymore. the fundamental claim I've made to SOC 2 auditors, and again, I'm still working through this with them, is agents are writing code and then a human is reviewing it. And yes, it's usually the human who actually, prompted the agent in the first place, but that's okay. Like there's, there's, there's still a fundamental separation of concerns there. There's still a human who looks at it before it goes into production, who's different from, how it was written. That-- So the, the fundamental value is still there. And, I'm hoping that, I'm, I'm hoping that becomes a generally accepted industry norm because the traditional code review is not something that works anymore. Like it's, it's not, it's not practical. It, it slows you down too much to your second question, you asked about like the structure of how like we arrange Shelley and XD Dev and all the rest. the default setup on XD Dev is when you make a VM, it's isolated from all of your other ones. And so there's no way for it to reach outside the VM and affect your account. And so when Shelley is working, the most it can do is suggest you go to XD Dev and change an integration setting to give it access to a GitHub repo or to give it access to something. And we've actually, we've been working on that. It can now send-- it can now produce XD Dev URLs for access changes, and it takes you-- You leave your VM, you go to XD Dev, and it presents you basically a box saying, "Do you really wanna run this command?" and so there is effectively a prompt there. Within the VM though, Shelley has complete control. It, it has access to root and can do whatever it needs. And that's actually really important for writing software. You want to give your agent complete access to the machine you're running on because sometimes as part of trying to debug your program, your agent needs to run TCP dump, which needs sudo. Sometimes it needs to apt-get install a package you don't have. You just gotta let it do those things. If you constrain your agent and you don't give it these tools, you get worse outcomes. And give it as much power as you possibly can in an isolated universe is, is very much the, the philosophy. And each one of these is isolated from each other one Then we have-- You, you can make SSH keys for HTTPS API keys for, for XD Dev that are scoped, scoped to tags and things like that so that you can give, you can give an agent in a VM power to make more VMs, so it can, live in its own sub-account. but that's, that's isolated away from all of the other infrastructure you're running.
Nisten
1:55:04
Okay.
1:55:04
And last, and to talk to each other, I would assume they're, they're using Tailscale. if you want VMs or-
David Crawshaw
1:55:09
A lot of people do that.
Nisten
1:55:10
Yeah.
David Crawshaw
1:55:10
And we, we have some that do that.
1:55:12
We have, we, we like to install Tailscale in our XCVMs to connect them. we have a corporate tailnet just like everyone else. there are other options built in. There's a, there's a, there's a peer, integration. so there's an, the Integrations tab is a set of features you can add to your, your VMs. You can give them… It's a, it's effectively a secrets manager in that you can give them access to services. you can put your secrets into the integration manager and the u- the keys in the, in there never actually leak into the VM. It acts as an HTTP proxy. And, you can use that to let VMs talk to each other directly. and similarly, you can give, you could give one VM power over some VMs by creating one of these scoped keys. And so you have all of those options available, but by default it's all, it's all turned off because they're, you want them, you want them all in their own box.
Alex Volkov
1:55:56
Uh, David, I, I wanna continue here because I do wanna talk
1:55:59
to you about, Amit and the stuff that, like you said, the open source, Yep stuff needs to be free. I'm trying to show, give me just one second. I'm trying to show here, on the stage, a slide of my recent talk at Ai.engineer. Let me blow this up a little bit. and this is basically the routing table that me and honestly, very honestly, Claude, Fabel, I think back then, came up with and that we talk about. there is this thing that I coined called the Zeal Continuum, whether or not developers even read code, and basically it lands somewhere in the middle. Could you talk, talk to me generally about, the PR process and also Meet as your like, solution to that? And, and that's one example of your blog post, which I wanna like- Yeah … hear more from.
David Crawshaw
1:56:34
I think you're, you're completely right, or you're absolutely
1:56:37
right, I should say, like Claude. Yeah,
Alex Volkov
1:56:38
absolutely.
David Crawshaw
1:56:38
that, you…
1:56:39
no one's writing code anymore. that's, we, we do it through agents now. and I'm, I'm sure as I say that, there's a million developers out there writing code still, because change takes a long time, but this is-- that's clearly the future. No one writes code anymore. And some code is read, and like this is true of my day to day, right? Whenever I make a change to, the XCDEV core infrastructure, I read the code before I, before it goes into production. and that's, I think that is still correct for me to do that. Mm. it's, it's very rare I'll actually catch a bug or anything doing that. it's mostly about like algorithm choices, architectural decisions. That's what I'm reading for.
Alex Volkov
1:57:14
Mm.
David Crawshaw
1:57:14
and so some fraction of code is read.
1:57:16
Obviously, if I'm building small programs on XCDEV, I don't read the code at all. I just, I just build it and i- if the program works, I'm good.
Alex Volkov
1:57:23
Yeah.
David Crawshaw
1:57:23
but for those programs I read, what I've noticed
1:57:26
is over the last few months, what I am looking for in the code has changed a lot as the models improve. And and a, a lot has changed since the days of me reviewing code for humans. I've been reading, people's PRs or change equivalents. A lot of them weren't called PRs, because, Perforce doesn't work on GitHub. those, those changes, the, the, the classic thing you look for is style. You're trying to match style. You're looking for obvious, error handling mistakes, the sorts of things that… The mistakes that are just so easy for us as individuals to make. You as a reviewer, you help the person you're
Alex Volkov
1:57:57
reviewing- 'Cause we're so bad at syntax and remembering and con-
1:57:58
and con- continuous things as humans. We just meet, meet people. Yes.
David Crawshaw
1:58:02
Yeah.
1:58:03
That's right. And what I've realized in the last few months is the code I'm reviewing from agents, they're far more diligent than humans are at these details. They make big obvious mistakes still, They're, they're still fallible. We're not, we're not dealing with, something perfect yet, which is why we read the code, especially around architecture. They make all sorts of silly decisions. but, so do I. I, I, I sympathize with them. but there's all these details of the code that just don't matter to me anymore. I don't need to make sure that you're nil checking records, fields in a record anymore, because agents are really good at that. The la- I can't remember the last time I saw a nil panic in production, in our logs. And, in the world before agents, that was a thing that if you're a Go programmer, you would run into every couple of months, some, some server with nil panic. That's just not a thing anymore. and it's not a thing because we got much better at code review. It's a thing because agents are much more diligent about these details. But if agents are really diligent about these details, then I don't need to read for them, and they're eating up some of the time I have for reading code every day. And that's really, that's really important to me because the current limit on my ability to ship code is how much code can I read in a day I can prompt more things and get more commits ready to push than I can actually read and be comfortable pushing in a day. And it's a real problem when I wake up in the morning and I realize I have a dozen commits from yesterday to read through, because it's a terrible way to start the day and I don't recommend it. and it occurred to me that I now know the shape of huge amounts of code I don't need to read anymore. Oh, and I have something that's really good at manipulating code, which is agents. So let's ask an agent to rip out all the pieces of code I don't need to look at. take the diff that I would typically read and remove all the nil checks, and remove all the error condition handling, from the code because I, I can ass- you know, show me that error conditional handling exists and I'm done. I don't need to read the error message and say, is the appropriate information in it? I d- don't need to know that, you check the, the field on the struct is not nil. and I don't need to read all the boring imports and the changes to the imports. There's just enormous amounts of detail about code that don't matter. if you added three parameters to a whole series of function calls, I don't need to read them 15 times. You can just put some dots in there, it's fine. And so, what Meet does is it uses multiple passes through an LLM to take, an existing diff, existing commit, and and make it as small as possible, for human review. And the, the final diff doesn't fully compile, even though, there's a lot of, semantic requirements on the change. So how it actually works internally, is actually pretty involved. it actually asks LLMs to produce edit plans and then edits, the, the code rather than, simply asking it to ed- produce a diff. the first version did that. I just asked, I handed it a diff and say, "Reprint this diff without the, important, without the unimportant things." and it worked really well most of the time, and then sometimes it would just make stuff up in the middle of, in the middle of a commit- Lovely … and there'd be some fantastical code in there. which is a- is, is nice and scary. it would fix things. That was the worst thing it did, is if there was an actual bug in the code I was looking at- Oh,
Victor Su Ortiz
2:00:59
yeah.
David Crawshaw
2:00:59
Yeah … it would fix it in the process of,
2:01:01
correcting the diff to show to me. Yeah. and then like that, obviously I can't code review that. So what it actually does is it generates an edit plan for the underlying code, and the, the model says, "Remove these four lines, change the, extract this chunk and reduce it to an ellipses," or something like that.
LDJ
2:01:15
Mm.
David Crawshaw
2:01:15
And so Meet does that, and it processes your
2:01:17
commit and gives you that result. And, I like it. It's, it's fun. It lets me read a bit more code in the day.
Alex Volkov
2:01:23
David, last question I ask for you.
2:01:24
I, I think I'll ask this many, many folks for, agentic minded. Are you, yourself in the token billionaire club? Are you using, billions of tokens at this point? Where- Ooh, that's a good question And have you been starting using significantly more since the Fable Mythos soul level capabilities jumped?
David Crawshaw
2:01:43
Yeah, those are good questions.
2:01:45
there have definitely been times in the last few months when I have been in that state. I'm not sure I am right now. one of the things about once your program exists is, the cha- what does ExeDev need right now? Mm. It needs, it needs to help people understand what you can do with it. And so a lot of my work is extremely targeted and is more product- Surgical … than engineering.
Victor Su Ortiz
2:02:05
Yeah.
David Crawshaw
2:02:05
And, that involves me staring at it for hours and then making
2:02:07
a small change, and I don't actually need a lot of tokens to do that.
Alex Volkov
2:02:10
Yeah.
David Crawshaw
2:02:10
I, I probably b- burn a lot of tokens in my data analysis.
2:02:13
I could go look that up. I don't, I don't track, honestly. I do have some, I, I guess loop engineering style things that use a bunch of tokens, and I only realized, yesterday that I had like a, a deflake thing that was using like 100 bucks every night worth of Fable tokens. Wow. I moved it over to Sol. It's a lot cheaper. It works just as well. It's fine. That's, so you know, it sneaks up on you. you can spend a lot of money with- without even realizing it.
Alex Volkov
2:02:35
Yeah.
David Crawshaw
2:02:35
but yeah.
2:02:36
I think right now I'm, I'm being very targeted in my work. but I'm very, I'm very happy to spend on tokens, right? I'm not, I'm not gonna try to optimize any of that. I'm, I'm optimizing for productivity.
Alex Volkov
2:02:45
Yeah.
2:02:46
So David Crawshaw from ExeDev, thank you so much for joining and telling us behind the scenes. I will send folks to your DevTools must be open source essay, which I completely agree with, in personalized software. It sounds like we truly believe here. I c- completely agree with you. That's the future. And so what- It is absolutely the future, and, you guys are heralding as well. Like folks, please follow David on, on socials. We agree with his positions, and thank you so much for exe.dev. And with that, I think we're at the end of the show. Thank you so much Wolfram, Nisten, Peter joined us, LDJ and Jan were here before. If you missed any part of the show, go to ThursdAI.news. We'll see you here again next week. Bye-bye, everyone.