---
title: "S02 E02 - Nobody Reads Anymore: How to Trust What You Ship"
shortTitle: "Nobody Reads Anymore: How to Trust What You Ship"
season: 2
episode: 2
slug: "nobody-reads-anymore-how-to-trust-what-you-ship"
publishedAt: "2026-11-20T14:00:00.000Z"
duration: "01:01:38"
canonicalUrl: "https://bloodsweatandtokens.xyz/episodes/nobody-reads-anymore-how-to-trust-what-you-ship/"
audioUrl: "https://api.riverside.com/hosting-analytics/media/8ac8e793410040f2c550a09af5aa750fd1ab03caedfd28aa0e29270dce8e8ea1/eyJlcGlzb2RlSWQiOiI3NjNjNDA0My0xMGYxLTQ4NzAtYjg3OS0yOWEzNzEwMTAyMDMiLCJwb2RjYXN0SWQiOiI3ZmE0NDhkNi0xMzhiLTRkOTktOTFhMi1lYzBkNGExMWI2MDYiLCJhY2NvdW50SWQiOiI2OWE1ZmJhOTRkZmVkYWY3NDViZTk1YTMiLCJwYXRoIjoibWVkaWEvY2xpcHMvNmFmZjM4MTAzYTFlY2UxNTQ1ZWViZDJkL3RheWxvci1tYWNkb25hbGRzLXN0dWRpby00VE9qRS0yMDI2LTExLTE5X18xNy01NS0yOC5tcDMifQ==.mp3"
youtubeUrl: "https://www.youtube.com/watch?v=rwYYdvokRP8"
---

# Nobody Reads Anymore: How to Trust What You Ship

## Summary

Taylor and Sean take on the defining question of the AI-assisted era: how do you ship work you didn't write?

## Show Notes

Taylor and Sean take on the defining question of the AI-assisted era: how do you ship work you didn't write?

Coming off Netlify's near-full app rebuild in four weeks, Sean walks through the multi-agent review workflow that makes that survivable — the agent that builds never reviews its own work, different providers cross-check each other's findings, and every issue gets graded must/should/could so humans triage instead of drowning. The punchline: you don't need to read code, you need to know what a wrong answer looks like in your field — which makes the same process work for lawyers, financial planners, and compliance staff.

Around that: a week where the internet kept breaking (tech debt, not a conspiracy), Jensen Huang's ouster one hour after last week's recording, Anthropic's record IPO, and a challenge — hand a personal AI agent a real credit card and see what it buys.

****This week:​****
• The Jensen payoff — Colette Kress interim CEO, export-ban questions, earnings day
• Outage cascade: Microsoft 365, the FAA ground stop, Apple, the NSA — tech debt comes due
• Anthropic's biggest-ever IPO: $83/share, 50.1% founder lock, $900B compute commitments
• GPT-6.2 Astra & Sol + the Agents Last Exam leaderboard
• Local vs. frontier models — OpenRouter, whisper.cpp, and client data-retention compliance
• Main segment: Sean's five-step review process, made "mom-proof"
• Vibe-coded wins: a Hong Kong funds lawyer, Nursa's non-engineers, Anthropic's own staff
• Persistent personal agents: Muse (and its Reuters mess), Grokbot, Claude Duet, ChatGPT Dots
• Could AI agents get groupthink? A speculative experiment worth running

Next week's challenge: one agent, one virtual credit card, real consequences

## Transcript

Taylor MacDonald: Hey, welcome back everybody. Blood Sweat and Tokens podcast season two, episode two coming at you. my name is Taylor McDonald, and with me as always is the venerable Sean Davis. Say hi, Sean.

Sean C Davis: What's up, Taylor? Glad to be back.

Taylor MacDonald: How are you doing today? What's going on in your world?

Sean C Davis: Well, we are one day post big launch at Netlify, which has been I mean that's that that's kind of a wild story where we've rebuilt a good portion of the app, almost almost rebuilt the whole experience in about four weeks, which we could have never done a

Taylor MacDonald: my goodness.

Sean C Davis: year ago. but is also tiring.

Taylor MacDonald: So you vibe you vibe coded Netlify dot app?

Sean C Davis: Well, we I what what I will say is we couldn't have done it without AI agents. But I think this is kind of a the it's an interesting topic because what we talked about as we ended last week was this this question of how do you build confidence in yourself to or how do yeah, how do you build the confidence in yourself and your team To ship something, anything to production in front of your customers and all of that when you haven't read every single line of code or every little detail. It doesn't need to be code. And

Taylor MacDonald: Right.

Sean C Davis: and so we'll we'll come back to that because I think it's, you know, it's vibe coded, I think gives the wrong it it's it's got this connotation that comes with it,

Taylor MacDonald: Yeah.

Sean C Davis: which is like a half-asset. And I think it's It's wrong in this case where it's like, well, okay, the work wasn't half assed. In fact it was a it's it's it's solid. we're now we're about twenty four hours out and have been monitoring it and it's it's gone extremely smoothly. Yeah.

Taylor MacDonald: Been pretty good. That's amazing. Yeah, we need a new adjective, don't we? It's not vibe. It's not like the vibes anymore. Now it's like, I don't know. It's it's AI assisted to a degree, but so let me ask you this. Were you guys were you using any of your internal products like agent runners or anything like that to build certain portions or certain aspects, maybe the marketing website, stuff?

Sean C Davis: Yes, we have so there there's a there's a couple of things that I've I've found really interesting lately. One thing is that as we do more sort of like show and tell internally, and I even talked to other external developer developers who aren't working at Nell Fi what I mean, that everyone sort of has their own bespoke setup, which I think is interesting. And I mean it's the way it's always been for engineers. You wanna you you have your own tastes and your own workstation. So I think there's a there's an element of that, but in terms of using Netlify to build Netlify, absolutely. In fact, we've done something that I that that is pretty cool. That is called we c it it's an internal tool that we call NetLaloop. And it's basically a we don't have to go too deep into the details, but it is a Netlify applic. It's an application that is hosted on Netlify that is a Slack agent. So it is basically like you can work through Slack and say, Here, go do this task, which hits a Netlify project, which then uses Netlify agent runners to basically go get the original task, build it, and send you it back when it's ready for review.

Taylor MacDonald: That's cool. Yeah, that's like netliception, man. That's crazy.

Sean C Davis: It is wild.

Taylor MacDonald: well, that's great. That's a wonderful segue into another topic that I want to talk about a little bit later, which is pers persistent personal agents. Before we get into that, there has been a lot of stuff happening in the news this week.

Sean C Davis: Yeah.

Taylor MacDonald: in fact, so much so that I want to go through this kind of quickly so we don't burn the entire you know amount of time we have here today. But One thing I thought was hilarious. So we spent a fair amount of time last week talking about what was going on with NVIDIA. If you

Sean C Davis: Mm-hmm.

Taylor MacDonald: remember at the time that we recorder recorded, I think the board was still meeting or it was the early morning after the board had met and we were asking some questions of each other. Do you think that, you know, Jensen will be fired or be asked to resign or anything like that? I think we both assumed that he probably wasn't going anywhere because of this climate of, you know, like Corporate irresponsibility that we live in. Man, wouldn't you believe it? About 10 minutes after we stopped recording, I got the news that Jensen had indeed resigned. Well, that was a week ago. He since there's since been a third affair that came out, I think, with a second employee of his, you know, only adding to the misery. I don't know if that would I I'm pretty sure that happened after he actually resigned or right around there there and thereabouts. well now they're talking about bringing him back already. Which is insane to me. But, board's gotta make that cash. so so we talked about that. They've now got a new interim CEO. Her name's Colette Crest. She has been the CFO for years. Now I'm reading some speculation that maybe some of the relationships with these you know, South Asian companies, particularly in Singapore and stuff like that, maybe weren't above board, which is kind of an interesting choice to promote the CFO there. But we'll see how that goes. There's a lot of interesting stuff to still come out there.

Sean C Davis: Because isn't that didn't didn't the US sign something with a a few other countries too that like has some effect with I I didn't go too deep in it. I saw a headline. Did you did you catch this?

Taylor MacDonald: So there there is a law on the books that has banned the selling of these chips, you know, to these countries on a banned list or whatever. I right. And the idea

Sean C Davis: Right. Yes.

Taylor MacDonald: is is that we don't want, you know, our geopolitical adversaries to get access to some of this technology and and kind of compete with us in the space race, so to speak. I think the argument that I'm hearing is is that the Transfer of these chips to this particular company who the Chinese spy worked for was not in accordance with some of those regulations. And it was questionable about why

Sean C Davis: Hm.

Taylor MacDonald: that company was not had had not been specifically placed on this band list. And apparently that erupted in some sort of like tit for tat like argument between some of the administration officials, you know, about whose responsibility it was or what was going on. I'm not getting into any of that stuff, but it's it's definitely

Sean C Davis: Yeah.

Taylor MacDonald: a mess. And that was just the beginning of the week. So let's move forward. So following that, we had, and I'm sure everybody has talked about this in your world, but okay, so on November 10th and 11th, apparently GitHub went down. I didn't even know this. I'm just looking at this list that was generated for me. So we had a series of internet outages, apparently starting with GitHub. Then we had AWS Tokyo. Then on November 11th, you had Slack had some issues. November 12th and 13th. This is the big one, I think, that everybody took notice of, which was Microsoft 365, was offline for about 24 hours. This was like a Teams outage, co-pilot outage. I mean, it was a major disruption to enterprise. A day after that, apparently there were two separate outages for Anthropic and Claude that impacted the app, the API, and Claude Code. The FAA went offline on November 13th. That we'll talk about that here in a second, because that one's an interesting one. Workday had an issue in some of their customer environments on the 13th and the 14th. AWS in the eastern part of the country was down on the 15th. JP Morgan Chase and other banking systems went offline on the 16th. United Airlines had issues on the 16th. On the 17th, we got reports that federal systems like the NSA Classified's internal network, which is absolutely horrifying, was going offline. there was routine maintenance that disrupted websites for the National Weather Service on the 17th. Apple had an internal update that disrupted iCloud, Apple Pay, the App Store, Apple Music, and other services on the 17th. And then apparently Marriott.com had some issues too on the 17th. What is going on? Are we under attack? Is this the end of the world? Have you ever seen that movie Leave the World Behind? Which incidentally was co-produced.

Sean C Davis: No, I don't even know if I've heard of that.

Taylor MacDonald: bro, it's crazy. Okay, so it's called Leave the World Behind, and it is about this exact scenario. And what makes it extra alarming is I believe it was like co-produced or something by the Obamas, like post-presidency Barack Obama. Just like horrifying to watch an existential crisis movie that you know the guy who was in the room had a hand in. In any case, kind of a big deal, right? Did you guys experience any outages over at Nova Five?

Sean C Davis: Yes, I we're work got disrupted. Now we're not on the office. but we have customers on office and so it's you know i it's it's always an issue for us when customers have any sort of related issue. yeah, I

Taylor MacDonald: Didn't you guys just roll you just rolled out a product called Netlify Connectors, right? Which is

Sean C Davis: Mm-hmm.

Taylor MacDonald: I mean, did that cause the internet outages when you guys started connecting to everything? Was it is it your fault?

Sean C Davis: Well that j yeah, we we we broke it. We broke the internet. yeah, most of what you mentioned here happened before that. So it affected some of our testing, but it didn't affect

Taylor MacDonald: Okay.

Sean C Davis: customers using it. But you know what the one that oddly enough, the the one that hit me what it was Apple, which was I think Tuesday morning or or middle of the night Tuesday for us on the East Coast. And I was like, Okay, well if Apple is going down. I cannot remember the last time I I came across an Apple issue. And you mentioned the NSA as well. It's

Taylor MacDonald: You mean outside of just the cr general crappy history of web services in Apple? my God,

Sean C Davis: Yes, yeah, yeah, yeah.

Taylor MacDonald: Syria is the biggest disaster ever. Sorry, go ahead.

Sean C Davis: You're right, right. But the reliability it seems like it's there. And and I so I was asking myself the same questions. You know, is this just business as usual and now we're gonna have to things are gonna be offline occasionally?

Taylor MacDonald: Do you do you remember? I don't know. I will say that like all these events are quite it makes anybody who's susceptible to conspiracy theory quite natural logical conclusion. Like how can all of these things

Sean C Davis: Yeah.

Taylor MacDonald: be coincidental and all happened all at once? I will say that as of this morning, according to the New York Times who published a podcast on this, it is largely tech debt, which is sort of hilarious. As an engineer, you look back at all the crap. And all the garbage and all the times you've been like, hey, client, we should upgrade your web app. And they're like, nah, we'll do that next year. And it never happens. that is largely the argument that is being made here. Now that's separate than the FAA thing, which I'll talk about in a minute. But as far as all of these issues, remember a couple of weeks ago we saw these widespread outages or vulnerabilities discovered on PHP and Node.js, it impacted estimates are upwards of 70 to 80 percent of the internet. Websites online had to be patched quickly. Well, that was widely understood to be like an AI awareness. Like it is discovering vulnerabilities faster than we can patch them. And it turns out that all of those years of kicking the can down the road and incurring this technical debt without addressing it and paying it down is finally coming home to roost.

Sean C Davis: Mm-hmm.

Taylor MacDonald: Okay, so that makes sense. As an engineer, and what I'm saying in layman's speak is like, no, this is not a coordinated attack. No, the internet is not failing. This just happens to be a ridiculous calamity brought about by years and years of neglect on on upgrading and dealing with some of these old crappy systems. Now.

Sean C Davis: Yeah, in well go ahead, go ahead.

Taylor MacDonald: Well, I was just gonna say the FAA is a different one. You talked about taking the internet offline. Now this one is fascinating. Okay, so according to the FAA, they were installing a router as part of routine maintenance, which is ironic because that is actually paying down the technical debt. And that some router in some remote part of the country that they had replaced all of a sudden started flooding the network with pings or random, you know, requests made to other parts of the network. And that that created this level of instability that caused the air traffic controller system and and all the people in charge to be like, we're not sure what's going on. Let's hit the brakes. And they issued approximately four hour ground stop, which halted all air traffic in the United States for that period of time. So they kind of did take off the internet. It just happened be like, you know, the airline, the internet of the air.

Sean C Davis: The airline, yeah. yes. So all right. I wanna address three things. One of them is conspiracy theories, the other one is government, and the third one is priv like private enterprise reactions to that. So all right. Where do we start? Let's start with The so the conspiracy theories. All right. So I I totally agree with you. And one I I just want to share a quick anecdote. Cause I was out with some friends last weekend. It was this couple where the the w one one person was like, Okay, this crazy thing happened to me. And she was telling these stories about like the most random encounters and the like I saw this person who grew up a street away from me in rural Iowa at this conference in you know some other city and we live in two different cities. And and her husband is like, that's not it's not a coincidence because or it's a or it's he's

Taylor MacDonald: It it c it can't just be a coincidence. Or it is a coincidence.

Sean C Davis: yeah, he's like it's just a coincidence. Think about what

Taylor MacDonald: Okay.

Sean C Davis: he likes to say is think about. How many people you grew up with that you didn't see at that conference, right? And so I I'm like, it's it's an absurd argument. It's really funny, but to, you know, you just listed off this long list of companies, but there's also like millions of other companies that you

Taylor MacDonald: Yeah.

Sean C Davis: didn't list. And you didn't list any company twice. so I I think maybe it's like the reliability is going down and we have to deal with this tech debt. faster 'cause it's yeah, it's

Taylor MacDonald: Yeah.

Sean C Davis: coming around to bite everybody in the ass, but it's also I don't feel like it's doom and gloom. It might actually end up being a good thing. let's make these services better.

Taylor MacDonald: So that's the bottom line on the New York Times piece, like where they concluded it is kind of like, look, you know, maybe once we get through this period of instability. I mean, because really what's happening is you've got these agents that are identifying vulnerabilities we never even thought of, you know, and they're operating at such a rapid pace that like, you know, it kind of makes sense. And of course, everybody's rushing to fix this. So that's causing us to be a little more haphazard. You know, to your point earlier about like keeping a human in the loop around QA. And review. You know, I'm not sure everybody's doing I mean, I'm sure the big enterprises, the netlifies of the world, you guys are taking every due diligence and care possible to make sure there's no disruptions. But at the end of the day, these are extremely technical and complex problems to solve. So it kind of makes sense that things would go offline and that it's all sort of converging in this one moment. It just happens to be fairly coincidental and ridiculous that it all happened in one week. Let's keep going. There's more news to talk about. So as of I don't know, maybe like seven minutes from now, Anthropics IPO, the biggest ever, will start trading shares under the ANT flag

Sean C Davis: Mm.

Taylor MacDonald: or whatever on the on the markets. today they are priced at eighty three dollars a share, Sean. Are you are you in line to pick up your pick up your latest piece of the future?

Sean C Davis: I man. I pulled so I pulled up my brokerage account and I had it prepared and then I was like I I wanted to do it just for this conversation to to be like, let's see.

Taylor MacDonald: man, you should have had Muse do it for ya. Damn it, missed opportunity.

Sean C Davis: I well, that's a whole other thing. and then I was like, okay, so I should I do this just for the conversation? And then I started looking at the history of these like really big tech hype IPOs. And I was like, okay, well, I mean, you could, you know, you could throw away a hundred dollars or whatever today. Cause that's the interesting thing. It's posted at 83, it's expected to go. To one five by the time we can actually purchase it or so. But then you

Taylor MacDonald: Yeah, correct. And what what did I read that the they're retaining fifty point one percent, like the founders, so they can't get Jensened. You know? But basically

Sean C Davis: okay.

Taylor MacDonald: there's no there's there will be no public shareholder oversight of of this company because they're they're gonna maintain just enough to like control the votes.

Sean C Davis: Okay, okay. So i it i I looked at some of these other ones like Meta and and and it was a a list of the combined top ten IPOs of the last I th I think twelve months and almost all of them lose somewhere between thirty and fifty percent

Taylor MacDonald: Yeah.

Sean C Davis: value in the first three to six months. So I was like, I think long term this is probably a good investment. But maybe I'll wait two months.

Taylor MacDonald: it's probably a good investment. Yeah, I agree. But you might want to you might want to wait until that bubble pops a little bit. okay, so that's crazy. The biggest ever, I mean, some of the interesting on my understanding is some of the interesting nuances about the the what is it, an S3 filing or whatever, is that they have made these enormous commitments on compute. that they can't get out of. they have to buy it's like some $900 billion worth of compute over the next like

Sean C Davis: Yeah, yeah.

Taylor MacDonald: 20 years or something, which that's very interesting. and as the at the rate at which the stuff is accelerating, it is bananas that they are still casting like f or forecasting along the same economic lines and progress lines that they would 10 years ago. In fact, this is one of the things that I've struggled with a lot thinking about business, you know, We have so many processes and so many different things that we have developed over the years to deal with business as it existed in 2010, but in 2026 it's a different landscape. And so I don't even know how you forecast 20 years into the future at this point, but there's a lot of smarter people than me on this. So other news, OpenAI shipped a couple new models this week. There's GPT six point two, Astra and Sol. these apparently perform even better. Than their immediate predecessors of like 6.0, I guess, or or whatever the the prior Astro model was.

Sean C Davis: Mm-hmm.

Taylor MacDonald: they cost less to run, and apparently they are extremely good from a security standpoint, and so they've even triggered like safety classifications within within you know the the AI world. I did come across this really cool leaderboard thing, which I wanted to share. so Berkeley UC Berkeley, man. Me I'm gonna have to maybe it'll work. Let's see. Okay. Can you see my screen?

Sean C Davis: Yeah.

Taylor MacDonald: All right, cool. So this is kind of like a leaderboard of the latest models. And and this kind of graph here at the top is the most important one, but this kind of shows the pass rate, which is the share of runs that earned a perfect score, right? So they're testing each of these models in the same challenges or whatever to get a consistent baseline. And then they're casting them based on the success rate on that first pass, right? And so according to this, six point two Astra. has a 47.5% pass rate, which is the highest of all the frontier models here. And it's cool. You can look down like Chemi 3.1 Pro is right in line. Quinn 4 Max. these are openweight models that you can run for a fraction of the cost. So definitely something

Sean C Davis: Yeah. Mm-hmm.

Taylor MacDonald: to check out and you can kind of explore some of this stuff. This is agentslastexam.org, and there's hyphen. So agents last hyphen exam.org. And you can check out the leaderboard. some of the stuff there that they're using to to benchmark some of these models.

Sean C Davis: Are you still using those open source models?

Taylor MacDonald: Almost exclusiv well, not I wouldn't say exclusively. I'm using it for all the kind of mundane lower level tasks. So I like to use the frontier models to do my planning and orchid and and general kind of like scheduling tasks. And then I use all these open source models. let's see what I forget which one. I think I'm using the GAI or or GLM. just a second. But yeah, I am still using it. What what are you up to these days? Are you mostly in Claude or

Sean C Davis: I'm I am mostly between Claude and and Codex. pop over to Gemini for a few things. And then I will what I've what I've started experimenting with is di I using these other models but through through open router because one thing that's really nice of the one of the One of the biggest benefits to Netlify's AI gateway is that you can essentially try any model and you just pay Netlify credits. You don't have to have accounts with them. So I really like that for just being like, hey, I'm I'm gonna screw around and see what this model does compared to this other model. Sometimes I'll often go through Netlify just because it's easy for me, or throw some credits up directly on open router and test that

Taylor MacDonald: That's what I'm using as open router. Yeah, it works pretty well.

Sean C Davis: Do you so you 'cause you had downloaded somet you were running something on your Mac Mini at one point, right?

Taylor MacDonald: yeah, yeah. I'm still doing a lot of like the whisper stuff. I gave that presentation at Momentum Conf back in October, and all of that stuff is all happening locally. So I'm doing a lot of like audio transcription and and other things of that nature directly on my Mac Mini, which is

Sean C Davis: okay. But if it's actual like chat, you're gonna go through open router.

Taylor MacDonald: There's certain things Yeah, correct. Because running some of those bigger models are you know, they're slow. You know, there's a lot of like little routines and cron jobs and things like that that I've got running that are using those local models because time doesn't matter. You know, if it's just like if if it's running at three o'clock in the morning to do, you know, like organization cleanup or review of inbox, things like that, all that crap, I don't I don't pay for that. I just use the local hardware to do it because there's no expectation of time. So

Sean C Davis: Yeah, and okay, slightly off topic here, but just 'cause we're talking about it, I was thinking, something that we I think you touched on during that conference talk, but we haven't talked a ton about here, is as we're talking to to various subject matter experts and people who need AI for all of these different things, I know that one of your driving decisions was around security. Like if I'm gonna put this agent inside of a meeting, How do I go through disclosure? I know neither of us are lawyers, but I'm maybe you can tell folks a little bit about how you've thought through like where are you working when from that security privacy standpoint.

Taylor MacDonald: So I typically rely and so what you're talking about is the AI transcription for meetings, Zoom meetings or in person meetings. Is that that?

Sean C Davis: Yeah, but also just just chat. if you are I'm sharing, medical details with Claude a lot of times and I'm comfortable with that, but I don't know that everybody should be.

Taylor MacDonald: Yeah, no, it's true. And I think you gotta be real careful about that. So the way we do this at we have company policies, acceptable AI use policies that we've defined and codified for our employees. Here's what you can do, here are the services that you're allowed to work with. Make sure you're using company sponsored accounts, that kind of stuff. So we have governance and visibility controls into that. And part of the reason that you gotta do that is because you'll have clients that have their own policies that they're imposing on you.

Sean C Davis: Mm-hmm.

Taylor MacDonald: So you'll start working with a client and they're like, hey, when this engagement ceases or when you get fired or whatever, we want to purge all of we want you to purge all the you know information that you have on file for us, right? It's like a data retention thing. And they'll say within 90 days of terminating this contract, you need to purge all this stuff. Well, if you've got 12 different accounts across all these different service providers and I as the business owner can't see where those accounts exist and where data is being stored, I got a big problem, right? Because I then I can't confirm that yes, everything is gone. There is no information outside of that. Now, there's always going to be human error and mistakes that happen and stuff like that. And that's where you get into like the the MDM and EDR stuff. But it's basically like the big brothery type of running a business where when I give you a laptop, it's got a a profile installed on it that gives me access to your computer through a remote management interface, right? And all I I'm not

Sean C Davis: Mm-hmm.

Taylor MacDonald: like watching your screen or babysitting it, but I can see the security, you know, is the disk encrypted? Have you updated to the latest operating system? How full is your hard drive, which is interesting, right? Like if you have a really full hard drive, you'll get flagged on some of these security audits because you don't have enough room to download the security patch for the software when you need it. Right. And so That obviously is complicated when you're using a bunch of different services. So as far as the services we use, I I do transcriptions and most remote meetings through a service called Fathom, which is part of our acceptable AI use policy. And of course we've got, you know, business level accounts with all of these so we can ensure, you know, that their data retention, their privacy assurances and things like that are aligned with our client expectations.

Sean C Davis: Mm.

Taylor MacDonald: when I'm doing stuff, I mean, and frankly, that's the biggest benefit of doing things locally, right? So if you know, the conference talk that I gave in October s focused on sort of a fun little pet project, which was to build an audio transcriber, like a physical piece of hardware that you click a button, it'll record a conversation, and when it's done, it syncs that content over to my Mac Mini and does the transcription on the Mac Mini. The benefit there is like that doesn't touch any third party services at all. Like it I'm using a a VPN to keep it, controlled inside my own network so it never hits the public interface. you know, it doesn't touch any third party API. I'm using the Whisper CPP library here on my Mac Mini to do the the local transcription. So it doesn't even leave the box once it gets on there. and that definitely helps me better understand and control and conform to whatever the next requirements are. for that next contract or that next client adjustment and the way they think about it. Does that answer your question?

Sean C Davis: Yeah, yeah, absolutely. And probably

Taylor MacDonald: Yeah.

Sean C Davis: a decent segue to talking about some of these personal persistent agents.

Taylor MacDonald: Yes, indeed. one last thing I want to point out. I

Sean C Davis: Yeah, go for it.

Taylor MacDonald: I love this. California sued open AI over the hugging face intrusion.

Sean C Davis: Yeah.

Taylor MacDonald: plant that seed. That happened on Monday. So more to come on that next time. But yes, let's let's move to the next topic here. So persistent personal agents. So we've talked about this in sparing terms. And the reason I'm bringing this up is not till I go deep down in the technical weeds, but to kind of talk about the the sea change here, right? So most of the Last year and a half, two years or so that people have been using these chat bots, it is largely a transactional one-way communication. hey, chat, do this thing, and it'll go do research and come back and give you the output, right? Cool, turn that into a markdown file. It'll do that, give you the output, you download it, do whatever you need to do with it. that's fairly limiting, right? Like so it's the reason that that I mean that was powerful because we're all very comfortable with like that messaging interface. And it can do

Sean C Davis: Mm-hmm.

Taylor MacDonald: really powerful things, it's great. But the real power is starting to emerge, is stringing a lot of those actions together. So if you were to say, hey, I want to you know, order a sandwich and I don't have time to do that, hey AI agent, could you here's my credit card, go order me a sandwich, you know, and I'll look forward to having that arrive in an hour. Well, that requires a level of both proactive kind of effort on on behalf of the agent, but also a trust that you would need to establish with that agent. And so far, pardon me, so far we haven't had a whole lot of I don't know, work done in that space around trust. However, this seems to be changing. So every major service provider is now offering some semblance of this kind of proactive, persistent, personal agent thing where you can give it a task and it can see through multi-steps to the end conclusion and give you the output. Some examples of this. We've talked about Muse. So Muse from Meta was the first commercially viable example of this. Very interesting news on Muse. Apparently, even as of yesterday, Reuters Reuters reported that Muse was having all sorts of issues. It was like mixing up conversations between users, and it was like talking about purchases that

Sean C Davis: Mm.

Taylor MacDonald: the user didn't. Make and all crazy shit. And even including bank balances that didn't belong to the recipient, crazy, crazy stuff. So so that's kind of so we're definitely still in the wild, wild west. But anyway, so this is their product Muse. I I talked to a buddy of mine who's running a contracting business and he's using it almost exclusively to run his whole business, which is crazy. Grokbot is another one. Their marketing is Everywhere for me. every time I open YouTube, it's like, Crockbot can change your life and do all these things. And it's like,

Sean C Davis: Ha ha ha.

Taylor MacDonald: all right, cool. let's see, Claude Duet, which is currently in a research preview. that came out back in October. I still don't have access to it, but I think I didn't sign up in time. Or apparently on November 13th, some anthropic staffer reportedly posted that everybody in the US with a wait list is now you know been invited. I think I I think I just signed up like. Week, so I probably don't I haven't gotten in yet. there was a product called Gemini Spark, which does the same thing. This is Google's AI version. they have now migrated this into a product called Tasks and Gemini, which kind of makes sense if you think about it, because all this really is is just like automation, you know, glorified productivity workflows and things like that. so I'll stop there. chat GPT dots. That's that's the open AI version of this.

Sean C Davis: You're right, yes, yes.

Taylor MacDonald: And I dude, I got confused because Chat GPT released these like goofy, like almost clippy things, which look like that I think they call them pets. And so I got really confused. Was a dot a pet or is a pet a dot? Like I don't Anyway, regardless, yeah, pretty cool. what do you what do you think about these? Are you using any of these?

Sean C Davis: I I haven't, and I haven't because I all right. So I I really I like piecing the workflows together. I think that there's there's gonna be a lot of benefit and and it's gonna be really interesting to see, okay, what does this mean for the future of e commerce and how we shop and all of that? Because it's even it's it's it's kind of a pain in the ass even to get Claude or Chat GPT to interact on on Amazon. It'll be like, well, I can do the search, but I can't read the product details. And so I think it's kind of interesting from that perspective because you've got you have the the service providers like the or the you have the e commerce providers like the Amazons, and then you've got the AI providers like the Anthropics and Googles and whatever. And they both want they they want your fingers on their product, which I think that's gonna be interesting to see how that shakes out. The I I like it. I like the idea in theory. Something I was thinking about was you know, I I I feel like I'm inching in that direction anyways, and and so that kind of follows my pattern like, I'm gonna build my own little bespoke workflow and then next week Google or Anthropic or Chat GPT or OpenAI are going to come out with something that's basically makes that a waste of time. But I the the things that that kind of stick in the back of my mind that I worry about a little bit are one is are is is there a point at which we're going to there the these AI providers are gonna stop consuming everything around them and try to run your whole world in their little tiny ecosystem. Where does that end? Because that feels a little scary to me. And then the other side of it is if you're gonna pit these against each other, you know, Muse did a really great job with marketing and they hit the market early and they got a ton of people to sign up. I hate Meta. I I mean like have a I have a burning hate for a lot of these big tech companies, but meta in general and here's here's the other thing. Meta makes its money from advertisements. So if you're gonna pick one against the other, do you wanna put all of like all of your personal interactions and all that when in a company that thrives or that exists as a result of advertising? It's like I want less advertising, not more.

Taylor MacDonald: I agree. that's a really great point. I hadn't really considered that. I just kind of operate under the assumption that my data has already been exfiltrated twenty times over. I mean, from Cambridge Analytica

Sean C Davis: Yeah, yeah.

Taylor MacDonald: to Doge, you know, basically liquidating data stores in the US government. Like it just it's so pathet it's I mean, I almost feel like a nihilist. Like I I what I believe in nothing. I don't believe in

Sean C Davis: Ha ha.

Taylor MacDonald: privacy. To the extent that there's so many bad actors out there that have a financial motive to take advantage of that. And there's zero regulation or, you know, oversight from our elected officials. So yeah, I don't know. I mean, part of me feels like when the the world is going to shift. Part of this internet meltdown is the world shifting before our very eyes. Like we have got to get right with some of this. And Whether not privacy is still a thing, which it should be. I mean, arguably should be. There's, you know, I saw some congressman from Ohio, I saw him speak at an AI meetup a while ago. And his big thing was digital currency. And he was like, you know, spouting off about how Bitcoin and all that stuff. It what we are doing by adopting these systems is putting record to something that formerly never needed a record. You know, you know. physical tender, being able to give you twenty bucks in exchange for services, that does not require you know, a tracking mechanism, right? And there's a lot of value in that. And the minute that we kind of couple ourselves to these digital systems and say, well, here's a record of everything I've done, that we are relinquishing so much control or we're giving up so much, you know, visibility into, you know, what was formerly Transaction between me and you. It doesn't involve Elon Musk. Now, what we've really done with this AI thing is we've taken all this human derivative data and we've put it in a place where it can be acted upon. You know what I mean? Whether it's human language or whatever,

Sean C Davis: Mm-hmm.

Taylor MacDonald: all we really did was digitize the words, thoughts, and meanings of things and put it into a space that now the interface can flow, like we can flow data between a human brain and a computer in a in a reasonable way. I am of the opinion that eventually they're gonna figure out how to pipe this stuff directly into your neural cortex. what's the that I know Elon Musk has or Neural Link, that thing. Like I read a report about how they're using technology

Sean C Davis: yeah, I've heard of this.

Taylor MacDonald: like Neuralink to to take like MRI like like have images of people's vision through MRIs and and digitize those and put them on computer screens. I mean, bro, we're real close like we're and we haven't talked about it yet, but there's a whole subset of technology called wetware, which is the idea of digitizing like cells in the human body and building like circuit connections into individual cells and shit like that. That's pretty terrifying. We'll talk about that at

Sean C Davis: Mm.

Taylor MacDonald: some point. That's definitely on my short list of topics that I want to go research and dig into, but but yeah, I don't know. I mean, it it feels inevitable. and it's hard to stave off like an avalanche, right? So if you're at the bottom of a hill and the avalanche is coming, what are you gonna do? You you know, you can try running. Or you just lay down and accept it. I it's a fairly

Sean C Davis: Ha ha ha.

Taylor MacDonald: Discouraging take, I guess, but I don't What are your thoughts?

Sean C Davis: yeah, yeah. I mean

Taylor MacDonald: Hold on. We

Sean C Davis: I th I think

Taylor MacDonald: gotta bring we gotta bring this back to our time. We're gonna run out of time here. And you have a demo that I

Sean C Davis: That is true.

Taylor MacDonald: want to get to. So so let me let's table that existential crisis for next week. And for

Sean C Davis: Yeah.

Taylor MacDonald: now, let's talk a little bit about the upsides of this. So these personal persistent agents are extremely powerful. If the idea is that we can give them things and access to systems and they can go do stuff, what could they possibly accomplish? Well, what my argument has been that today. You don't have to be an engineer to build these applications. Web applications are a dime a dozen anymore, right? Today it's almost better not to be an engineer. And I think what you're seeing is a lot of these organizations focusing on how do we speak to the subject matter expert, the person who knows their craft so well and knows exactly what they want, but they're not an engineer. Well, now that person can sit down and build an app. So This past weekend I had drinks with a guy who a financial planner right here in Cincinnati who spent three months like basically vibe coding his an entire operating system for his business, not an engineer. I watched a podcast on Substack the other day about a personal finance guy or a lawyer in Hong Kong who basically built an app to extract red lines from legal documents. on the fly, because what'll happen is if you give it to Claude, it'll just collapse it all down and just it can't read the red line. So he he vibe coded an app to do this. That's amazing. These people are not engineers, right? None

Sean C Davis: Mm. Mm.

Taylor MacDonald: of these people can actually read the code. And to that prior comment about, you know, we need to be able to read every line of code, obviously they're not, right? But they are solving real world problems and making measurable impact on their businesses or their, you know, sphere of influence. they know what a wrong answer looks like in their field. And that is why they are able to do this. And they arguably can do it better, faster, and quicker than an engineer who is not trained in that subject matter. the problem therein is how, if you're doing this, Sean, how do you keep track? You know, we've all read all these articles about hallucinations, massive security vulnerabilities, leaked secret keys and passwords and crap like that onto the public domain as a result of vibe-coded apps. So how Might the enterprising non-engineer subject matter expert apply those principles in a safe manner?

Sean C Davis: Right. Cause you you're it's it's such a it's such an interesting question because you're the the the engineering answer I would give you, of course, is it depends, right? There's it's a spectrum. and what we really need to do is on an individual level, you've got to establish where where is the bar for It's what is what's it's like that risk spectrum, right? There's risk on one end and there's I don't know, fully confident, or or no, maybe the other end is more like capability. And so we've gotta you gotta meet somewhere in the middle. Like what amount of risk are you willing to accept? And only and and I think the big thing here is that only the subject matter expert can really make that call. And so here let me. We were talking about the pull review process and that was more like, Okay, well what do I do when

Taylor MacDonald: Let's let's define what the poll could could you articulate what you mean by pull review process?

Sean C Davis: Yes. I think I I was thinking about how do we translate this process to anything the the the testing process for any any scenario across any industry. You have something that you are trying to deliver to somebody who's willing to buy it, a cer a good or a service, you need to test it in some way before you provide that thing and earn money for it. and so when we talk about pull requests.

Taylor MacDonald: And if that system is already online and you're making an update to that, it's doubly more important that you're testing this in advance of rolling out those updates, right? As we've seen with all these outages

Sean C Davis: Exactly. Exactly.

Taylor MacDonald: recently.

Sean C Davis: And so when we talk about pull requests, we're talking about code essentially. Code is ready to go into your product in production. what do we need to do? But I'll just try this more generically. Okay. We're gonna go deep here. It's gonna look a little crazy, but I'm gonna we'll I'll I'll come back to it. Or I'll

Taylor MacDonald: Sure.

Sean C Davis: I'm I'm gonna speak in in general terms really, because I think there's Th looks like there's a lot going on because it's really but it's it's five steps that I'm involved in. And I think one of the things that actually what we kicked this episode off by talking about was where do you as a human belong in this loop? And that question is absolutely variable and should be individualized to whatever you're trying to do. And so what I'm There's there's a big part of this here that is this assumption of I want these decisions to go through me before they make it into the real world in some way. And what I'll what I'll do here, and and so this this is published on our website on blood sweat and tokens.xyz. And so you can go in and and dig into the details, but I just want to focus on a Couple of things here. Really, it's there's there's two parts of this, maybe I'll say three, that I think are are worth discussing for a moment here. The first one is zoom in here, whoa, zoom in here. And that is that if you're using one thing I I firmly believe in, if you're using an agent to do the work, you need to use a different agent to review the work. Now it doesn't mean It needs to be a different model or a different provider. It just needs to be a different session because Yeah,

Taylor MacDonald: Like a f fresh session. Yeah, yeah. Got it.

Sean C Davis: what what you and I talked about a lot, Taylor, was that one of the one of the underlying reasons that we had such is or that OpenAI had such a big issue with the the hugging face scandal is that what these agents are trained to do is achieve the task you give them. Right. And so if you think about that, if you give an agent a task to complete it, and then at the end, after it's completed it, you say, okay, now review your work. Well it's its task was to get that work done and in production. It's going to be skewed how it reviews. Now if you spin up a new agent and you're like, review something that you have no stake in and your task, your goal is to give feedback on this thing, it's actually more inclined to find issues. And what I found is it finds way more issues than actually matter and need to be addressed. And that's where your human brain comes in and you're like, okay, cool. You gave me 10 things that are wrong with this. We're gonna pick the three that we actually need to address here. So I think that's a that's the big distinction, or that's that's the the first one. Review with a different tool than you're building with in some way. And then the second one is what I do with the the reviews and what I have found, and I think it's still the case, even though we did see some I did see some content come out this week that's saying don't worry about subagents anymore. Is that okay, Taylor? You were talking about having using a an agent. To orchestrate some other tasks and you know, like go, hey, go transcribe this, and we're gonna send that off to Whisper to do that work. I do that in the review process, but what I have found is that so similarly to having a an agent, a different agent do the review than does the work, that having agents from different This is this multidimensional. Having agents from different providers with a very specific sub focus works, gets us different results. So what I often do, and I should add more in here, but what I often do on an everyday basis is there is a Claude reviewer who basically just says, here's what is it tells me here's what changed. And then I can ask some questions about it. And then once I feel like I understand the change, I say, okay, cool, go. And instead of it reviewing the code or the change or whatever, it tells other claud agents and other codex agents to review, but very specifically. So it says, Does this meet the requirements? Is it safe? Or do you can you find any security issues with it? And is it simple or you know, is it can could it be simplified? And then those agents work in the that two-dimensional context. They have the goal of what they're looking for, and they've been and and they're you know that and they have a specific model that they're working against. So then the main agent doing the review, this guy over here, gets all of that information back and then it ha and then it checks. So it says, like, okay, well, Claude said this thing. Codex said this thing, if they're the same, it's like, okay, well, this is probably a thing. I'm I'm just gonna pass it on. It seems right. What the almost the more interesting thing is when Codex finds something that Claw doesn't, and then that agent has to be like, okay, well, is that right or not? And either way, it's gonna tell me what it finds. And and then eventually draft it. But so those, those to recap here. We're using a different agent to review the work than we did the work with. So it doesn't matter who did the work. It's just we need something else to separate that work. And then we want to that we've seen a benefit to using different LLM providers to do very specific focused reviews. And those things are coming from me and my expertise as a software engineer. So you can apply that to any sort of review if you're building a physical product, right? you can you you should pit different agents against each other to do that review, or come up with manual tests that you can do, record, and send back. and then what I have found is that I don't really want the agent saying, No if it's if it's like, here's some feedback, it's not valid, that's one thing. But if it's like this isn't important, I want it to grade that, not hide it from me. And so I follow this. where I I had it in here and it and it I I removed it recently, but it basically follow this framework of must, should, could. And so the review agent grades what the the level of the feedback. this is gonna block this being released, or this feels like a preference that you should you should give feedback. And so like for this last week where we're we're sprinting up to the launch, it it's nice for me to be able to read every little detail. Like I s like I said, you know, might come back with ten different points of things that could be improved. At times we might actually want to share all of that with the person who did that work. At other times we're like, no, only this thing that's actually broken. These other things are edge cases. We don't expect this person to use this product in this way, you know, whether it's a piece of software or a dishwasher or whatever. You can't work on the thing forever. so yeah, and and then the other thing I I like to say is, so for that reason. It's nice to be there and to just get succinct information from an agent and be able to ask questions. And that's that's where I'll leave this is that I hate getting a wall of text. I'm not going to read that. And so every step of the way here is like just give me the high level and then I'll ask more questions if I have those.

Taylor MacDonald: Mm-hmm.

Sean C Davis: and this has worked really well so far. There's a visual aspect that I'm not gonna get into today because I think that's interesting.

Taylor MacDonald: Okay. So I have a couple of questions here. So when you're executing this type of workflow, this is a more of a QA review process for a feature or an entire product. Is this like where in your process do you use this workflow? Do you use it at the delivery of every like pull request or do you use it, you know, in routine, like every Monday morning? I'm getting this feedback automated to me, like Talk me through how that gets integrated into your daily workflow.

Sean C Davis: This here is I've described this for essentially any feature request that is going to go

Taylor MacDonald: Okay.

Sean C Davis: into Netlify into production. Now there are there's a variation of this that I have put against, you know, like I I want to make sure that the you you let me say, yeah, just succinctly, like when you go through this process, you also test that change. in isolation and you i I'm I'm giving up some control of I I'm hoping that the that the reviewing agent is taking the appropriate amount of context and if in terms of how does this feature fit into the larger picture and if it's missing that I've gotta fill it in. But I do think to your point It looks slightly different, but something I do on a regular basis is back up and say, for example, okay, is the onboarding process for a new user the same as it was two weeks ago? Is it is it hitting what we want it to hit? and I think that's really interesting because in software development, what we've had to do for many years is basically be like defining this big list of scenarios that we want to be true. And then whenever we make a change, we run through that whole list. And it's it's really slow and it's really expensive to do that kind of work. And I think what's interesting about this is we can now we can separate those things. So when we make a change, we can test the change within the context of that change, but we can also routinely test that the whole system itself works really well.

Taylor MacDonald: Right. Yeah, this is interesting because like I can see how this workflow works for engineers. I mean, if you're dealing with code, you're comfortable with the pull request slash review, code review process, that this is very much an engineering flow attached to you know the way software gets built. One question I would have, if you were the non-technical lawyer in Hong Kong that's building the app, how do you How do you build that with the same degree of confidence that you have? Understanding that person is a subject matter expert in the legal nuance, not in the coding nuance. Right. So if you have to go through findings and you're saying things like, Well, I want you to prove this. Is this real? Is it worth it? These still require a high level engineering perspective to make those evaluations of like, Yes, I agree. Let's move forward with this, but let's disregard that. That mom and pop app builder in 2027. doesn't have that level of of nuance or inference. So do they just take

Sean C Davis: Mm.

Taylor MacDonald: the agent's recommendations verbatim? Are the quality responses coming back in such a manner that, you know, you have a ton of confidence in what that is? Or or moreover, I guess I would ask you the questions like what are the symptomatic steps that a subject matter expert with zero engineering could apply to their own domain and still get the same level of confidence that you're getting.

Sean C Davis: Yes, yes. I I think you you pull a lot of these practices out and that I mean it could be it could be the legal world, which I know a little less about. It could be say finance, for example. I've got a I've got a buddy who works for a company that that runs financial audits for organizations. And they are they're generally slow to adopt technology anyways, but they're they're still checking everything line by line, but one way that he might speed up that that workflow is like, okay, well, they need to make sure that it's all correct. Right. But so what do you do? I think you don't take one single agent's results at face value. Like you've gotta be you've gotta take part of being that expert is being skeptical. And and being like, my my spidey sense is up. This feels like it's wrong. And I think the way to to get started with that is to use multiple agents. You could you could even do it manually, right? You could be like, I'm gonna open Gemini and Chat GPT and Claude and and just give it the same prompt and then cross compare those. Because I think one thing that's happening here. is there's layers upon layers and this workflow is fairly complex. You could start just manually and doing it.

Taylor MacDonald: Yeah, it's interesting. We you described, the way we do product re in product development world, we'll have we'll get all the engineers in the room together, we'll go through refinement. Refinement's just a fancy word for walking through the you know, we wanna do these ten things in the next two weeks. So we'll pull up each one of those ten things of like, okay, what do we think? You know, where are the issues? A lot of that is very similar to what you're describing, right? The idea is that no one person really knows the right answer, and you can democratize some of those questions around the group, a bunch of subject matter experts, and they're gonna theoretically come up with a better potential outcome that works better and considers things that it wouldn't have considered in isolation. using multiple agents and pitting them against each other to kind of gut check each other's work and do the review process. And did you think about this? Did you think about that? largely emulates that same thing. What I found was interesting, there's the a psychological phenomena called groupthink. So groupthink can happen where you get a bunch of smart people in a room, but maybe one personality is dominating or something, or there's you know, like sociodynamics or whatever going on there that would cause the group to ultimately lead to a poor conclusion. I'm doing a terrible job of describing this, but anyway

Sean C Davis: No, you're yeah, I know what you're saying. Yeah.

Taylor MacDonald: I wonder, I just read I wonder if AI agents are susceptible to groupthink and or if the if the per if there are personalities underpinning the the models, right? disaster 6.2, hey, I'm the biggest, bad as badass. I got a 50% on the first pass score run on t agent teams leaderboard.

Sean C Davis: Ha ha ha

Taylor MacDonald: Like I wonder if they are susceptible to that type of thing. I mean, in a lot of ways, the hugging face incident sort of felt like groupthink, didn't it? It was like, hey. This is probably not something we should be doing, but fuck it. Let's go on. I don't know. It's kind of an interesting. I think it would be cool to run a like a experiment on that. Like, could you pit agents

Sean C Davis: Mm-hmm, yeah.

Taylor MacDonald: in a position where they come up with one really stupid outcome because all of them were participating similar to the way humans are? Or is that just a uniquely human phenomenon? Anyway, listen, we're getting close on time. So I want to leave you with one challenge here. Sean, okay, so we've

Sean C Davis: Okay.

Taylor MacDonald: talked a lot about human in the loop today. We talked about managing this review process for vibe coded apps. We've talked about personal persistent agents like Muse and Grokbot and things like this, and how people can take advantage of these technologies as subject matter experts to do things, but you gotta have some amount of control attached to that. We've also talked a little bit about our deck degrading perspective on personal privacy. So here's the challenge. Before next week, I think both of us should get one of those virtual credit cards, put a small amount of money on it and give that to an a personal persistent agent and say, okay, you get one real task and there are real consequences here, right? Because it could blow your budget. Let's say you give it a hundred bucks and you're like, I want you to buy a Christmas, a really thoughtful Christmas gift for my wife or something.

Sean C Davis: Mm-hmm.

Taylor MacDonald: And let's see what happens. and of course, along the way. We're gonna observe, what are those potential gates that could help keep us in control throughout that experience. And then we'll report back next week. What do you think? Okay.

Sean C Davis: I love that. That's a great idea.

Taylor MacDonald: All right. Well, with that, we'll call it a wrap. Thank you so much, Sean. This is wonderful conversation. I'm excited to dig into your agent workflow and put that into practice on some of the stuff that we're doing in AMPL, some of our engineering processes, as well as exploring more about these personal persistent agents. In the meantime, we'll see if the rest of the internet melts down before we meet again next week. Hopefully we're here in the week. but if not, hope hopefully

Sean C Davis: That should be our new tagline for this show. Hopefully we're here next week.

Taylor MacDonald: there's still podcasts this time next week.

Sean C Davis: Ha ha.

Taylor MacDonald: Anyway. All right. Well, thanks so much, Sean. Look forward to it, and I will talk to you soon.

Sean C Davis: All right. Thank you, Taylor.
