Critical Thinking

About this Episode

This Acima Development Podcast episode connects a NASA document on critical failures to the modern temptation to over-rely on AI. The hosts open with examples of reward-function failures: a reinforcement learning agent in the game Coast Runners that racked up points by endlessly circling a lagoon instead of finishing the race, and an early GPT model that compulsively opened a calculator because its reward was dialed too high. These illustrate NASA's identified root cause of major incidents: lack of critical thinking and over-reliance on computers, processes, and procedures. The recurring metaphor is "playing with your head up," meaning you execute a strategy efficiently while continuously re-evaluating whether it's still the right one rather than blindly trusting the process.

The middle of the conversation explores how much to trust AI, particularly for code review. The consensus rejects all-or-nothing thinking: AI is another tool, like a human reviewer, and catches different things, so the best approach is using both. Dave shares wins (AI flagging a subtle nil-returning bug he'd have missed) and failures (AI ripping out a carefully crafted Arel query to jam in raw SQL, or re-implementing a feature the team had deliberately killed). The takeaway is that AI can't run unattended because it lacks institutional lore, and an experienced human needs to stay in the loop, especially for high-stakes changes. Low-risk, templatized work like certain database migrations are good candidates to let AI handle, applying risk frameworks (mitigate, eliminate, transfer, accept) to decide where it can safely "cut its teeth."

The discussion broadens into how skills evolve as technology shifts. Using analogies of welders replaced by robots, miners re-skilling into adjacent jobs, and the Pony Express being rapidly outpaced by the telegraph and railroads, the hosts argue that giving up certain "critical thinking taxes" (like manually smelting iron, or writing low-level compilers) frees people to build at higher levels, while foundational skills survive in niche "ranching" spaces like microcontroller programming. The unifying through line is proprioception, a visceral, grounded feel for what you're doing. Even as AI tools do more, you can't fully let go of the reins, because grounding in the underlying craft is what lets you sense when something is going wrong.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I am hosting again today. With me today, I have Eddy. I've got Dave. 

DAVE: Howdy, howdy. 

MIKE: Ramses and Kyle. It's possible we'll have a little bit of ebb and flow in who we've got here, as we sometimes do, which is great. We have a group here that can go into a good topic.

Just a heads-up: we're going to talk a little bit more about the NASA document we've been talking about on the last couple of episodes. So, as usual, I'm going to start with some background story. This one, I'm going to go more on the technical side. 

There's a great example, and you can look it up. It refers to a game called Coast Runners. So, there's a video game called Coast Runners. And some years ago, they built a number of reinforcement learning agents to play it. This is some of the early days of, "Hey, we can actually build AI that can play games, and it's working." Because there were a lot of years where they couldn't make that work very well, and it's kind of some of the first steps, "Hey, this is actually working." It led to some of the breakthroughs where we saw, you know, amazing things happen a few years back in video game playing, you know, including beating human players in a variety of games. 

Well, this game, you know, it's not chess; it's not Go. It's a video game where you drive a boat, and you try to win the race. And they built the environment, and they built these reinforcement learning agents to play it. And not surprisingly, the reward function was to maximize your score. And here's where things get interesting. The score in Coast Runners is partially determined by winning the game, but it's also determined by how many of these little boxes that pop up that you can hit along the way. And what the agents learned to do is there's a lagoon that you can go into on the side. And...well, I'm going to read this. This is taken directly from a write-up that OpenAI did on it. It said: "The reinforcement learning agent finds an isolated lagoon where it can turn in a large circle and repeatedly knock over three targets, timing its movements so as to always knock over the targets just as they repopulate.

Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way. Our agent achieves a score on average 20% higher than that achieved by human players, [laughs]" which is to say, computers will do exactly what you tell them to do. 

DAVE: A follow-on that. In one of the...I think it was ChatGPT, in their training —the early ChatGPTs —if you'd ask a math problem, it would get it wrong because it was just doing words and tokens, right? It wasn't actually conceptualizing the numbers. And one of the things that they started reinforcing was open the calculator. If you get a math problem, open the calculator, type in some numbers, and add it, and when you hit equals, you get a cookie, right? We're going to increase your reward satisfaction score. 

And they shipped this thing, and I want to say it was GPT-3, 20 to 40% of queries that you sent, no matter what they were about, it would silently open up the calculator, add one plus one, and hit equals, and give itself a cookie, and then go back and resume [chuckles] and answer your question.

They had prioritized use the calculator instead of confidently guessing off of the words, right? Because AI's confidently [inaudible 03:42] the wrong way. They had dialed the reward up so high that it liked playing with the calculator. And it just reminded me of being a junior in, like, geography class playing [inaudible 03:51], you know, waiting for math class. Bless his little heart. 

MIKE: Exactly, well, yes. And the reward will be maximized. So, NASA looked at some of the failures that they had had, and we've been talking about this document they published about a decade ago. And they said that one of the root causes of the key failures that they've had in some of the major incidents was lack of critical thinking. They refer to it as over-reliance on procedures and computer codes. 

And there's a longer paragraph; I'll quote from this as well. Sorry for reading the quotes, but they're well-written [chuckles]. It's worth giving some context. "The third root cause is closely related to second. It is the lack of critical thinking with an over-reliance on computers, processes, and procedures. Computers, processes, and procedures are necessary but can never take the place of the human mind. In the end, all of our resources, tools, et cetera, are there as an aid to the human mind and the human creativity and decision-making."

Now, this is fascinating a decade later, where we have AI becoming so much more successful, and we're outsourcing a lot more of our thinking. And if we rely on that without supervision, we get what we ask for. You know, like King Midas [chuckles], we may get exactly what we ask for and regret it for the rest of our lives.

DAVE: There's a fun takeaway to this as well. I'm going to take this immediately and go meta, which is that, like, when I was in college, somebody barked at me and said, "Dude, you need to learn to play with your head up. Did you not ever play basketball?" And I'm like...I was not athletic at all. I don't even know...That's the round one, right? 

MIKE: [laughs]

DAVE: And, yeah, right? So, he said, "When you play basketball, what they teach you is don't look at the ball. You have to play with your head up because you have to be watching. You have to be looking at the net. You have to be looking at the other players. You've got to be able to dribble the ball without watching the ball."

That stuck with me because he wasn't talking about dribbling the ball. He was talking about I was writing a piece of software that nobody in the room wanted. And one of the executives who had direct control over my paycheck directly did not want, and it was costing me dramatically politically. And so, basically, it was like, read the room, right?

But the phrase "play with your head up" has stuck with me as, once you've decided on a strategy, you should execute the strategy efficiently. But you constantly have to be evaluating: Is this still the right strategy? When you find a good idea, it's very valuable; but also valuable is when you get rid of an idea that's no longer valuable —that's no longer earning its keep.

And if we say, "Don't use AI because you won't learn nothing," we are at risk of falling into the very same trap that we just outlined because, for instance...I told this story on a previous podcast. I needed an auto clicker in Linux, and I know how to write one, kinda. I mean, I know the principles. I know how to write a timer. I know how to send input to the mouse. I know how to capture keystrokes. I know conceptually that I'm going to need to intercept a keystroke in another application, and that's going to require elevated privileges because that's technically a keylogger, right? And I don't know how to do any of that in Linux.

15 years ago, I could tell you from memory how to do it in Windows because I was a Win32 programmer for years and years. But I never learned how to do it under Windows, under the X11 system. I went to an AI, and I said, "I'm just playing Minecraft, dude. I want to AFK and grind some resources. I just want an auto-clicker, and apt search on open-source repo isn't turning up an auto-clicker. Write me a clicker." And the AI went off for two hours, came back, says, "Here's your auto-clicker." I looked at the code and went, "Yep, that's an auto-clicker," and I've never looked at it since.

Have I just cheated myself out of the knowledge of making an auto-clicker? Yes. Yes, I have. In the same way that every machinist working on an aircraft fuselage is cheating himself out of the fundamental knowledge of how to smelt red earth into iron to forge the...right? That's great if that's what you want to do, but if you're trying to do this other thing, pick and choose your battles and know when to do and when not to do.

I posted in our AI horrors channel a while back that I vibe-coded an AI chatbot, and it did it─Eddy, you'll love this─in React, and I didn't take the time to learn React. I'm like, "I'll just trust the AI to do it." And it ran great for about two weeks, and then the app slowed down, and slowed down, and slowed down. And every time I would hit a keystroke, it was redrawing. It was re-rendering the entire chat in order to catch some live event. It was doing the most inefficient way possible, literally re-rendering every character on a, you know, in a chat that's getting longer, and longer, and longer, and longer, right? And it would just grind the server to a halt, and I found myself unable to debug it, and I ended up scrapping the project.

I chalked it up to a lesson learned about don't vibe code something if you need to understand it. I don't regret doing it, but I definitely...I paid for that one. That one I should have gone to the AI and said, "Teach me how to do this, or tell me where to go learn it." And it's which strategy is right right now, and which strategy...We don't want to just blanket choose this is what we are going to do; this is what we're not going to do. You have to play with your head up because the right course of action is going to change from second to second during the game. 

That's my speech. Yeah, so [chuckles] didn't mean to go all soapboxy on it, but yeah. 

MIKE: Well, it matters; it's directly correlated here. Because we're talking about...you say playing with your head up. If you're looking at the ball, you'll miss it, and that's the ability to not look around has increased here because we've got tools that can do some of that for you. So, you can get yourself further in the hole [chuckles]. 

You know, you can go out...you know the old cartoons where somebody would run off the edge of the cliff, and they don't fall until they look around [chuckles]? You get a lot farther off the cliff before you look around here, and the consequences can go up higher. Because it looks like it works, and, boy, there's just so much potential to go wrong because there's so much potential to accomplish things, right? We've been given these big tools that do lots of cool, powerful things. It could sound like I'm being gloomy, but what I'm saying is, wow, we've been given superpowers, and what next? 

DAVE: Yeah, absolutely. This actually harkens very strongly back to the very first thing we talked about last week or two episodes ago, or one episode ago, sorry, two weeks ago, which was the normalization of deviance, right, where you get stuck on the process. We are launching a space shuttle. Don't bother me about it's too cold. Don't bother me about some engineer actually knows for a fact that those O-rings are going to freeze and leak, and we're going to shut that guy up and shut him down so that he doesn't have time to think all the way through to, "This is going to leak fuel onto the engines and blow up and kill everyone," right? They shut him down, shut him up, and we didn't bother...

So, it's like, we normalize this. We're going to adhere to the process. Stay on target, stay on target, stay on target. Sometimes you need that. You know, when you have to take that hill, even if you're suffering, you know, catastrophic losses, you have to follow the process. You have to execute it. Artists will sometimes say, you know, trust the process. When you're trying to paint a grassy hill and the first thing you're supposed to do is pick up purple, and, whaat, trust the process. It'll be there. Trust the process, right? 

But trusting the process is playing with your head up. Normalizing deviance is trusting...no, is blindly trusting process to keep you safe, and expecting the process to be the sentient overlord that it isn't, that it absolutely isn't.

MIKE: So, we've been talking very meta here, very high level because it matters, right? This is kind of a philosophical question that is going to continue to haunt us here for a while. I'll propose a practical example. How much do you trust AI reviewers to review your code? 

DAVE: You're not going to like my answer. It's increasing every day. 

MIKE: Well, I didn't say I was...That's a legitimate answer. 

DAVE: Yeah. I guess what I would say is, so, I actually led a discussion a couple of weeks ago in our AI channel, where I said, "It's too quiet in here. Let's start a fight." And I said, "We're going to be vibe coding in prod by next year," and it started a fight. Because when you hear vibe code, you automatically think blindly trusting the process of something that's going to be catastrophic, and it's going to da, da, da. 

And, like, we have a codebase that is complex and has enough lore and enough decisions that changed midstream and weren't well documented that you need a human, not just, like, the complexity of human thought, but literally, you need somebody who was here 10 years ago and watched the campaigns, who's been to see the elephant a couple of times. And Claude doesn't have that, Copilot doesn't have that. 

And so, Claude will come back and say, "Oh yeah, this code doesn't match this code over here because it's missing this feature. Let me go add that feature." And we're like, "No, no, no, no, no, we killed that feature in 2021. But we weren't able to remove the code because we were at a dead run, and we were committed to it. It's technical debt. We want to get rid of it."  And the AI, if we blindly let it go, will go re-implement that feature that we decided we didn't want. 

That said, I'm getting reviews now, at least one a week, where there will be a detail that I never would have caught. Like, I've had two now where, like, the...and this might be a terrible habit, but when I review code, I read the PR, the diff on GitHub. And I try to think about it, but I don't mentally catalog, like, "Oh, this is one example of a query. There are 37 other queries in the file. How does this fit shoulder to shoulder with the rest of these?" 

Claude is the one that will come back and say, "Uh, by the way, there are five queries that share the same shape, and this one over here is now missing this...you just deleted an instance variable, and this one over here uses that instance variable. And because it's an instance variable and not a method call, it's going to silently return nil, and you're going to end up with a blank spot on your UI," because it was in one of the view files, right? And I never would have caught that. And we scooped it up; we put it in. 

So, yeah, I don't know. I wouldn't trust an AI to review a sweeping scope architectural change, you know, pivoting the entire codebase from, you know, Postgres to SQL Server, or something like that. I wouldn't just throw that at an AI. But, like, a tiny self-contained change where...Oh, I genuinely don't think database migrations should be touched by humans. I would be willing to, like, if I had budget for two weeks on that, most database migrations, I'll put it that way.

Adding a column, adding an index, some backfills, backfilling old data, you know, that kind of stuff we have templates that we use on our team. We have a standard template for this is the column we're adding; this is the type. We have a standard up migration. We don't use change. We have a standard up, a standard down migration. They have to work a certain way. This is all templatized. There is no reason...I'm almost offended; I'm almost to the point where I'm offended that humans are wasting their lives and their attention on this. 

You should go into Jira, right, or whatever ticketing system you're using, and write up a ticket of, "I want favorite color added to merchants. It should be a string, free form, 50 characters. We might make it an enum, but we're not doing it right now. Go. And we don't need it indexed." And the only review...in my opinion, you could teach an AI reliably to make it to the point that another engineer reviews the Jira ticket and says, "Yes, this is good," and half an hour later, it's in production. I don't see any reason we can't build a system that is trustable at every single step.

Would I let it backfill? Probably not. And would I let it write that PR today? No. No, I would hand-hold, walk it through the entire system. And I would also have it add all the missing pieces that we don't have that...because we, as humans, we deploy something, and then we watch all the things afterwards to make sure everything is still stable, right? Those aren't automated. Those should be automated. Not just automated, but they should be watched by an AI that's thinking about, you know, threshold and alarms, and will go find a human when stuff's panicking. And if it can't find a human, might even be, you know, if it was an auto deploy, might even be empowered to auto rollback.

KYLE: I think about this one a little bit different just because...and maybe people won't like my comment here. But at the end of the day, it's another tool, and at any company that you're at, so are you. Like, you're just a tool, right? And so, in my mind, do I want AI reviewing my code? Yeah. But do I also want Dave reviewing my code? Yeah. Both of them are going to find different things. 

DAVE: Absolutely. 

KYLE: I don't want the pendulum on one end or the other. I want both. I want the best of both worlds, right? And auto-deploying, like, I would be a bit leery there, but, you know, if the AI reviewed it, and Dave reviewed it, and it looked good there, yeah, sure, send it out. But I think we get hung up on that; it's an all-or-nothing.  And when we get into these conversations about, like, "Hey, AI will be reviewing your code," cool. If it's in addition to, that's wonderful. 

And, I mean, there are going to be specific use cases where, I mean, my eyes are going to look at a DB migration and just glaze over, and I'm useless there, you know? AI is great there, and we probably should rely on it there. But it's like introducing different skill sets, right? Like, sometimes you want a DevOps engineer on your review, sometimes you want QA on your review, sometimes you want devs, IT, you know, depending on what you're doing. It's just another mindset. So, at least that's my take on it. Use all of it. We're all tools. It's a tool. 

DAVE: Yeah, yeah. And playing with your head up, right? We're not just going to say, "Oh, we're going to make the AI do all the stuff and stop thinking about it." That's literally what we just said don't do, right? 

KYLE: Yeah. 

DAVE: You're absolutely right.

We had a PR come through this week where one of our most senior engineers and one of our smartest contractors...the contractor couldn't figure out how to do a really difficult subquery without going straight to SQL. And so, they put a heredoc in there and jammed in a bunch of SQL. And Adam, one of our...Adam [SP]Loper is one of our devs, and he can shatter a human mind at 50 paces. You do not want to step to him in mental combat. I love him. He pushed back, and he said, "We should be able to do this. We don't want to put SQL in the code." 

And the developer that authored that PR was like, "But what about, you know..." And so, Adam helped him write the Arel query, not ActiveRecord. He actually had to go to Arel, A-R-E-L, to build it out. And they had to push back and forth on how big the subquery was and the cardinality, and, you know, it got into, like, the kinesthetics.

I ran Claude over it, and Claude said, "Oh yeah, here." And it ripped out the Arel and said, "Here's a heredoc with the SQL embedded." I'm like, "No, no, no. You are helping in exactly the wrong direction," right? 

MIKE: [laughs]

DAVE: Yeah. 

KYLE: Right. 

DAVE: So, yeah, you absolutely cannot let this run unattended, right? And if you find anybody that's managed programmers, just ask them if they've ever had a very enthusiastic, very brilliant junior programmer who's going to show up at 8:00 and plow rows and then go home at 5:00. And if you set that developer down next to the freeway, if you don't watch them, you'll come in in the morning, and they have plowed the freeway, you know, for their eight hours of back and forth. AI's just doing that faster. We're just seeing it on fast-forward, basically.

I don't want to turn this into the AI show because...or maybe we do, I don't know. But specifically, we're talking about, like, the NASA critical failures, and I'm mapping it more, for me, mapping it more into human space of, like, what are the mistakes that we make? Because you can make these errors outside of software. You can make these mistakes in your marriage. You can make them in your career. And that, again, trust Dave to take it meta. I like to see what is the shape of this error? Does this shape fit anywhere else? And what does that teach us?

KYLE: Well, it's just, I mean, it's just one of those cases for critical thinking, right, that we run into, which is we're...Be it AI or something else, we're handing off our critical thinking. And when you do that, you can run into issues; you run into holes. 

DAVE: It's the, what part of the critical thinking is going away, right? It's I want to give the AI, like, if I'm a blacksmith, I want to give it the knowledge of how to smelt the ore into usable iron. Like, I don't want to solve that problem anymore. I want to be thinking about building civilization.

I've used the analogy in the past of, like, "Is AI coming for my job?" And I said, "Guys, it's 1970, and we are welders in Detroit, and they are wheeling robots off the truck. Yes, this is going to change our job." One or two of those welders stayed back at the plant and learned how to run the robots, and they had to do all the critical thinking. And they had to get smarter at their own job in order to stay on top of that. But the actual job of, like, stacking dimes, which is what it looks like when you give a really good arc weld down between two pieces, the melted metal looks like a stack of dimes, a robot can do that better than a human. And I'd rather have the robot do it. It's going to build a safer car. 

But the humanity, yes, I want to give up the critical thinking of how do we stack those dimes? No, that's not even right either; that's not even right either because the 98 welders that lost their jobs, they hated the robots, right? They put them out of a job, and they were miserable. But then follow up with those people six months later, and they are now running welding on other teams in areas where the robots can't get to. They are welding upside down on the bottom of broken water tanks.

It's like, we put these guys out of a job, but what we did is we freed up 98 skilled welders to build more civilization in other areas. Because the critical thinking that they gave up was also a tax that they had to pay, and the only thing they got out of that tax was stacked dimes on a car frame. And now they're, you know, fixing battleships and water tanks. You know, it's...

This is a personal thing for me because my dad worked as a miner, and they shut down the mine, his uranium mine, in the 1980s. Nuclear got frozen out, and it put him out of a job. And six months later, he was working construction, and two years later, he was a water operator. And those weren't just random re-skill, start over your career. It absolutely was, "I'm going to move adjacently from this related job to this related job, and then from this job to this job." And it wasn't just who do you know, and where do you go work? It was entirely what are the skills that you learned from this that apply here that nobody else has because they didn't have 26 years of blowing up rock a mile underground and mucking it out?

MIKE: Definitely gotten deep into the AI. How does this idea apply in non-AI context? That is, can we think of examples where we've relied too much on the process and fallen really short? 

DAVE: Or where the process was outdated, and we didn't inspect the process. 

MIKE: Ooh, yeah. I think a lot of...When I was preparing for the podcast today, I was thinking back to a time when I wrote some code I was proud of. I wrote the unit test. It worked perfectly. I went to do some manual testing, and I realized this code works perfectly at doing the wrong thing. It doesn't meet the requirements. I had followed the process to the T, and I had tested it. Everything worked perfectly, and it was wrong. If I had done some earlier [chuckles] manual testing or, you know, something to break out of my thought process, pairing, for example, I could have saved myself some time. Well, a little example.

Do you have all examples of times that you have depended on the process and ended up regretting it? 

DAVE: We were talking in the bullpen today about having product people in refinement. And I was arguing vehemently that not only should we have the product people, basically, we should have all the way up to the customer. Like, Agile programming, you take any principle, and you dial it to 11, right? And their idea for the most effective way to make sure you're always building the right thing and the thing the customer wants is to have the customer in the room. You dial that to 11. You have the on-site customer. You literally have the person who's going to use the software sitting with the engineers 40 hours a week, like, literally on site. 

And then big companies came in, and they wanted to keep doing Waterfall, so they canonized it. They said, "We're going to firewall this. We're going to have a product manager, product owner, and that person is going to now play telephone and hide the fact that it's Waterfall because this person's going to gather, you know, data from the customer, and then come be the product owner in-house."

That can be an efficiency saving. Some people will want you to write software, and they don't want to come sit with you. They don't want to give an entire employee over to you just to be the authority on a...especially if you've been hired by a one-man shop, that's, you know, that person's the chief cook and bottle washer. They can't be your onsite customer 24/7. So, product owner is a good idea, but if you fall into Waterfall, you can end up with the customer talks to the product owner once, and then the product owner develops some stuff once and hands it off to the team leads, and the team leads come up with a design.

That's the 1980s SDLC that, you know, provably has a track record of failing 90% of the time. Because by the time you are in the debugging phase, you're going to find errors in the design phase. You're going to find out, like you said, Mike, we built the wrong thing. We have successfully put the ladder on...We've climbed the ladder and realized the ladder was leaning on the wrong wall the entire time. 

So, yeah, this is a process that we're using here. Our parent company uses a very formal, time-honored software development process that I don't particularly love. It's not super Agile. It's a lot more...what's the word? It's like Scrum rather than Agile, like, Scrum with a dollar sign and a TM at the end of it. Like, it's very enterprise, very process.

MIKE: [laughs]

DAVE: I'm trying not to get myself fired if...No, I'm kidding. I'm kidding. But it's very formalized, and it's very process-oriented, and it's good for working at scale. It's good for having all your product owners talking about the same thing. But at Acima, we were always small and kind of startup-y, and every team was doing kind of its own thing. And you kind of get apples to oranges, and standardizing that is kind of miserable, and you can make the mistake of standardizing the wrong things.

Pulling in this kind of Waterfall-y bit that works at scale, if all the other pieces at scale are present, if all the other bricks in the wall are there, then you can use the same kind of mortar. But if you grab this one brick that isn't well-supported because we don't have the other pieces of that scale in, it's a terrible idea. So, that was one for me. 

We ended up with a design decision of, like, how are we going to deal with this particular lease in a specific condition? And the answer was actually something that should have gone all the way back to the customer. Because the decision that we made kind of turned it into, are you viewing the entire purchasing process this way, or slightly this way? And it was a perspective shift all the way back to there. And yeah, we were at risk of building the wrong thing.

MIKE: I've certainly been in that situation before. I worked previously at a company where we built a content management system for mostly small publishers, newspapers, magazines, that sort of thing. And I remember meeting with somebody from the newspaper, and he said, "Yeah, I can tell you're not newspaper people." I'm like, "Why?" He said, "Well, the way that you label these fields is totally different from what newspaper people would say. They don't call it the author; they call it the byline [chuckles]," and so on. Like, "Oh yeah, well, we can change that." 

But we were building this interface that was foreign to them because we hadn't spent time just sitting together and saying, "Okay, you know, how do you build out an article?" We'd built what we had imagined that they would want, and it generally worked, but it was a mismatch. It was a mismatch.

And it led to also us building some things that they didn't really need. Like, we improved the interface for them to build the content, what they really needed was, you know, what they needed to do was make money [chuckles]. And so, they needed better ways to make money from their advertisers. I could go way into this, you know, what killed newspapers was not the internet. It was Craigslist because it took all their classified ad revenue away.

DAVE: This is another thing that came from the bullpen. I'll ask...so Ramses knows the answer to this one. The Pony Express lasted for a surprising amount of time. Does anybody know offhand how long the Pony Express was around? 

MIKE: Randomly, I've looked this up within the last two weeks.

DAVE: Okay. So, you know it wasn't, like, 100 years, right, or 50 years. It's the other end, right? It was, like, 18 months. It wasn't because it was inefficient or wrong. I mean, it was a brilliant idea. It was an idea so good that it changed the way people thought.

If you were in Kansas City, San Francisco was a foreign country. You heard about it on the news. If you wanted to send a letter to San Francisco, you sent it south to Louisiana to get on a boat to go out of the Gulf down around South America. You would find out, hey, how's my investment doing in the Gold Rush? You wouldn't hear back until next year. And all of a sudden, the Pony Express comes along and says, "Hey, the Mormons have settled out in the middle of this deadly, deadly desert. I think we can actually make a shot across this desert. And by whipping the horses hard, we can not only get across there, but we can do it quickly. We can get a round-trip message, like, what, 10 days?" Something like that? 

MIKE: Something like that, yeah. 

DAVE: Maybe less. It was a matter of weeks, what used to take a year.

And now you got people in Kansas City that can hedge fund arbitrage and say, "Hey, I will sell you futures on your shipment coming back at this rate, and it's going to sound really, really good to you because you don't know whether they struck it rich. But I do because I've heard, and you're not going to find out for another 10 months."

And it was so good that everyone...all of a sudden, America was...it wasn't two countries separated by Mexico anymore or the Utah territory, this deadly desert, right? It was now all of a sudden one cohesive whole. And we could actually communicate from, you know, Kansas to San Francisco the way we could communicate Kansas to New York, contiguously. And then with the Gold Rush and the hedge fund arbitrage and all of that going on, people said, "This is absolutely brilliant. Let's build railroads and a telegraph." So, basically, the Pony Express was so good it immediately attracted a much better competitor. The Pony Express revolutionized it, and then immediately got out-revolutionized by someone coming behind it.

MIKE: So, we've had a lot of ideas thrown out here of varying connection to the original thought, which is that you need to pay attention to what you're doing and have a...I'm thinking words like tactile or visceral that suggest you have some human sensory connection that you can have a gut response to what you're doing. You can't be sipping drinks out on the beach thinking that the system's working. You have to be engaged, and there is temptation with the tools we have out there, because they're so good, to disengage. But you see, this is over a decade ago that NASA wrote this document. It's not a new temptation.

When things start working well, you think, "Oh, they're fine." Or in the Pony Express era, you think, "Hey, we've solved this problem. We're ahead of this. And we took out a lot of debt to do it [laughs] because it was a great idea," which it was, but now, you know, they're in trouble. You know, it's not a new problem. You have to be engaged with your work, and have, as we've talked about for the last couple of episodes, be actively engaged with your work.

You have to strive because you can't possibly know everything. But you want to have...So, you talked a lot about blacksmithing, and, like, it helps if you've smelted iron once, right [chuckles]? 

DAVE: Yeah. Yeah.

MIKE: It just gives you a different perspective on things. This idea of, you know, doing things without having that grounded connection to it is the through line here. You have to get grounded somehow and keep that. Even though your grip is going to be less because these tools are going to do so much more for you, you can't let go. 

DAVE: That's actually a good way to view it. I love that, like, the kinesthetic proprioception, there we go, is the sense of your own self, right, and especially kinesthetically. You've been...you said visceral. 

MIKE: Simone Biles, she has amazing proprioception. She knows where she is, but she's doing flips through the air [laughs]. 

DAVE: Yeah, yeah exactly. You need to have a feel for, like, where am I? When I'm developing software, I do. I have a feel for, like, how close to done am I? How close to my next problem am I, or, you know, this thing that I'm trying to solve, how close is this to working and testable? And what can I touch, and when and where?

And all of that are...Kyle, you mentioned, like, we give up some of our critical thinking. All of this is under an umbrella of, what are we doing? And what we're doing is we're shipping software and...Well, okay, but go up a level. What are we doing? We're a leasing company. Okay, but what are we doing? We're a company. We're making money, right? At some level, you go up, and when you're at that level, the things underneath you might be interchangeable skills.

We've all had engineers that, well, I mentioned at the top of the thing. I had an AI write a React website, and it was catastrophically bad. It absolutely...I believed the overpromise, and I paid for it. Executives have had big layoffs last year because AI promised them some things that it's turning out AI couldn't deliver. This is that. They vibe-coded their org chart, and now they're getting caught in the shorts for it. And, you know, we sit back, and we eat popcorn, and we go, "Oh, this is very satisfying." And I'm like, this is the same pattern. And those executives need to smarten up and go, "Okay, what part of my critical thinking did I give away that I shouldn't have? And what part can I keep?"

And at every level, that proprioception is what's enabling me to feel things. Like a database migration, that, to me, feels like something that is nothing but paperwork 9 times out of 10. And I can look at...and I would need a human, and I need to be that human. I can look at a database migration and tell you if it's a candidate for a rubber-stamp AI to paperwork it, right? You know, adding a column, adding an index.

Maybe dropping an index I wouldn't trust an AI to do. This is a little trickier. Actually, yeah, you get the idea. Some backfills, you know, oh, we're going to backfill the leases table. We've got millions and millions and millions of those. No. I want to fine-tooth comb that all the way along.

But I needed to add an index to one of our tiny tables. The table had fewer than 100 records in it, and it wasn't something that got accessed by the customer main. It was something that got accessed on the admin side. I could have dropped that table, and the company wouldn't have lost money. Some people would eventually have been upset, but, like, it wasn't catastrophic. That would have been a fantastic candidate to let an AI cut its teeth on because it could flame out. 

You literally could just choose accept out of your META, you Mitigate, Eliminate, Transfer, Accept, the four things you can do with a risk. We could just choose accept. If the AI catches fire, we let it burn down, and we eat popcorn, and we roast marshmallows. And you can't do that with the leases table, right? We do that with the leases table; we're all looking for work. Make sure the AIs can help us write a shiny resume. 

Yeah. What are you doing at a high level? How do the skills that you use, how do they feel in your hand? And that, I think, is something that I'm just now realizing, as I'm talking through that, that is kind of what I'm feeling is, an experienced person needs to be over the AI, and that a junior programmer might not have that internalized skillset. And they're going to give away some critical thinking that they shouldn't have, and they're going to learn some lessons from that.

There were some big events early on in my career that I did not learn enough of the lesson from, and I had to learn it again, right? And that's going to happen to a lot of junior programmers. And 10 years from now, the guys that are interns that are coming into AI straight, they're not going to know Scheme. They're not going to know how to, you know, write a Forth compiler, you know, in assembly language to bootstrap a C compiler so that the C compiler can write itself so that they can then develop a new CPU. That's a skill that they'll be able to just go to Google and say or go to AI and say, "Write me a Forth compiler for it." And that's fine. That's smelting iron.

And there will be a YouTube channel of one crazy guy out there going, "Let me show you the old ways of this. We had the ones, and we had the zeros, and that's all we had, and we liked it," right? And, you know, he's out in his blacksmith garage, you know, typing away on a Tandy TRS-80, you know? That'll be there. 

And the stuff that's not essential to our job, the smelting iron, is going to be a couple of people. There are still farriers. There are still people who heat iron and bend it around an anvil, and we need them for now [chuckles]. But there are other things that we don't need from, you know, we don't have leeches and [inaudible 37:59]

MIKE: But it's mostly for show. 

DAVE: It is a little [inaudible 38:01] yeah.

MIKE: Like, people, even 100 years ago, horses were a major part of the economy. 

DAVE: Yeah, that's true. That's true. 

MIKE: Today, they are something that you have because –-

DAVE: It's almost a luxury. 

MIKE: Yeah. It's a luxury. 

DAVE: Or you're in a very poor area, yeah. 

MIKE: Yeah, exactly. Like, in many economies, like United States, there are rare cases where you would actually need that.

Well, so my grandmother, when she was young, they didn't have an automobile. They used horses to get around. And now things have changed. Horses are very different because the technology has fundamentally shifted. Likewise, there are still zeros and ones. We have just built compilers that do that part better than we do. 

DAVE: Wow! Okay, sorry, just huge epiphany. Huge epiphany. So, I grew up in ranch country. I grew up down in southern Utah. I grew up in Moab before it was a famous place, and that was uranium mine out of town, and then everyone else was ranching. And the ranches were up in the mountains, and so a lot of the terrain was, you know, at 30 degrees. And a horse was the only four-wheel drive vehicle that could get, like, you couldn't get up there on a four-wheeler, you know, unless somebody built you a road. You needed a horse.

So, there were blacksmiths who made shoes for the horses, and the horses needed those shoes fitted. You don't buy a size nine wide. You have it fitted to the horse. So, that skill is still there, and those people are still there. But ranching is getting less and less common, and I just realized where the ranching is in software. And it's still there, and we're still seeing it, and it's in micro computing, micro devices.

You want to write your own keyboard controller? That's going to have a little teeny-tiny Arduino on it. Now, the Arduino that you kids are using that's called a Pi, and it's got a 1080p port, and you can put 32 gigs of RAM in it, and it's a computer. 

MIKE: [laughs]

DAVE: But the Arduino still exists. There's still a version, I think, of the Teensy USB out there that only has a megabyte of RAM. You can still buy a PIC chip, a programmable IC. This side of the millennium, I saw a guy program a video poker game on a PIC chip that only had 64 bytes of RAM.

Poker, there's 52 cards in the deck, so he's only got 12 bytes of RAM for your score, you know, how much money you have, how many cards are in your hand, what your current bet is, in eight bytes of RAM. And that's fun. It's delightful to do that. And 40 years ago, that's what Steve Wozniak was doing, and he was inventing the future in a garage with a soldering iron by figuring out how to lay those 8 bytes in just the right way and abuse them. And we only need these 6 bits of the byte, so I'm hiding some extra data up in here because I've got 5 of them. That's 10 extra bytes that I could be, you know, 10 extra bits that I could be using. There's wonder, and there's delight in that. And now, yeah, it's kind of ranching. 

But we're not going to launch a GeForce 4090 and a 10-pound PC into orbit and fire it at Jupiter if we can launch a 100-gram computing device that has enough compute to do everything that it needs to do and not be an AI platform or general computing, right? And anything you can do in a general purpose can be optimized down to a for-purpose thing, and that's where ranching is today. That's where the old skills are still necessary. And they're not old. They are still current, and they're still valid, and you have to be good at them.

If you watch the show Forged in Fire, those guys are bladesmiths, and, for them, pushing iron is very definitely an art. And watching them is actually beautiful and entertaining to do it. 

MIKE: My son actually does blacksmithing. 

DAVE: Oh, right on. 

MIKE: He makes knives and other implements. He doesn't have to. It's not a viable career path. They say a blacksmith can make anything but money, you know, like, except in, like, farriers, like you say. There's special cases where it's needed. 

Well, I think that's probably a good place to tie things up. We've talked about, you know, the need to stay grounded, how the underlying stuff doesn't go away. And knowing it is still useful because it gives you an edge that allows you to keep that connection. 

And if you miss that chance, explore that stuff a little bit, you know, try something at a low level. It'll change the way you think. And keep connected because you let go of the reins [chuckles]. Use some horse there, you know, talk there. It might go well for a little while, but eventually you'll be in trouble.

DAVE: Yeah, you're going to go where the horse wants. And you better hope the horse wants to go where you want to go. I like that. 

MIKE: Okay. Until next time on the Acima Development Podcast.