Infrastructure Tools

About this Episode

In this episode of the Acima Development Podcast, the panel dives into the complexities and consequences of choosing the right tools for software development and engineering. Justin opens with a story from his first job, where his team implemented an Oracle Customer Management System that ultimately led to disaster. The system, despite promises of extensive customizability and robust functionality, failed spectacularly due to performance issues, particularly with search speed, which took minutes per query. This failure, which ended in a costly rollback to the previous system, highlighted significant lessons about the importance of realistic testing, validating vendor claims independently, and having fallback plans for major tooling changes.

As the discussion progresses, the team shares additional insights on the risks associated with both large, established tech tools and smaller, emerging ones. They debate the pros and cons of "buy vs. build," generally advocating for buying established solutions rather than building in-house, particularly when these tools fall outside the company’s core value proposition. However, they recognize that established tools, while more reliable, often become stagnant, and smaller, innovative companies may offer better solutions that come with risks, such as sustainability and long-term support. A balance is often difficult to strike, and each choice comes with potential downsides, whether it's cost, dependency, or eventual obsolescence.

The episode closes on a broader note about the inherent "bitter lesson" of technology: that simple algorithms powered by ample resources often outperform intricate, specialized solutions. This leads to a candid reflection on open-source contributions, corporate influence, and the evolving tech landscape, where many advancements are the result of unsustainable VC-backed ventures. The hosts humorously conclude with the advice that any tooling decision likely involves trade-offs and inevitable challenges, encouraging listeners to accept the imperfect nature of these choices.

Transcript:

EDDY: Hello and welcome to Acimas Dev Podcast. Today, we have an awesome crowd, right? We have Mike; we have Kyle; we have Justin, and Melina, right? And today we decided upon popular opinion, right? That we'd be talking about tooling, right? Should you be married to a tool, pros and cons of switching to a new tool, when to do it, why to do it. And Justin actually was talking off-air on having a horror story behind something that sort of segues into this. So, Justin, by all means.

JUSTIN: All right. So, what I have is a story from my first real programming job, and it was the decision to move to Oracle. Boy, I don't even know what it was called, but it was an Oracle customer management system, the CMS. And, if you're a programmer, sometime in your career, you will either write or rewrite a customer management system. That is, like, a given. I was lucky enough to do it or unlucky enough to do it on my first job. And the decision was made at levels far above me that we were going to do Oracle’s customer management system, which was, you know, Oracle a huge company, huge corporation, and it had a tool for everything and, especially, it had a tool for managing customers that we had never used before.

And the thing about it was is, like, they advertised it as, hey, you can use this tool, and you can customize it any way you like it because we have a field for everything, and you can add your own custom fields. And every field could be validated, and every field could be a search, and every customer can have any number of objects associated with the customer. And so, it was everything to everybody. And the VP in charge was like, oh, this is going to be awesome.

And we were a medical company that did medical tests. And we had customers in the numbers of tens of thousands, if not pushing a hundred thousand. So, the order of magnitude was actually not that big. But as we went along, we got given a deadline saying, “Hey, we are releasing this day, do or die.” And by gosh, we did it. We released that day. And come to find out the next day that nobody could use it. It wasn't a login issue; rather, it was a performance issue. Every single search took upwards of five minutes.

And it was just a horrible, horrible experience. And it slowed our customer relationship team down to a crawl. Phone calls couldn't be completed. Doctors couldn't update their customer fields and things like that. And for the next, I think it was week and a half, we tried to fix it. And then, the CEO came in and said, “Screw it. You guys are moving back to the old way.”

And, ultimately, we ended up throwing that away. And it was a, I don't know, I think it was on the order of a 5 million project, which some people think that's huge. Some people think that's not that huge. But to us, it was months and months of work and licensing fees and data. And it's actually probably over a year of work, come to think of it.

But the key there was that it was an environment that nobody really knew how to do, an infrastructure that was new to folks. I mean, we'd done Oracle databases before, but, all of a sudden, we were asked to develop within the Oracle framework and using their tools, and it just sucked. And not only that, but our test environment was certainly not populated with a large enough data set to validate everything. And I believe that the fallout from that was the...I think that VP got fired. The rest of us just, you know, kept on going. And so, that's, like, the tale as old as time is, you know, some middle manager or upper middle manager gets canned because their project was a horrible implementation.

EDDY: So, what's the underlying problem here? Was it lack of training? Is it the fact that you're developing in Java, or is it just Oracle in general [laughs]?

JUSTIN: I don't know. There's probably plenty of blame to go around, right? But the lessons that I learned was the fact that you cannot depend on salespeople for the numbers that they give you. You have to prove it out yourself. And then, you have to test with as real production data as possible, and that includes complexity. That includes number of rows. That includes number of transactions, all of that.

And you actually should have a rollout period where you have a beta with production data for X number of time because it basically shut the company down for a week and a half while we tried to make it work because they didn't have a fallback. And, luckily, they were able to fall back. They were able to roll back to the old way after a week and a half, but it was just...it just sucked.

And, also, from the developer point of view, using the Oracle tools...because we had to use their packages. We had to use their architecture. Previously, we were just using their database as a data store. But now we were using their libraries. We were using their SDK a little bit. We were using their, you know, all these things, and it just sucked. Long story short, Oracle sucks, and use Postgres [laughs]. Even back then, Postgres was a thing, so...

EDDY: I'm just surprised the company was willing to go a whole week with production being down until they finally decided, oh, you know what, cut the losses [laughs]. Let's go back to...

JUSTIN: It's funny because different companies have different tolerances. This was a medical company. And so, when I say the whole company was shut down, I mean just, like, the customer side. We were still doing tests and things like that. And so, there was a certain amount of tolerance there for a couple of days. But the CEO was just like, you know, after a little while, he just couldn't take it anymore.

KYLE: So, with that, were you privy to why they concluded on Oracle at all? And was there any competition why that tooling was selected?

JUSTIN: That was all so far above me.

KYLE: Far above you.

JUSTIN: I was fresh out of college, but, man, it was a trial by fire. And just to follow up that a little bit, we were a development team of about 20. Over the course of the following year, I think they lost 15 of the engineers just because, you know, there were other problems with that organization, technology-wise. That was kind of the same time I left as well.

MIKE: Yeah, I can say it's not always the big players. In my first job [laughs], we switched to a new customer management system, and it was built by, like, some guy in his garage. And it really was just somebody [laughs], I think, the CEO knew. And the entire thing, the entire system, was built around stored procedures in SQL Server because [laughs] they provide functionality to call out to externally compiled binaries, which, of course, you don't have the source code for that he had built. And there's no source code control, right? Because it's all just implemented straight inside the database. And there's no searching. You just have to know which store procedure to look at or look through them one by one until you find what you're looking for. And that was the entire system.

And we bought it, and then, of course, the person who we bought it from wasn't available for assistance. So, it was on us to manage it ourselves. And we kept that tool [laughs] for as long as I was at that company, meaning I learned how [laughs] to navigate through every bit of that code in order to make it work.

There's a lesson here about how you provision tools at a company, I think. As a company, whoever you are, you're probably not a customer management system company. And if you're putting all of your efforts into building that in-house, you're diluting your expertise in what you’re probably supposed to specialize in, whether it be medical products, or some sort of medical services, or financial services, or whatever the case may be. You are focused on somewhere outside your area of core expertise, and that often goes badly. I am, in general, in favor of buying the tools that do the right thing, that do the right job. But there's some caveats in there, right? It's the right tools. You go through some processes that involve people actually using it.

WILL: Yeah. I mean, you should never, I don’t know, you should never invest in a tool outside, in my opinion, I don't think, without an overwhelming reason, you should never invest in any kind of tooling that's outside your sort of core value proposition past a certain scale. Because there's people who've done stuff where Facebook basically rewrote PHP, which is ludicrous for anybody but Facebook.

EDDY: They built their own framework, too, right? Like, it's insane.

WILL: Yeah, yeah, absolutely. And building their own framework, I think that one's, ah, you know, like, how do I put it? I think that proposition is dubious. I think React might be stunting a little bit, which is because if they invested that amount of resources into one of the existing things and just fixed the problems that they had with it, would they really get a worse result? I bet not. But, eh, whatever, you know, probably [inaudible 10:31], you know, like, they have their thing, and it's really deep, and they did good, and they’re like, React Native is actually really dope these days, you know what I mean? For, like, cross-platform, mobile, like, that's it. That's the game. And they actually did it, and, like, it kind of sucks for web now, but that's fine. Anyway, that's a tangent. Yeah, but don't invest in tooling outside of your core value proposition.

MIKE: You're saying buy, not build
if it's –

WILL: Always. Always. Always. Always. Always. Always buy, not build. Never. As an engineer, just...engineering's the worst. Just don't do it.

JUSTIN: [laughs]

EDDY: So, you're basically saying don't build my own AWS system, right?

WILL: Is that a sub-tweet? Like, if you're discussing it, like [laughter], entirely, like, I'll be the hatchet man. They can't fire me. Literally can't. So, like, no [laughter].

MIKE: One nice thing about not being employed at Acima is you can speak your mind [laughter].

WILL: You know what, actually, you know what I mean? Like, an engineer, one of the core engineering mindsets is everything is a trade-off, right? Everything's a trade-off. And I'll say this, man, because another...it's not a requirement to be an engineer, but they [inaudible 11:52]. I'm a miserable cheap bastard, and AWS is expensive. These people have been tightening the screws pretty hard. I've been seeing some bills, and I've been like, how much does a rack cost these days [laughter], you know? Like, racks aren’t that bad. It's not so bad. I could plug cables.

KYLE: Well, and a lot of the other cloud providers are up and coming, right? And they're kind of cheaper at the moment. So, yes, you get a lot of the large support from the Azures, the Googles, the Amazons. But there's others that, depending on what you're doing, they're completely viable.

WILL: I mean, large support. I've never been large enough to get large support, so I don't give a ****.

KYLE: Exactly.

WILL: But I also, like, my experience with AWS is twisting their arm to fix something they broke in the first place. I was like, oh, man, us-east-1 is down again. I better email Amazon so they can tell me to go stuff it. I'm like, all right, cool. There it is, us-east-1 down again. What are we going to do about it? We'll get back to you. Cool. Thanks for the support.

KYLE: And that kind of brings up a good point. You're relying on a tool to provide you a service, and you're running into a situation where you're like, hey, your product’s down in a certain zone. What are they recommending? Well, become Multi-AZ, then. Become multi-region. Buy more of our product. Use more of our services. And it's like, where do you define the line? And it's, okay, do you go into more regions? Do you invest more in their product? Or do you go multi-cloud, or do you go hybrid, or, you know, like, how do you solve that? Because is the right answer going multi-region and putting more money in their pockets at that point?

EDDY: I think that --

WILL: Is the answer, like, is the answer, like, how many cancellations am I going to get really? Because, like, okay, we have to always be up. We have to always be up. And I'm like, actually, no. If you can't get a TV today, you'll be okay. You're going to be all right. It’s okay.

EDDY: So, can you ever consider --

WILL: But can you check that cap spreadsheet, you know what I mean, before I go on vacation? It's like, well, maybe you hit it from the plane. Like, a lot of this stuff isn't actually a life-or-death situation. And to the extent that it doesn't cause churn or cancellations or bottom-line stuff, you know, like, I know we're not allowed to say it out loud, and it's not the Will's boss podcast. So, again, I could just say what I want, but I mean, like, eh, [inaudible 14:52].

JUSTIN: It totally depends on what company you're working for. Because I worked at a FinTech firm that, the downtime was measured in millions per minute, and nobody was going to die, but it was literally millions per minute.

WILL: I don't know, man. FinTech and downtime goes together like peanut butter and jelly [laughs].

JUSTIN: No, this --

WILL: I have worked with a financial system before. Y'all people [laughs].

JUSTIN: Yeah, no, this...I’m trying to [crosstalk 15:28]

WILL: ...Trading type stuff?

JUSTIN: Yes. Yes.

EDDY: I mean, I'd argue that part of the conversation that happens when switching tooling is cost, right?

KYLE: Yeah, big time.

EDDY: So, if at one point the tooling that you're using becomes way too expensive, either to maintain or to license, I think conversations will be had about switching to a different system, right? I always advocate to having a fallback, right? So, if you're using Amazon exclusively for your hosting service, if one of them goes down, like, one of the regions goes down, like, you're done. And so, conversations need to be had about what do you do in that scenario? Do you go to a different provider that's not Amazon? And if you do, what's the cost? What's the risks? I think that's something that needs to...

KYLE: It's a conversation of expertise, too, because a lot of the time the company will invest not only in a cloud, but it will invest in engineers that are specialized in that cloud. And so, then, if you're going from Amazon to Google, well, now you got a bunch of Amazon cloud engineers. And what does it cost then to switch to Google Cloud? And that's sometimes a lot more costly than just eating the increased billing cost.

WILL: And also, man, in all honesty, it's like a parallel algorithm. You might have to rewrite your whole thing to be like, oh yeah, that data center is down. It's like, oh, I'll just use the other one. Okay. Whoa! Whoa! Whoa! This database went out. What happened to all the transactions? And is that cool? That was...whoa! Whoa! Me and Mike spent, what, a year reading a book on that, and it was heavy duty, man [laughter].

JUSTIN: So, I think there are a couple of things that you need to look at when you are deciding which architecture you want to be. And I'm glad you brought up that Google Cloud versus AWS. If you can write your infrastructure as code, which, you know, you guys are, and we are, and most people are these days, you can be provider agnostic to a certain extent and be able to, you know, the migration from AWS to Google Cloud or to Azure would hopefully be less painful because you've got everything defined in your Terraform or, hopefully, using Terraform, maybe not CloudFront or whatever.

But as you go along, I think that is the key. You design your systems to be implementation agnostic. And thus you can, if something better comes along, you can make the switch as painlessly as possible. That's not to say that's easy, but I think that's the ideal.

WILL: The eternal lie of cross-platform development [laughter], hmm, [inaudible 18:50] so much to do that.

JUSTIN: Hey, I was a Flutter developer for four years.

WILL: Oh, I'm so sorry --

JUSTIN: And it actually worked out. It worked out. iOS and Android, it can happen [laughs].

WILL: Sure.

EDDY: Just use Kotlin.

KYLE: Just use Kotlin [laughs]?

JUSTIN: Kotlin multi-platform isn't there yet, but it's getting better. Anyway...

WILL: Nor will it be because iOS is actually, like, in the back end is actually...iOS is really stingy on the RAM. They don't tell you how much RAM an iPhone has, and that's not because the numbers are too good.

JUSTIN: No, no [laughs], they aren't. iPhones suck that way [laughter].

WILL: They really do. Well, I mean, you could say the iPhone sucks, or you could say the JVM. JVM, come on, buddy.

JUSTIN: Oh, wow.

WILL: Do you really need that much?

EDDY: Which is why having open-source software is really important.

WILL: Yes, absolutely. The operating system should be open source. No operating system in the world ever, ever, ever should be closed source. It shouldn't be allowed. It should not be allowed. But I don't think that is podcast for the day [laughter].

JUSTIN: No, but you look at the couple of other things that you have going on there in terms of what is running your actual system. And if you have everything dockerized, you have everything set up as infrastructure as code, you can have it all be pretty provider agnostic because you could run your Kubernetes on AWS. You can run that on Google Cloud. You can run that on Microsoft Azure. And the principles are mostly the same theoretically on this specific thing. I don't know.

WILL: No, you're right. You're right. I was just busting. I was just busting balls. I mean, in the end, right? You need to have...in the end, it's different flavors of containerizing Linux, right? Except for Azure and those people. Whatever, man. There's a lot of .NET developers in the world, and they technically are people, too. Legally, they're humans.

JUSTIN: They have high therapist bills.

WILL: Yeah, yeah, Kubernetes and Docker and all that stuff, they can flavorize with different flavors of Linux pretty well. We can do cross-platform as long as we heavily water down what the notion of platform is, where it's like, hey, you can run both kinds of Linux, Alpine, no problem. You know what? You can even have Ubuntu. Suit yourself [laughter].

MIKE: You're hitting on a key point there. If you are using a pretty...and it goes back to what you said before about the operating system. If you use something that's standard underneath, and you're doing something that's really standard, that is, spinning up virtual machines within a cloud that provides compute, right? Then that is pretty commoditized. That's straightforward. But what if you want to do functions like AWS Lambda?

JUSTIN: Lambdas?

MIKE: Exactly. Is that going to exactly map from platform to platform? And the answer --

JUSTIN: Nah, you shouldn't do...I've been burned too much by Lambdas [laughs]. I don't recommend them anymore.

MIKE: I'm not speaking for or against. My argument is [laughs] that once you diverge from doing something that is extremely well-proven and straightforward on a standard platform, then it starts to get platform-specific, and then the costs start to skyrocket because then you have to...then you have to split, right? Now you're not really doing the same thing anymore, and it goes back to the cross-platform. Yeah, you can do the basics pretty straightforward, and then, all of the cost comes into everything that goes beyond the basics. And if you have to make it work across both sides, it's probably not going to.

JUSTIN: So, is that the lesson you should take? Like, if you are straying from the path, that should be a gigantic code smell or architecture smell.  And you should really analyze your requirements to see, are they really forcing me to go that way?

WILL: My takeaway personally is the eternal lie of cross-platform development because, inevitably, developers are going to...it's like a game to them. They’re like animals. They can't help themselves.

JUSTIN: [laughs]

WILL: It’s a compulsion. It’s a sickness, a disease. They're going to find the edge of the map, every time. They're driven towards it like conquistadors. I don't know what pushes them, but they're always going to do it. And they're always going to find the edge of the map. And they're going to write some ****. And it can't go down. And then, you start again over and over.

JUSTIN: Haskell. Haskell.

WILL: You know what I mean? Some kind of thing, man.

MIKE: Let me throw a metaphor out here. You're going to get married [laughter].

WILL: Oh, man. Oh, it’s getting hot. You’ve been trying to [inaudible 24:32].

MIKE: You keep your finances separate. You carefully schedule times so that neither of you have any disruption to your schedule with your old friends.

JUSTIN: [laughs] Why are you getting together again?

MIKE: That's exactly where I'm going. Were you ever married? There's a tricky line here that sometimes if you're going to go to a provider, you have to accept that you're going with that provider. If you're going to go AWS, you're going to go Google Cloud. You're going to go Azure. There are some things you can probably do cross-platform, and there's other things that you probably can't. You've got to make a choice. Go back to the marriage [chuckles]. You've got to make a choice. If you're hedging and you haven't really committed yourself, it's probably not going to work out very well. I don't know. Thoughts? It's a metaphor. I don't know.

WILL: Where does Larry Ellison as a divorced lawyer, like, fit into this equation? I'm trying to make the joke, and it's just not coming to me.

KYLE: So, I'm thinking about this because I'm more of an Android fan and to the ecosystem talk here and, you know, considering that tooling. I have always looked at Apple and said, I don't want to be tied to an ecosystem. I don't want to be in that. I would prefer Android. Android allowed me that, like, what, 5, 10 years ago. Now you're either in the Google system. You're in the Samsung.

Like, at some point, you do have to pick a marriage. It's not really, like, with all of our tooling, it feels like you're being pigeonholed into some type of a marriage, and you have to kind of accept it, at least I have, I feel like. Because if I want my watch to work with my phone, I got to pick a brand. I got to pick something. And what do you go with, and where do you settle? And yeah.

WILL: I think that's maybe more of an interesting question than the eternal lie of cross-platform development, right? Like, how do you pick who you're going with? Because you are right. I'll say this straight up. Me, personally, in terms of back-end infrastructure, I'm a Linux ride-or-die. Whatever kind of server crap they've got, forget it. I'm not doing it. I won't. I won't do it. I also jumped on Apple, which I have misgivings about because I jumped on Apple, right?

Maybe this is an illustrative example. I jumped on Apple because I was an indie app developer. And Android people don't pay anything, right? So, okay, so my choice was clear, right? I wanted to eat. And so, to have money to purchase food, I was going to have to do Apple instead of Android. And so, then I was stuck, and, philosophically, yeah, they rub me the wrong way. I believe in, like I said, I was rather militant in saying that all operating systems should be open source, without exception. I don't think anybody should have a closed-source operating system, and Apple ain't that way. And so, what’s a good way of choosing? Like, how do you pick the pony, you know, that you're going to ride?

JUSTIN: So, I'd like to give you guys an example of what we're facing right now, and I'll pick your brains a little bit. Three advisors. So, we are trying to decide between JFrog Artifactory and CodeArtifact. JFrog, the big industry name, CodeArtifact, not so big, but still, they get the job done. We need some place to park our builds, and be available, and we can validate them and everything, right? And we can have them part of our build process, and we can do dependent libraries and all this and that. And so, there are a lot of questions about which way we should go. JFrog, obviously, is the big industry one, but they want 150,000 a year.

WILL: Woooo.

JUSTIN: Right? But all the tools work with it.

WILL: I want to say two things, like, one, as a permanent JFrog user, JFrog sucks, and I hate them.

JUSTIN: [laughs] I'm going to put that down on my con list.

WILL: Yes. Two, this is really, like, meta, and it doesn't have a lot to do with engineering. It has, I think, more to do with finance and the rise and fall of empires. So, for me, I believe that every company kind of follows the same arc. Some people have a longer arc with better management. Some people have a shorter arc for various things, like, not going well, but every company eventually calcifies, you know what I mean? And then, like, the sort of the MBA managers take over, or a hedge fund buys them, or they're trying to make their quarterly results, and things like engineering get squeezed to death. And then, their feature set, and their agility, and stuff like that locks in place.

Now, what's the upside to that? The upside to that is they're the standard, and everybody has to meet the standard. And they have to be the thing. They suck. They're going to continue to suck. They're going to get worse every year, forever, until they collapse because they've locked it, so the concrete is set. This is what you get. And every year, it will get worse.

So, what's the other side? You got these new, up-and-coming companies. And they're hungry, and they're modern, and they're building things better because the second time you build anything, you do a better job, right? What are the dangers there? The danger there is they don't make it, right? The danger there isn't support and tooling around them. The danger there is they, in turn, do not get to make their numbers for their IPO, and they get bought by private equity, and they fire everybody. And then, they are in turn setting concrete, right? And then, they start to, you know, their inevitable slow degradation. And so, you're sort of, like, jumping on a wave, right?

The new up and comer they're going to do a better job because they can avoid all the landmines that JFrog hit, right? They exist because they didn't screw up the same way that Artifactory did. If Artifactory did it right, these guys they'd have no sunlight. Like, they wouldn't have anything. So, the question becomes, well, all right, are they big enough to be viable at the scale you're at? And that is a question that I can't answer because I got to sign JFrog. And I think about this stuff the absolute bare minimum that I can, other than cursing JFrog's name when they do me, which happens quarterly.

KYLE: That is something that we did go through, very similar story. Nexus is what we were using, and CodeArtifact was something that was on the table as well as JFrog. And we ultimately landed on the tooling that had the best support for our packages and had probably the best support for infrastructure as code, which happened to be JFrog. So, that's kind of what it came down to for us. What it comes down to for you guys, I don't know. I mean, we are a very vast, what, language base here at Acima, so we had to support a lot.

MIKE: Will got down deep there, talking about the innovator's dilemma. Big companies get stagnant and then they get...an innovator comes in, and the big founding company laughs at them like, well, they don't offer any of the features we do [laughs]. But they're cheap, and they work well for the people who want to use them. And, surprisingly, there's a tipping point, and, eventually, the old company becomes obsolete. You know, there’s Standard Oil [laughs], you know, a lot of the railroad companies, you know, you think about it, they were huge. Enormous. And --

WILL: Standard Oil still exists, baby. They changed their nameplate, but they're still around. I forget which one they are. Like, are they Exxon? Are they...I think they're Exxon.

JUSTIN: Going back to the tech world, you look at the rise of Wiz, like, specifically in the security space, Wiz was nothing until...I think the company was founded in, like, 2018, something like that. And it was a company that had to rapidly and cheaply connect to and identify all the processes that could exist on a cloud infrastructure, you know, multiple different kinds of services, multiple different versions, multiple different data stores, and everything else, and be able to present that in a great format in a cheap way. And they had to compete with the other scene companies or the existing security monitoring companies.

But, basically, this, what, six-year-old company turned down a 20-billion-dollar buyout offer from Google because they thought that they were worth more than that, so 6 years, 20-plus billion dollars. But it's a classic example of somebody, you know, coming in and just wiping the floor with existing competitors because turns out that those existing companies, they had a large feature set. But they didn't, you know, probably most of the people didn't use half their features that they had. The industry moved on. And Wiz was able to, you know, rather than try to maintain these old feature sets, Wiz was able to come in and build exactly what people wanted and do it in a cheap manner that attracted all of the industry folks to switch security monitoring systems.

And, you know, it’s nuts in some ways. But I think it's a perfect example of somebody coming in and creating exactly what people need at a price point they need, and people recognizing for them, and yeah, so good on Wiz. And I feel bad for all the other companies in the space because they took a haircut, massive haircut.

WILL: Yeah. And when you’re betting on sort of, like, startups and stuff like that, like, you're rolling the dice, right? I mean, you are rolling the dice to a large degree, right? I read a very interesting article about OpenAI and how they actually might be screwed. I don't know, you know, like, generative AI is not my deal, really. Like, I've used a bunch of it, right? I’m playing with all the toys as they come off the assembly line.

But, basically, what they were saying is the model stuff has achieved an asymptotic level of goodness, right? And Facebook is just giving their stuff away. They're pulling a page out of the standard big company playbook and saying, like, we'll just give it away for free because we make money on ads, but we could starve you out, and you'll just die, and then we win, which, okay, right?

And so, generating the large language models that's asymptotically valueless, and everything is going to go around tooling and shaping the data, in such a way that it can be consumed by these models. It was an interesting concept but more broadly to what we're talking about, that you're betting on this startup, this new idea, right? And so, what are you going to bet on? I know I have some clients who are operating on OpenAI models. And I'll be like, “Hey [laughter], how much are you paying OpenAI? Is Llama 3 good enough? Because if Llama 3 is good enough, I bet you'd save some money paying me to swap you over because OpenAI ain't cheap, you know?”

JUSTIN: So, I have a great example of that. Last month...it’s a Utah Java users’ group. They invite people to come present every month. A company CEO founder from a company called...what was it? Qualiti.ai. They do automated QA for companies. And, basically, they automatically create tests that will test your product end to end, and they will maintain themselves basically. And they spent...prior to, what was it, 2 years ago, they spent tons and tons of resources developing their own models. And all their resources went in to train their own models and everything, and that was the number one expense.

And then, and it was like the thing that made them valuable for these other companies, their clients were these models that could go and create these things. And then, OpenAI comes in and, specifically, ChatGPT. And they came in with this general model, and with the general model, it wiped their...all the stuff that they made was useless because it was, like, next generation or two or three generations beyond what they were doing.

And this guy, the CEO, looked at it, and he was like, “I've just spent the last six years of my life trying to make this work, and I made a mediocre product compared to what OpenAI was.” So, he had to make the really, really painful decision to examine what his company actually did. And what his company did was not train models; it was provide a way for companies to create automatic QA.

So, once he recognized that his value add wasn't the models, rather it was giving people these automatic tests, then he made the decision to throw away all the training that he did. And he basically contracted with OpenAI, and all the automated QA things are now just API calls into OpenAI. And he's doing his best to make that third-party agnostic. But it's just like, OpenAI was, like, head and shoulders above all the other competitors for his use case. So, it was just like...it goes back to, you know, what are you trying to do as an engineering org? And recognizing the value that you add and getting the best tool for the job when you're doing buy versus build.

WILL: Right? Never build anything that isn't directly related to your core value add. Are you a computer scientist, you know what I mean, convolutional neural networks research company, and you're the best in the world at that? Or are you a QA company that uses these neural networks? Like, the neural network stuff, like, that's heavy-duty, man. Like, we're still in, you know, very, very early days with all that tooling and building all that stuff and stuff like that.

Like, I was trying to get...and I'll tell you how I know it worked in early days because I was trying to get one of those image generation models running on my local computer. It was a real pain in the ass, and all the documentation sucked. When I find myself on a forum, like, if I find myself on a forum page looking for something, I'm like, oof, all right, look at me, Oregon trailing this thing.

JUSTIN: [laughs] You're going to die of dysentery, man.

WILL: Wouldn't be the first time.

MIKE: Have you all read the...It's by Richard Sutton. He published something years ago called The Bitter Lesson. He had learned that a number of companies will spend years building this custom machine learning model to match their use case. And they'll come up with all these tricks and do everything just perfectly to eek out just a little bit more performance. And another company comes along that uses just a dumb algorithm. And they throw a bunch of compute and data at it, and it works better. Then [laughs] they were able to accomplish all these years of work. That's the bitter lesson. Then, in the end, a simple algorithm and a ton of compute and data win every time.

WILL: Yeah, yeah. No doubt. No doubt. No arguments there. I think it's another topical tech in the news today. It's amazing how many amazing open-source projects, and products, and platforms, and such are really just the result of a VC-funded company blowing their budget and imploding in an altruistic manner [laughter]. It’s WordPress. WordPress, right?

JUSTIN: WordPress.

WILL: Automatic, right? Automatic. Invented WordPress. No. No. WordPress is a VC failure. They built this beautiful code temple, and then they just abandoned it. And good, old Matt he rolls in, inherits the whole thing, plants a flag. That's WordPress, baby, open source. But if you don't want to host your own thing, you can pay me, you know. WordPress was not...he didn't invent WordPress. Like, Ruby on Rails is actually one of very few, you know what I mean? Look at good, old Java. How's Sun doing? Not so good.

JUSTIN: Oracle's doing fine.

MIKE: [laughs]

WILL: Oracle's doing...yeah, Oracle's doing fine because, well, yeah, anyway. Yeah, Oracle's lawyers are doing fine.

JUSTIN: True story.

WILL: I actually think Oracle, the technology company, is not so great, to be perfectly honest with you. I know there are Oracle databases in the world, and I know your bill goes up every year. But I don't know how many new Oracle databases are being spawned in the wild anymore.

JUSTIN: No, everybody --

WILL: I don't know how many people have paid for that Java license to be perfectly blunt.

JUSTIN: Yeah, OpenJDK.

KYLE: Something you kind of glazed over there, in the open-source community, too; it's kind of amazing how many of the end products by the larger companies are just wrappers of open-source products. Just for instance, today, I was looking at upgrading Packer only to find out that the Azure Image Builder is just a wrapper of the Packer software. That's like, so [laughs], I mean, everybody uses these smaller pieces, you know, in the open-source community, whether or not it's them hosting it themselves and having you pay for it, or just a tool that they provide you as, you know, a paid service.

WILL: There you go.

MIKE: We've talked a lot about tooling, and there really hasn't been one conclusion but a lot of different threads [chuckles]. There's not easy answers here. But sometimes you got to make a choice, and you might get it wrong. But you got to consider those trade-offs.

WILL: Yeah, I feel like this is our least conclusive one to date. Honestly, you know what I mean, like, so, remember, kids, any choice you make is probably wrong and sucks, no matter what [laughter].

KYLE: If it doesn't suck now, it will later?

WILL: Happy Friday!

[laughter]

MIKE: It's a bitter lesson, but an important one [laughs].

WILL: Thanks, comrades. All right, well, I'll go get started on the weekend drinking [laughter].

MIKE: Until next time.