Working With Legacy Systems

About this Episode

This episode of the Acima Development Podcast delves into the challenges and strategies of working with legacy systems in software development. Mike begins by drawing a parallel between his past experience in construction and software engineering, emphasizing how businesses often opt to demolish and rebuild rather than work with existing structures. However, unlike construction, software developers frequently lack the luxury of starting fresh, as applications must remain operational while being updated. This leads to the episode’s central theme: how to effectively manage and modernize legacy systems without causing disruptions. The discussion also highlights how software often evolves in ways that diverge from its original vision, making maintenance and scalability a significant challenge.

The conversation explores various approaches to dealing with legacy systems, from full-scale rewrites to incremental updates. Matt likens outdated software to old pants—once a great fit but no longer suitable for current needs. The panel discusses past examples, such as the rise and fall of ColdFusion, and the complexities of working with outdated languages like COBOL and Fortran. They emphasize that while complete system rewrites are sometimes necessary, companies often benefit from gradually modernizing through a “strangler fig” approach—wrapping legacy components with new systems until they can be fully replaced. They also stress the importance of considering long-term viability when choosing programming languages and architectures, avoiding trends that might fade over time.

The episode concludes with insights into best practices for managing legacy codebases. The panel discusses strategies such as isolating components, minimizing dependencies, and maintaining clear APIs to improve maintainability. They also highlight the importance of version control, automated testing, and structured modernization plans to avoid the pitfalls of outdated software practices. The conversation touches on AI’s potential role in modernizing code, though they agree that current AI tools aren’t ready for fully automating legacy system overhauls. Ultimately, the team underscores the value of continuous improvement—whether through refactoring, gradual replacements, or thoughtful decision-making—to ensure software remains functional and adaptable over time.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Matt, Kyle, and Eddy. We may actually have some other people showing up later. We'll see [laughs]. We've got some delays out there, but, for now, we've got Matt, Kyle, and Eddy. And well, I'm not going to introduce the topic quite yet. Let me lead with some storytelling here, and I think we'll have some good storytelling in today's session.

Years ago, I did construction work, family business. My dad and his brothers they did building, so I did that some growing up, did it in my 20s, did a fair bit of construction work, among other things [laughs], but I did that. One place I worked was an indoor commercial construction, that is, we'd go into office buildings, sometimes high rises, sometimes smaller, you know, I've worked in banks, high rises, and in between [chuckles].

And we would go into spaces that were being leased by a new tenant and build it according to the new client’s specifications. The interesting thing about that is you have these...particularly, like, you have high-rise buildings. It's basically just concrete walls, right, and [chuckles] flat in between other than support beams to hold the whole thing up, and everything else was kind of open to the discretion of the client. And they did things in a lot of different ways.

And what usually happened, and this is interesting, is that it was cheaper for them to just tear the whole thing out rather than work with what was already there. You think, well, maybe, you know, you can keep some of the stuff. That's not what generally happened. The way that, you know, spent a lot of time just completely demolishing [chuckles] stuff that was already there. Actually, the demolishing didn't take that long, but, you know, a lot of time demolishing and then rebuilding from scratch, you know, dropping it down to the concrete.

And that's really interesting because you think it might save you to use the existing building, but that's not what we did. I can't think of a case where we, I mean, that's not entirely true. We did do...some cases we'd leave some stuff, but, you know, drop ceiling torn out, [chuckles] the walls torn out, carpet torn out, wallpaper torn off. A lot of times just tearing things apart before we rebuild it because it's hard; it is hard to work around something that's already there if it's not quite what you want.

You know, in engineering, we a lot of times don't get that luxury. We have to work with what's there because it's in use. It's very different. When the old tenant leaves, you got to strip things out and rebuild it. We don't get that. Like, I can't think of any time I got that luxury, right? You got an application in production. You're trying to make money, and maybe you're desperately trying to make money if you're running a startup. And you can't have downtime. You can't inconvenience your existing customers. You've got to build around what's already there, and what's already there may not fit at all what you want to be doing.

I've heard it described as starting by building a shed and ending up building...you're thinking you're going to build a high rise and ending up building an airport. It's just [laughs] the way that we build because the way business needs change just diverges over time, so we have to work around it. And that's the topic we'd like to talk about today.

We'd like to talk about what it involves working with legacy systems, which is what almost all of us are working in, you know, every system gets older over time. And this is a chance to share stories, and we'll be sharing, doing some of that, but also talking about the principles. What do you need to think about as you're building around things that are already there? Sometimes they fit, sometimes they don't, but you just got to live with that. That's my introduction. And I'm going to stop talking and see what my fellow panelists here think, just off the bat. Any thoughts about this topic? Do you have any stories? Do you have any ideas about what you need to do?

MATT: Well, it's like buying a pair of pants in the ‘90s, right? Here we are, and now 2025, Happy New Year, everyone. Something we bought in the ‘90s, great at the time, in style and fit. Now, here we are in 2024, many years later, gotten older, put on some weight, styles have changed. And those pants, while we still may be able to cover ourselves up with them, they just don't fit, and I think you run into that a lot with software.

Business rules change, direction changes, and teams change, right, code languages come in and out of style, popularity, hard to find support. So, when we're building something, we generally think, oh, what's new and shiny? This would be great to build this in. And anyone who's been in the industry for any lengthy amount of time has probably done this, and that's, oh, there's this hot, new language. Let's take, for instance, ColdFusion, right? I think that's a great example.

MIKE: That’s the one I was thinking of, by the way [laughs], the same one I was thinking of [laughs].

MATT: Yeah, at the time when it came out, a lot of people jumped on that bandwagon and thought, oh, this is going to be awesome, you know, it's Adobe. They're not going anywhere. It's going to be around forever. So, we build applications in it. Then, a few years later, come to find out, nobody knows the language. It's been abandoned, and here we are stuck with a system that's built in ColdFusion. So, what do you do when you need to update it?

MIKE: Well, that's the question, right?

MATT: Right.

MIKE: Assume you're a new tenant now. Throw out the old system, build a new one? If you had a ColdFusion system, what would you do?

MATT: Yeah, I mean, perfect example, we have this case here at Upbound. We have a system that we use to track our capitalizable time, initially written in ColdFusion. And Acima historically has not used that system, and we had a choice, right? Either we adopt it with it not meeting our needs and try and figure out how to adjust it to make our needs, or we build something new. Well, here, we chose to build something new in hopes that it will be able to scale and support our needs for the future, and that's not always the answer.

MIKE: Oh, it's not. There's something to be said that scale matters. And that's why really big systems tend to be built in really old languages. You want to find COBOL? Go look in the airline, right [laughs], or a bank. It's been there for decades, and it just needed to keep running. You're going to find old tech because it was cheaper to keep it running than to build something new, even decades later. Go ahead.

MATT: Yeah. Well, that's a very, very good point. You know, we also didn't really build things in the past in service-based architecture, right? We weren't using microservices. We were building monoliths. And the bigger the application, the more longevity it's had, the harder it is to be able to rewrite that system. So, maybe an answer is, hey, what can we extract out of here? Can we isolate domains, pull things out, and then start to modernize, that word modernization, then we can start to modernize that way, right? Okay, what's a new language that's popular, has had some longevity, so we know that it's going to be around, and how easy are resources to find for it? And then, you start to make decisions that way.

MIKE: You hit right on it. A lot of times, it's not cost-effective to rebuild the whole thing. But there may be a component in that system that, well, a new component, right, that's attached to it that you can build externally. Or there may be a component you want to rewrite wholesale, right? Just rebuild that component. You're not rebuilding the whole thing. You're just extracting that portion of it out to a service, and then you get a new service that wraps the older one. Have you ever seen a strangler fig tree?

MATT: Yes.

MIKE: They're cool, right? They’re interesting. Birds poop the fig seeds into the upper parts of the tree, and it will start growing way up high in the branches of the tree, send down a root down to the ground, and then root itself there to get the nutrients it needs. And that tree will gradually wrap around its host tree until it completely envelops it. And, eventually, the host tree is gone, and you're left only with the strangler fig. And you end up with this big, beautiful tree that is still there, and the original tree has, you know, completely vanished.

You can build software that way, right? You start with just a little piece, make sure it's well grounded, and build it out around the old section until the old section is no longer needed, and it's just gone.

MATT: Yeah, and it's important, too, when looking at these legacy systems, are we sticking with the same language? If so, that makes a big difference, right? It's a lot easier to refactor, make changes than it is to try and work a new language into an existing system, you know, some can be compiled into other languages, you know, if you're running in the JVM, et cetera. But I think that's a big component where you have to start to make these decisions is thinking, okay, what technology is going to be used for it, and what components can we build with this technology?

MIKE: Well, and there's some pressure there. If you're thinking about extracting components, and this is, I think, a healthy thing [chuckles], it provides some pressure to be decoupled because if you're really loosely coupled between systems, then you can completely change the tech stack on one of them, and it doesn't matter, right? They talk to the API. As long as they both talk to the same API, it doesn't matter. Now, this applies at all scales, even within your app, but between them, if you have really loose coupling, it works out great.

There's the famous email that was written by Jeff Bezos at Amazon (Was it 20 years ago?) where he said “Starting today, no applications get to have any internal private APIs to any other. Everything's all public. You can only communicate over these public APIs, and if you violate that rule, you're fired. Thank you [laughs].” And people were kind of panicked by it at first, but it forced teams to have strong APIs that were decoupled from each other, and that's how AWS was born. It matters [laughs].

MATT: It does matter. And it'll help you build things that will have that longevity, right? Just because something's old doesn't mean it's bad. Maybe your business rules haven't changed in 25 years, and you built this initial system in COBOL, but it does exactly what it needs to do. Why would you rebuild it? There's no point in redoing things just for the sake of redoing things.

MIKE: Right. As long as you are still able to get sufficient resources at a reasonable price to maintain the system. And if there's some core business rules that seldom, if ever, change, and, you know, COBOL [chuckles]...I laughed a little about COBOL. It was such a huge technology. There's still a pool of developers out there. They're kind of pricey, but they're available, right? You could get one if you needed to.

MATT: Yeah. One of my very first jobs in the industry was converting COBOL and Fortran over to PERL and PHP. Now, in hindsight, was that a great idea, eh, you know, probably not. Nobody wants to write in PHP or Perl anymore either. But it's interesting that it didn't really need to change. All we were doing was duplicating the exact same logic in a different language, and it really didn't make sense. You know, maybe they were trying to be a little bit forward-thinking in that those languages were going to be dead languages, but they could have lasted for another 10, 15 years and been just fine.

MIKE: Honestly [laughs], Fortran's still got a pretty active community. That's nowhere near a dead language. I haven't looked at language rankings recently, but Fortran might be ahead of Perl [laughs].

MATT: Yeah, you know, and for a while in the early 2000s, Perl was it. Everyone loved Perl, right? CGI scripts and its file handling was exceptional. You could get a lot done with very little code. It still, to this day, wins Code Golf almost every time.

MIKE: Well, and Perl is actively developed. It's not a dead language.

MATT: No, no, not at all. It has its place, and web applications just aren't it.

MIKE: But there's still a lot of shell scripts out there in Perl.

KYLE: I think there's something to be said, too, about some of these older languages. The programs that are written in those they were written in a different time with different constraints. So, we're talking about some of these COBOL or Fortran programs. Sometimes you don't want to get rid of them because they perform. We're not hindered by the same constraints that we were a few years ago, so we don't take into account that we only have half a gig of RAM, right?

And it may not be worth to switch it over to, you know, Ruby, Node, or whatever you're working with that will use two gigs of RAM because of how different the performance metric is. Maybe you need to write an adapter or another layer around that to take advantage of your old applications.

MATT: Yeah, I think, as engineers, we've gotten a little bit complacent in thinking about those types of things, like memory usage, disk space, because it's so inexpensive these days, right?

KYLE: Yeah, you can just throw hardware at it. That's a common solution these days, isn't it?

MATT: Yeah, absolutely. Just spin up another container.

MIKE: It's not necessarily the wrong one, either [laughs].

MATT: No.

MIKE: Hardware costs, while pricey, depending on the hardware, depending, you know, this is conditional, right? But, you know, they can often be less, even if they're somewhat pricey...developers are really pricey. And if you can write something in a language that is easy to think about and easy to communicate about and you're not going to write bad code, it may save you money, even if it uses a lot more resources.

MATT: Absolutely. And, you know, just in this short conversation we've had already, we’ve already listed, what, 10, 12 things that you really have to take into consideration when doing this.

MIKE: We've talked about this idea of wrapping a legacy application, gradually splitting out components, building them in. I'm going to use...you laughed a little bit at modernize [chuckles] because it's a loaded word. What's considered modern is not necessarily a clear concept. But what meets the current business needs, building in some framework that meets the current business needs that, you know, wraps the older one, gradually replacing components of it until, if necessary, the original is entirely replaced.

Sometimes though, we work within the system, right? You're not going to split out a component, but you're working inside legacy code. And maybe it's been there for a decade or more, and many hands have been in there changing different pieces. There's lots of different design styles, lots of different architectural ideas, all kind of crammed in together, like one of those old farmhouses you see that's had additions put on three or four times, and they all look completely different from the other ones [laughs]. It looks like that inside. What do you do when you find yourself in a system like that? How do you stay oriented and stay productive if you have to do all that context-switching between, okay, what's going on here?

MATT: I think that's where you really have to keep design patterns in mind, right, and see what you can isolate. Can you eliminate dependencies, if possible, and really confine your new code so it isn't dependent on other things? I think that's a really good way to start making your application better and more understandable and just make it easier to work in in the end.

MIKE: You said, if I heard you right, make it so it isn't dependent. Can you elaborate on that a little bit?

MATT: Yeah, when we're a little less experienced—and I'm not saying that every legacy system is written by inexperienced programmers, because that's far from the truth—but if, let's say, I don't know Ruby, right, and I just started writing in this language. Likely, if I'm new to this industry, I'm going to couple things together that probably shouldn't be. You know, I'm going to pass dependencies through all my methods, and Class A may rely on something from Class B, et cetera, right?

And I think that's where you need to think about, okay, how can I isolate this? Is this really dependent on this other thing and could I make a superclass? If they require resources to be shared, let's share them through a superclass or whatever, you know, just eliminate dependencies and try and decouple things as much as possible so that code is usable outside of its current form.

MIKE: Got it. So, you're talking about, you know, detangling the spaghetti, decoupling so you don't have mutual dependencies and, you know, single-purpose code [chuckles] that is reusable.

MATT: Exactly. You're much more eloquent than I at stating that, but yes.

MIKE: [laughs] You said it first, just rephrasing [laughs]. Yeah, absolutely. Improving your little corner of the world, right? And you said, specifically around best design principles, getting each component to be serving a single purpose and to have a clear boundary between itself and the rest of the code so that it's reusable and not, you know, like you say, mutually dependent on other code.

MATT: Yeah, and you also said cleaning up your little corner, right? If I have a really messy bedroom, you have to start somewhere, so let's clean the closet. If I have a nice, clean, organized closet, that's a really good start because it makes room. And then, as you start to get that organized, it makes it much easier to organize other things because you've established a pattern on how you're going to do it and what your approach is going to be.

MIKE: That's great, and it reminds me of something my wife does. She borrowed this idea from elsewhere, but it's a brilliant idea. Every week, you focus on just one area, so you don't have to do everything all at once. So, like, this week, we're going to work on making the kitchen clean because there's always the things, right, the things that you put off, maybe the deep cleaning of the stove, or [laughs] cleaning out the junk drawer, right? And if you try to think about all the things that got to get done, you're never going to get them all done. You just throw your hands up. And the tech debt piles up, and [chuckles] you never get to it at all.

But if you allocate some time on a regular basis and a rotating basis to each of the areas, right? And in the home, maybe this week we're all going to work in the kitchen. Next week, we're going to work in the living room, next week bedrooms, whatever the case may be. Like, in your code, maybe we're going to do some front-end cleanup this week or this sprint, right? Next sprint, we're going to do some cleanup in area X, or maybe it's a group of sprints, maybe a quarter, whatever it takes to get some real meaningful work done. That's not everything you're going to be doing, but it's a portion of what you're doing so that you're continuously paying down the tech debt and improving the code.

MATT: Yeah, what's that show? Marie Kondo, I think, her name is, tidying up and about organization. There are so many things that are applicable from life into software engineering and software design and vice versa.

MIKE: Right. Absolutely. Exactly. There is hugely relevant information there, right? That's why, usually, at the beginning of our podcast, I talk about something outside of engineering because getting ourselves out of the mindset of just what's right in front of us, I think, helps us to see those connections.

MATT: Yeah, for sure. Eddy and Ramses, you guys have been in the industry for a while now, but maybe a little less than us old folks. What do you think on this topic? Have you had the opportunity to really get into some legacy code and take a look and say, “Oh, man, what am I looking at, and why is this such a mess?”

RAMSES: Yeah, every day [laughter]. Well, quite a bit. I’ve worked in a few different projects, I guess, now, and some are more than 6 years old, 6 to 10, maybe even 12 now. But, yeah, when you get in those kind of deep pockets, it's, yeah, it's just, you know, no one's touched it for 10 years, and people are scared of touching [laughs] it. It’s brutal. Yeah, that's always just interesting. You just have to kind of, you know, take a step back, figure out what's going on. And, you know, depending on the severity or the complexity of your change, just try and maybe not poke it too much.

MATT: And I think, too, when companies start out, you know, startups, people have a tendency to get a little more cowboy. And when they build software, they're looking to be first to market, right? Let's just make it work, get an MVP out there so we can get to market, and then we'll come back and fix it later. And, you know, often that doesn't happen.

Also, you know, you have a lot of freedom in that kind of a world where you may have seen a podcast or listened to a podcast or seen a video somewhere, video tutorial on some new pattern that you've never tried. And you think, oh, I'm going to try it inside of our application, even though we don't do it anywhere else. And those are the kind of things, I think, when we come across, we just kind of scratch our head and say, why is this here, and why did we do that? That and if I look back at something I've written two, even three, years ago, if I don't look at it and think, what was I thinking? I'm lying to myself, right?

RAMSES: Oh yeah.

MATT: Because we grow every day, and we're always getting better. We're learning from our peers, you know, other ways of doing things. And it's just, there's always room to make things better.

MIKE: How do you navigate a large system that's, you know, got a variety of different styles and approaches? I've got some thoughts on this, you know, on how you approach that. You're in a codebase that you don't fully know, right, and in a large system, nobody does [laughs]. Nobody knows the whole thing. So, what do you do to quickly figure out where you're located? You know, it's like one of those shows where they drop you off without GPS and no obvious landmarks, and you have to find your way out [laughs]. How do you do that in the code?

MATT: I like to start from the outside in, if I can, you know, you find your entry point then it makes it really easy to navigate your way to where you need to be working, or at least find out if you're in the right place, right? If we're in the Ruby world, lots of Prys. I like to throw a Pry in once I get to a comfortable place, you know, maybe it's inside of a service object a couple of layers in, throw a Pry in, and then look at the methods inside of that service and run them inside of my Pry and console just to see if I'm getting the expected results. That's one of the ways that helps me navigate my way through new code. Run the tests, I mean, look at the tests. First thing, right?

MIKE: You know, you said find an entry point. I was thinking something similar. When you have something you need to work on and no context, you've got nothing, right? It's a blank wall. And finding that entry point is everything. So, if you've got an error, see if they've got a stack trace, right, because that can take you to someplace. Now you've found a place in the code. You've got that entry point where you can put your debugging statement, or your print statement, whatever kind of debugger you are, and start tracing things down.

If you have some UI, then you can go to that page and try submitting some things, right, put in some logging on the server side and see what's getting hit. If you're doing heavy front end, you can kind of follow a similar approach. Start with something that you do know because you've got to know something, right? You've got to know something. If you really know nothing, then you probably either don't have a sufficiently written request or you need to go talk to somebody [chuckles] to get them to give you some context. There's always someplace to start. Getting that starting point is more important than almost anything because once you can pry in, right, then you can start digging down and going further.

I find that new developers often get...they get paralyzed. They get paralyzed by all of the unknowns, and that's totally crippling. Instead, if you don't worry about all the unknowns and start by finding something that you do know, then you can start with that seed- and expand that knowledge outward until you know enough to solve the problem or build out the feature that you're looking at. That having an entry point, as you called it, is most of the effort in many cases, in my experience.

MATT: Yeah, and as long as you have...and maybe entry point isn't the right word, reference point, right?

MIKE: Reference point.

MATT: As long as you have a reference point, you can navigate in all directions out from there. If I know I should be working in some, let's say, user class, right? I can also look and see, what is calling that user class? What's instantiating it? What's using its methods? And work your way out from there, and then slowly, but surely, your circle gets bigger. And you start to learn your way around, and you're able to navigate things.

MIKE: You know, thinking about the geography reference I made before, you know, you’re dropped off. You don't know where you are. If you can find a landmark that you recognize, that changes your world [chuckles]. That changes so many things because now you have, again, you have that center that you can navigate out from and start growing that circle. Otherwise, the Cheshire Cat, if you don't know where you're going, it doesn't really matter which way you go [chuckles]. But if you know where you are and where you're going, it changes everything.

So, storytime, maybe for us older folks here especially, what are some times that, you know, some crazy stories that you’ve found in legacy systems? You know what? I'll lead out here. I'm going to start with a story. I'm thinking back to...I think it was my first full-time dev job, although it wasn't right away. I'd been there for a while. We were given a customer management system, and somebody in the company decided it'd be a great idea to buy this. And the guy who sold it to us was just, like, some person out of his garage.

And the whole thing was built inside a SQL Server as stored procedures. And when I say the whole thing, literally, the whole thing. There was, I think, a little bit of UI written in Microsoft Access that gave it a little bit of forms on the front. But then all of the business logic was written in stored procedures, which were stored inside the database that were not under version control.

There was no source control at all, were not searchable [laughs], and had no, you know, there was no rollback. You know, if you make a mistake, it's permanent, and it's live. There were also some calls out to compiled executables for which we didn't have the source code. [inaudible 31:31] here you go, make this work, and we had to make a number of changes to make it work for us [laughs]. It's hard to describe what it was like doing that [chuckles], you know, but --

MATT: That’s the thing nightmares are made of.

MIKE: [laughs] So, say, we make a feature request. Well, I'd start with the form in the UI and then figure out which stored procedure it was calling to, right? And then, in that stored procedure, it would usually call to four or five others, which themselves would call others. They were nested, so, you know, they were nested all over the place. And then, just step through those again.

It's not searchable, so you have to find its reference in the UI [chuckles] for a SQL Server and go...and this was a long time ago. It may be better now. But go find that other stored procedure, find what it linked to, and you could trace your way down eventually. Again, no debugging statements. That was not possible. You just had to try to understand it. And by looking long enough, you started to understand the geography [chuckles] of it and know what was needed.

MATT: Yeah, that, you know, we think today, what did we ever do without tests in our software? And then, you go back further, and no version control, no tests, multiple people working in the same code. And it was like, let's shove this file up via FTP, and then someone else is working in the same file and overrides your changes, and it breaks everything. Oh, man.

MIKE: [laughs]

MATT: Those were the days where you’d get really good at tailing logs.

MIKE: Yes. Or the system I'm describing where there is no test system. Any changes you make are live in production the moment you make them. You get really good at understanding what you're doing before you make a change. So, there's my story.

MATT: I don't know how we navigated that world.

MIKE: [laughs]

MATT: But somehow we survived it. And you're also in charge of keeping the servers up and the network and everything else at the same time.

MIKE: All at the same time [laughs]. Oh, I can tell stories of mistakes I've made.

MATT: I've brought down a number of production systems throughout the years [laughs].

MIKE: So, I gave my legacy system story. What's your legacy system story?

KYLE: What I'm thinking about right now is back when I was an intern for a company that I won't throw them under the bus. But I was an intern and then a full-time employee at one point. And the issue that we had there was we had multiple versions of their software to serve different customers, all written in different versions of Java. And so, what it was is you had different looks and UIs and different functionality bleeding through these different versions of the applications but also had the same functionality.

And my task at that company is I was a manual QA engineer, so I was tasked to test these. And so, we would be provided with specifications, but we had to know how this was implemented in every version of the application for each of these customers. Just how bad that was, like, just the test matrix on those types of systems was just insane just because I was writing bugs.

And half the time, I was right in the sense that, oh yes, it should work this way in this application, but not in this version of the application. And there was times where I misunderstood the requirements, and, no, it shouldn't work this way for this customer, but it should for this other customer. Choices like that where we're creating different versions and getting customers stuck on different versions like that that's one that really stands out to me.

MIKE: There used to be a popular way of doing things some years back, where every time you got a new client, you'd spin up a new version of the app, like, their own complete, hosted copy of the code, rather than having a shared web app. And oh, what a nightmare [laughs]. That is so bad.

KYLE: [inaudible 35:52]

MATT: Yeah. I worked in the lead gen industry for a long time, you know, spammers. And I worked for a place that did that with education. It was all of the for-profit lead gen education. And every time we got a new advertiser, we had to spin up a new instance of the application. But they would share, like, the cost per click, cost per acquisition, all of that.

So, if it changed when spinning up a new advertiser, we had to go in and change 50, 60 different independent applications for every partner every time we made a change, and no version control. It was all FTP. And this would have been, I don't know, I think, maybe 2001 somewhere about. It was just a nightmare. It was an utter nightmare to try and maintain that and keep it up to date.

The application itself was built in a hacked version of Mojave framework, so maybe it was a little later, 2003, 2004. Someone had torn apart Mojave framework and customized it to do this lead gen stuff. And it was just...it was miserable. Just the folder navigation alone was a nightmare.

MIKE: So, I’ve learned a couple of things from these stories. Lack of version control is...are there any systems out there without version control? I hope not. But if there are, if you're thinking about doing it, don't [laughs]. If you're starting in a new project, it's worth it from day one. Get it up in Git. It's worth it. There is no question. There shouldn't even be a hint of a question. If that's a question in your mind, you're doing it wrong [laughs].

MATT: First thing you do, git init in your local directory. It doesn't have to be hosted. You can do it locally if it's only a personal project but do it.

MIKE: Yes. Second lesson is, don't copy your code. Have a shared version. If you need to have customizations, build those in as some sort of feature flagging that you can turn on and off, and you have a configurable system. It's easy. Now you've got a configuration system that users can use to toggle, and it's great, and everybody is happy. Don't build a new system for every client. That is the path to madness.

MATT: And test your code.

MIKE: And test your code. Test your code. You know, things that are best practices sometimes you don't think about because they've just become so ingrained. Have you ever been in a situation where you don't have those? And, you know, a lot of students doing their coursework haven't done unit tests, right, maybe haven't done version control. Maybe just make another copy of their code when they need to make some changes. I mean, there are some things here that are easy to do when you're just getting started because it seems like it makes sense at the time.

There's a reason that these [chuckles] have become standards in the industry because they will save you. It's like autosave in a document. Back in the days when you could write up 30 pages and then the power goes out, and you lose it all, knowing that that would happen, autosave is the thing you do. You just don't write a document unless you have something persistently saving in the background, right? I mean, this is so universal now. It applies from command line editors up to everything, right? But there was a time when that didn't exist. There are some things that you just do from the beginning, and you just don't consider them negotiable because it matters that much.

MATT: Yeah, and it's a little off-topic, but it's just making me think about some of these horror stories. Looking back on the very first time I ever saw Subversion, [chuckles] and, you know, what was it before that, CVS?

MIKE: Yeah, concurrent version, what is the S for?

MATT: Which just stored the entire file, not just changes, and was a huge memory hog, but it worked. And then, one day, I was working for a place, and our manager came over and said, “Hey, as of today, we're going to start using Subversion.” And that's all we were told, no training or anything. Needless to say, about the first 20, 30 merges that happened always broke everything [laughter] because no one knew how to do merges, no one knew how to resolve conflicts, and it was just...it was chaos.

MIKE: Well, there's also something you said there about gradual transitions, meaningful, planned transitions, and not hard cutoffs, training.

MATT: Yeah, and training.

MIKE: Subversion was a great step forward. I loved Subversion back in the day.

MATT: Oh yeah.

MIKE: And then, Git was a great step forward beyond that. There's a reason why the industry...there are a few others out there, Mercurial, that some people use. But Git has just dominated. I can't remember the last time I saw a repo that wasn't in Git.

MATT: And now it's just second nature, right?

MIKE: Yeah.

RAMSES: Yeah, Git is awesome. It's great.

MATT: I wonder if this new AI boom will change the world of repository management.

MIKE: That's interesting. Totally off-topic [laughs], Nvidia recently came out with their new graphics cards, and they get, like, somewhere near 250 frames per minute, or something. And they do it, you talk about the AI, by predicting what the next few frames are going to be via AI. It takes the frames that you've seen in the past, predicts what's going to happen next, and pre-renders those frames, pre-renders those frames, and then just makes any tweaks as necessary, which seems wildly inefficient and crazy, but it makes a tremendous difference. It pushes things forward so much further.

To your point about...I bring this up because you think, well, with version control, what's AI going to do? Well, sometimes, if you have good enough AI, you can do surprising things that you'd never think was possible before.

MATT: And, you know, it's relevant to these legacy systems, right? A lot of these legacy systems have tasks that could be accomplished integrating some of these LLMs and, you know, even machine learning.

MIKE: Now, as of today, in early 2025, I would argue pretty strongly that you can't just take existing AI tools, point an old repo, and say, “Hey, convert this into something new.” It's not going to happen.

MATT: I second that. I've tried.

MIKE: [laughs]

MATT: Just, you know, like, fiddling around with it, seeing how intuitive it really is. I still would not trust it to write my code, though I do trust it to write base tests. I've been doing a lot of Python work lately, and Copilot is great for writing unit tests. You do have to validate them and make sure that it's testing exactly what you want, but it sure saves you a lot of time.

MIKE: Well, that kind of goes to where I see those tools being right now. Think about having an extremely fast but somewhat inexperienced and clumsy junior engineer, and they’re there at your disposal and can do a lot of things, and they don't mind any task you could throw at them. Big tasks, they're probably going to get wrong, and so you shouldn't. But something like writing unit tests, if you can knock those out, well, that's fantastic. That can save you a ton of time.

KYLE: I’m finding it very good for the idea of a rubber ducky. That's what I've been using AI for is it's a very intelligent rubber ducky.

MATT: Oh yeah.

MIKE: Having that rubber duck to talk to is super useful [laughs]. It's super useful. I think that we're kind of reaching a good stopping point here. We've talked about the fact that we're all working with legacy systems, almost all of us, right? Most people don't have the luxury of always working on something new. And the thing you're working on new is not going to be new forever. Eventually, it becomes a legacy system. And there's a variety of techniques you can take to deal with that. Wholesale rewrite, if it's small, might make a lot of sense. Usually, that's not something you have the luxury of doing, but you can split out components and rewrite them in a manner that is better aligned with your current business needs.

And if you have a system that you're just continuing to work in, you can find reference points within the code to navigate your way. And then, as you make changes, you can gradually improve. Pay down the tech debt. Take the time to pay down that tech debt so that your system stays maintainable or becomes more maintainable. Maybe it's not very maintainable, but it becomes more maintainable over time.

And there's some things we can learn from bad systems, too. Follow best practices. Follow best practices. They're there for a reason. Write unit tests. Use version control. Do the things that play well with others because one of those other people is future you, too [chuckles]. It's all worth doing.

Until next time, on the Acima Development Podcast.