The Evolution of DevOps

About this Episode

In this episode of the Acima Development Podcast, Mike, Kyle, and Will dive into the evolution of DevOps and its emerging subfields like Platform Engineering and CloudOps. They discuss how system administration has transitioned from hands-on server configuration to abstracted, automated infrastructure management. Kyle highlights the ongoing debate about whether these new titles represent genuine specialization or are merely rebrands of traditional DevOps roles. He explains that while the roles share significant overlap, Platform Engineers focus on building tools and workflows for engineers, while CloudOps specialists handle deployment and configuration in cloud environments. Despite these distinctions, Kyle acknowledges the fluidity of responsibilities within modern DevOps teams.

The conversation also explores the complexities of managing cloud environments, especially within Kubernetes ecosystems. Kyle explains how tools like CMOs and YAML configurations bridge the gap between engineering and cloud infrastructure, enabling smoother workflows. They touch on the growing abstraction in networking, including dynamic layers like sidecars and service meshes, which add flexibility but also significant complexity. Kyle emphasizes that while these advancements streamline deployments and observability, they also introduce challenges in maintaining clarity and efficiency across multi-cloud and multi-region infrastructures.

Finally, the team discusses the future of DevOps, including the increasing role of AI, automation, and multi-cloud expertise. Kyle predicts a continued shift towards higher levels of abstraction, with AI potentially optimizing auto-scaling and failover processes. However, he notes that AI struggles to keep up with the rapidly evolving DevOps landscape. Observability, cost efficiency, and auto-scaling remain top priorities, with multi-region deployments posing significant challenges. The episode concludes with a lighthearted reflection on the inevitable "closet of servers and box fans" lurking behind even the most advanced infrastructures—a reminder that despite the abstraction, physical and technical realities still anchor every deployment.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Kyle and Will. Today, we're going to lean heavily into Kyle. Kyle is from our DevOps team. Well, I say this...we were just talking in the pre-call that we say DevOps, but even DevOps is kind of falling out of favor as a term because there's platform engineering, other things. But 10, 15 years ago, there was no DevOps; it was system administrators.

And for those of us who were doing engineering at the time, a lot of times, it was just a developer given some system administration tasks, and [laughs] we’d get them done. I remember once I was driving to my...well, I was a passenger in the two-hour drive to my parents-in-law and banging out and configuring a server on the drive on a really weak mobile connection. And I got that done by the time we got there. And I was custom, just shelled straight into the server, configuring it as I went, based on stuff I’d done. And it was always that way. You shelled in; you set up the server yourself, and sometimes you went into the data center [laughs], and you brought in the hardware.

We live in a different world today, and there's a reason that people have done that. If you’ve got 50 servers that have all been hand configured by somebody, and then he decides to go somewhere else, or she, or they [laughs], even if you have a pool of people that have been doing that, you lose so much knowledge. It's just a horrible way to maintain things. Just like you don't want to have your code sitting on somebody's personal laptop, we use a version control system. All of the best practice from software have made it into system administration. Totally changed the world.

I'm talking too much because I'm not the expert. I'm just remembering how it was back then and knowing how much better it is right now. I don't even think about DevOps anymore, about the platform. It just happens. I merge [laughs] my code, and it ends up on production because of what Kyle and his team do. Kyle, how do you see that history of how we got from system administration? And Will was asking on the pre-call as well, is it just a rebrand, or [laughs] is there something really fundamental here that has changed in that process of deployment and infrastructure?

KYLE: So, this one's been a little bit interesting to me because I was talking to a coworker of mine that he went out to do some interviewing. And I asked him, “Hey, if you were to do a mock interview for DevOps, how would that go?” And he was just like, “Oh, well, I've done system admin-type interviews.” And he’s just like, “So, what's the difference?” And even some Google searching yielded, like, there's quite a few similarities. There's some differences, you know, maybe they manage the actual servers more, or they manage, you know, a system admin would manage more of, like, physical hardware more type thing. And DevOps is more of that CI flow, programming flow, and then the cloud infrastructure type scenario.

But then it kind of breaks down more because then you've got your specialties, observability being one of them, and each company has their opinion of what that should be. Should that be SRE? Like, here at Acima, we were the DevOps team, and we are still the DevOps team, but we have rebranded. There's half of my team that's now platform engineers and half that's CloudOps engineers. We're still doing the same skillset, but now we're called something else. I came in on the DevOps naming convention, so I don't have the full history there. And I don't fully understand the difference myself in a lot of them because there's so much crossover.

Even looking at new jobs or job postings at different companies, it's like, okay, well, that's a new name, you know, just CloudOps or cloud engineering. It's just like, oh, those are just DevOps tasks, and it's like, they're wanting a DevOps guy. Where's this new name coming from? But I think that's part of...part of the divide is coming in from trying to divide DevOps as a culture instead of team. We're trying to teach those key concepts, and it's dividing out, like, what a DevOps engineer is actually doing and giving them more of a title that's appropriate to that. So, if you've got somebody that's working on your platform all the time and designing your platform, maybe they're that platform engineer. If you’ve got somebody that's working in one of the cloud spaces, then maybe they're more of a CloudOps or a cloud engineer type person.

WILL: Okay, so, it's not the cloud. Like, your platform engineer and your cloud engineer they're different. Is it a situation where, like, okay, you got to have your own data center, or else you're a cloud engineer? Is that more/less it?

KYLE: So, I'll explain the difference on our team, which is we have a few of us where we actually are writing the code for the platform. So, like me, I'm sending out the observability infrastructure and designing how that flow is actually done. And the CloudOps individual is the one that's taking in the requests and configuring the alerts, configuring the dashboards to deliver to the requesting team. It’s platform would be, you know, your CI/CD Jenkins, setting that up, and then Ops also be setting up --

WILL: Can you be more specific? Like, it doesn't have to be everything. So, like, case study it through for me. Because these are just sort of broad platform technologies and you can cover a lot of ground, right? Can you be more specific about case studies and workflows, and like, this workflow goes here, and this workflow goes there? Does that make sense at all?

KYLE: I'm not quite sure what you're asking me.

WILL: Ah, okay. Ah, who knows --

KYLE: But we still cross. Regardless, I mean, it's one of those things where we're crossing each other's boundaries. And we call ourselves separate teams, but we're still one team. I guess that's where it's a little bit foggy for me is to, like, well, you're wanting a defined line, I think. And even on our team where we have the separate sub-teams, there's not a defined line of who does what.

WILL: Right.

MIKE: Is there, like, aside...do you have some people who are more dedicated to actually interacting with the clouds and a side that's more integrated with interacting with the software side? Because there's the cloud side, right, and then there's the software that goes, you know, you interact with the engineering side. And so, there's two interfaces, right? The cloud interface, the one side that you deal with, and the other side is dealing with the engineering. So, I could kind of see that you've got an API of sorts on either end, and you have people assigned to either end. Is it that sort of division?

KYLE: So, not quite in our setup. We’re doing the legwork. We're setting up the actual platform to be used, so setting up the standard workflow, maybe. Okay, so, like, our team, we've developed, like, a CMOs is a tool that we've wrapped several CLIs with. And in order to use that tool, we have a configuration YAML that will call different components in order to use that, so the platform people would have been the ones designing that, designing the tool, programming it out, and getting that functioning. And it would be the CloudOps guys that then take that from the engineers’ requests, and then they'll take what's been configured in that YAML file and generate the Terraform configuration that would then generate the cloud infrastructure from there.

MIKE: So, that sounds like somewhat like the division I was thinking I was describing, where one side is building the interface that the engineers use, and the other side is actually building the cloud side so that that can be deployed. It's not like there's a singular interface, but there's two ends of the interaction. There's the interaction with the engineering on the engineering side, then the interaction with the cloud [chuckles].

When I say the cloud, obviously, it's not a singular thing. You may have three or four different clouds that you're working with, all of which have slightly different services. And you have to make that work and use the tooling that works with that. So, you know, it requires a team to manage that and make people think about it as just a cloud. And likewise, you need to have a team that actually gives the engineers the interface. They don't have to know, right, they just know your interface.

KYLE: And that model is just what's working for us here. I'm aware of other companies where they have a DevOps team or a systems team that is specific to their Kubernetes environment. They're the only ones that touch that Kubernetes environment, and then they'll have a DevOps engineer on every actual development team that interfaces with that. But they don't have any permission to touch their Kubernetes environment, but they can configure different things.

So, it's kind of dependent on what each company is doing. And it is something, too, that goes through the industry in waves, too. I've seen it both in DevOps and in QA, where, you know, how much of it is a culture? And how much of it can developers themselves do, and how much of it do we actually need teams to do this work as well? And I think that's a size and stuff like that.

MIKE: So, you talked about a lot of companies, [inaudible 10:28] this as well, that have DevOps distributed among the teams. And you all have done some pretty amazing work by creating a tool that kind of replaces those people so that we interact using YAML files and the CMOs you mentioned, too. You know, rather than having to have an embedded DevOps person, you tried to build the tooling so that it's self-service. Kudos on your team for building such an easy-to-use system. But it doesn't mean that...most companies don't have that system yet, right?

KYLE: No.

MIKE: And, in that case, you almost need to have a person from the platform engineering, DevOps, whatever you want to call it, who interfaces with engineering so they know how to use this mysterious...this platform that you've built, it would seem like.

KYLE: We’ve even tried to do an ambassador program a few times to get engineering individuals from each of the teams up to speed on what we're doing so that that is a bit more seamless and easier for them to deal with.

WILL: Yeah, it is tough, man. The bigger the infrastructure gets, the more of a sort of, like, a dark art it is to actually spin up a box and put it onto the web, you know. I'm a [inaudible 11:42] system administrator. But at Acima, or the place that I currently am, like, get a box up there, get it on the load balancer, get an endpoint, assign it to all that. I don't know, man, whatever [inaudible 11:57] people's problem; that's it; I'm out, which is it's just such an odd...it's an odd place to be, if you really think about it. If you sort of came up just spinning up servers, plugging them in, let's roll, you know, like, wiring switches, I literally don't know how I would get a box up and an IP assigned, and get a route assigned to a thing. I'm like, I don't know, but you got to do that.

MIKE: Well, and usually when people are using a system...you've mentioned Kubernetes. Kubernetes is not for the faint of heart.

KYLE: No.

MIKE: [laughs]

WILL: Is it really bad?

KYLE: There's so much to Kubernetes. It's almost got to have its own degree, I feel like. It adds several layers of networking, like, just diagramming it out. We've had a team member that...it's the running joke because every time we add something to it, we have to diagram out the network ops because we're adding in another network ops just about every time we add a feature.

And you've got standard Kubernetes. You've got your gateway. You've got the Kubernetes internal networking. Then you'll have...if you add something like Linkerd, a mesh of any kind, you're going to have more networking layers, and it terminates here, and it terminates here. But outside of Kubernetes, you're also terminating your TLS, or you're doing internal TLS. That's just talking, like, I'm just throwing out buzzwords, I guess, at this point for referencing how large it is.

WILL: [inaudible 13:39]

KYLE: Huh?

WILL: I don't [inaudible 13:41] transport layer security matters.

KYLE: But it's its own ecosystem. And then, on top of it being its own ecosystem, you've got to figure out how to run your servers and/or your code in that ecosystem. And then, with Docker, it's been made easier by, you know, it doesn't really matter your host too terribly much. You can just run an AWS host. You can just run an Ubuntu host, whatever you want to do. But then there's works and different things between any of the languages that you're running, too, as much as you want to think it's just straightforward, oh, I can spin up a Ruby app, and it's going to act just like a Java app, well, no, there's several differences. And is it going to act like a Haskell app? No. Haskell is a world different. So, configuring all of them in Kubernetes it's a lot of investigation and learning just to get it all working together.

WILL: I always felt like that was one of the more interesting aspects of going back to the primordial ooze of system administration. It's like, system administration in the battle days you were administering a Linux or, God help you, a Unix, or a Solaris server. And Google wasn't a thing; Stack Overflow was not a thing. You just kind of had to know. It was like a London cab driver where you just had to drive [laughter] the car around as an apprentice for a couple of years until you just knew.

And I feel like a substantial portion of that is still big thing, right? Where it's like, okay, Kubernetes, that's fine. But if I have to set up a Rails app, and I have to set up a Java app, and, oh, my authentication headers need to look a little bit different here or a little bit different there, and I have to get certificates added to the Java runtime, right, these are just things that I know because I've burned my fingers on the hot stove before.

There is, like, I don't know, like I said, like a level of...because you're responsible for every platform's thing they didn't get quite right, or things that they just kind of zigged when everybody else zagged. And you have to make sure no, no, smooth it all out. There's a level of just witchcraft that's just attached to it.

MIKE: If you want to have your brain bent, explain a network sidecar [laughs].

WILL: A network sidecar. I don't know what a network sidecar is, but, like, hit me.

MIKE: [laughs] I could try. Kyle talked about layers of networking. You can take your normal network stack. You can decide, hey, you know what, I'm going to add something onto my packets. And it's just magically going to work in this infrastructure because I'm going to add a new protocol on top of my networking that I'm going to run between my servers in order to allow me to do monitoring, or security, or you name it. You are augmenting the network protocol by adding a layer on top of it, and you can do that dynamically and use plugins for it. Did I capture that, Kyle?

KYLE: Yeah. That's kind of similar to what that Linkerd that I brought up is doing, kind of your meshing. They've got an injection pod, and if you have the right label on your pod, it will inject a sidecar on your pod to do the communication between pods. And it will terminate before it gets to your app, so that you have that layer taken off of your communication, just in time to communicate with each other then you communicate back. It's putting it back on there until it's the next sidecar.

WILL: That makes sense to me. I don't know; there was nothing about that was all that crazy. It's like a little middleware, where it’s like, oh yeah, I'm going to take a packet. I'm going to wrap it with this other stuff, this other thing. And at the server level, at the server layer, or, I don't know, the switch layer, whatever, okay, all right, that makes sense.

MIKE: Which is cool. But going back to, like you said, the primordial ooze days, you had a network stack, and that's what you got. And now we're talking about a network stack that is dynamic, that is configurable, and that you could plug stuff into. Let's say I wanted to add some fancy security protocol to the communication between servers. Nobody has to know anything about it. You just inject it into this mesh, and it just magically works, which is, of course, sometimes it doesn't work. You just might [laughs] have a whole team that administers it.

KYLE: Well, now it's getting a little bit crazier, too. Speaking of how deep does your networking going, we haven't yet to explore it here at Acima, but there's a new tech, and I'm not sure if there's others, but it's called a vcluster, where you can do virtualized Kubernetes clusters inside of your Kubernetes cluster, which is an idea for similar environments, development environments, QA test environments. Those type things are good use cases for that, but they're even touting it for some production-type loads and stuff, too.

And it's like, how far down do you go? How many layers of networking and systems do you go? It's getting to where we're trying to configure, I’m trying to think of what it's called, what Vagrant and Docker is, but, you know, one button push an entire environment. You know, it can configure an entire Kubernetes environment and, virtualize it and put that in your Kubernetes cluster. Like [laughs], I don't know, it's going crazy.

MIKE: Yo, dawg, I hear you like networking. [laughter] That's interesting. So, you mentioned virtualization and advanced networking tools. Would you say that virtualization is part of what has driven some of the fundamental changes we've had in infrastructure?

KYLE: Yeah, it's created that...just back in the day when I was in QA, you always wanted to verify the binary. That was one of the first things that you always tested. You wanted to verify the binary between the one you were testing and what was being delivered to make sure the same thing was going out, the same thing hadn't been manipulated at all. Virtualization and being able to package that virtualization specifically is making it to where we can deliver entire environments rather than just single packages. And we're just expounding on that, right? Just expanding on how much we can deliver, how much we can keep in commonality because now we can build an entire OS with an application running on it, and that's our binary.

It's a docker image, which is kind of what I'm referring to at this point, but now you don't have to...we spend so much time configuring an environment kind of like Will brought up earlier. It was back in the day that you babysat that. That was your baby. But anymore, it's you build that environment once with the application that you need. That’s your deliverable. Now you're delivering that.

And we're trying to get to the point where we're delivering entire platforms, entire systems. It's just moving faster and faster. And I think that's part of, yeah, like, why the title keeps changing, why our expertise keeps changing is we're doing something different because virtualization and dockerization, those type of things, have allowed us to pivot so much faster.

MIKE: So, it's not just trying to put a few, you know, use better resources by putting a few servers on a single physical server. Now it's completely changed the way you think about deployment, is what I'm hearing. It's nested.

KYLE: Yeah, it's very nested, yeah.

MIKE: So, you've got...there’s not just an advantage in resource utilization, although there is that. It's an advantage in being able to configure your...all the way to the OS level, right? And even which libraries you're including on your OS to be a deliverable. So, you don't have the dependency hell that people have run into forever because it's just configured as part of the binary that gets deployed. So, all of your components, all the way down to the operating system level are specified, and it goes together as a bundle. You remove a lot of the questions there, so it's not...although this does end up using resources more effectively.

You get all this crazy configuration. You say, well, then you just take it into layers, have an entire system of...if you got a hundred services, it's what I'm hearing, is you can configure all of that, and it's all just configured. And you could deploy that in multiple places if needed, so if you want to spin up an ad hoc development environment, for example.

KYLE: Yeah, and it's going deeper and deeper. You said, more effectively, using resources. To some extent, we're getting to a point where you can, but you don't have to run on dedicated hosts. I'll speak to the one I'm aware of is, you know, there's Fargate and serverless options that will run Docker. And in that, as long as your app fits into the constraint of their limits, it'll run it. And you don't even need to manage your hosts anymore. We do over here at Acima still, you know, in some of our production workloads and development workloads. But you don't have to even manage your host at all if you don't want to, and that's where we're at these days.

WILL: Yeah, but how much of that is, like, how do I put it? Like, how much of that is, like, a business model looking for consumer demand? It's just, like, it's a little...you don't even want to have a web host, right? Like, for some stuff, for some specific use cases, okay. But it seems like --

MIKE: It’s a level of abstraction. Once you get away from the idea of thinking, I have to manage this physical server, it changes the way you think about the problem. I'm thinking about back when people wrote, well, before even Assembly language, right? People punched cards, or they had the machine language. And you had to just map those codes over to something that you were going to put in as numbers. It doesn't allow you to think about algorithms in the same way as if you add another layer of abstraction on top of it. And, at first, people thought, well, I can do that better. I can write my own Assembly, and I can do a better loop than you can, and, at first, that was true. At first, that was true.

WILL: Oh, it's true now, right now.

MIKE: I saw an interview with Bjarne Strusup, the guy who made C++, said, if you tried to carefully craft Assembly to do better than the compiler does for a modern compiler for C or C++, they will almost always do worse.

WILL: Oh, no way, dude. I've done it myself, personally, hand-rolled Assembly. It's easy to build. I was a compiler researcher for a while.

MIKE: Okay, well, you were a compiler researcher [laughs] [crosstalk 25:59] compiler researchers.

WILL: Well, you got to know Assembly, right? But, yeah, you can beat the compiler most of the time.

MIKE: I think that for small circumstances, I'm sure that's true. But the larger you get, the more complex the system, the harder that gets to beat because it just takes too much time.

WILL: Well, exactly. It's never worth it, right? And the compiler is pretty good. And if you know how a compiler works and you know how your hardware works...and those two things, especially for a modern operating system, modern processor, modern memory management, you got to know some stuff. But yeah, a compiler is easily beatable. You should have that knowledge. And you care, right? Because you need a CPU-bound hot loop. Yeah, they're very beatable, but it's usually just...it's almost never worth it, right? Because, like, how many million times do you have to run that thing to get to win, right?

MIKE: And I had to do that for a class once years ago, you know, beat the computer by doing some loop unrolling. I know that you can do it, but it's, again, the amount of time you spend on that is almost certainly not worth it because think about the --

WILL: Oh, never [crosstalk 27:18] flow, like, database, bytes on the wire, network downstream calls. End of list.

MIKE: [laughs] There you go. You need to optimize something else. And where I was going with all this is that thinking at that higher level of abstraction. Because the more you have to deal with those low-level details, the less you're thinking about, well, maybe I can add an index to my database [chuckles]. You want to be thinking at a different level of abstraction.

So, I think that what Kyle's describing here is just thinking about the problem from a different layer. Now, at first, it's going to seem like, well, that doesn't make any sense to me. But I think that 20 years from now, people will wonder how you ever thought differently because they will invent stuff that we haven't thought about yet.

WILL: I'm just thinking about, like, how I’d do it if I wanted to virtualize my server. And, basically, what it would be is I would just have to have some sort of routing table thing live, that's like, oh, if I go to this request, if I go to this thing, and then I would have to basically have a web server that I awaken from memory, and then it would do the thing, and it would go back away. So, to serve my request, I would need to go through all the process. It would just be like, I [inaudible 28:46] my infrastructure, right?

And I was thinking like, okay, well, where does that make sense, right? Big stuff that's not used very often, very infrequently used servers, like, places where you don't have a performance thing. But, in the end, it smells, to me, like a pod in cold storage that you could boot up again relatively quickly.

KYLE: They have the different types now. So, you can have warm hosts, warm, like, instances. The boot time isn't nearly as bad.

WILL: Well, exactly. But you have a warm host, but, like, a warm host, but not a hot host, a warm host. But it’s like, okay, what is that? I just took the memory, the memory image, and I flushed it to my quick SSD, so on top of my virtual memory ecosystem. And I was just off to the races, like, weird, you know, it's just odd. It's not like you're not put on ice in one of these big cloud data centers, your virtual host, like, your virtual EC2 instance. But that thing hasn't gotten a hit in a couple of weeks. You really think it's like...it’s warm, maybe not simmering, maybe not [inaudible 30:18].

[laughter]

KYLE: It’s not ready to take the Black Friday load.

WILL: I'm just saying.

MIKE: It's above absolute zero [laughs].

WILL: You'll never catch him. You’ll never catch him paging you out. You'll never, like, there'll be no smoking gun, and you won't be able to subpoena anybody. You'll just be like, hey, I paid for that thing to be up all the time, and it wasn't. It was just warm. And they’re like, you'll never [inaudible 30:47].

[laughter]

MIKE: You're saying the same thing that I was saying a minute ago is that once you've got to that level of abstraction, you don't have to think about all the details that you just said, waking up from memory. And those details are interesting. They're all interesting on their own terms, within that layer. And the engineer who wants to deploy to it they don't want to know about all that. They just want to merge their code and see it go to production.

WILL: Right. Yeah. I publish my code. It makes it to the CI. I get a little object, my little binary that goes in. It gets integrated in the pod, pod goes into the rotation with the load balancer, we do our...what do you call it? Green/blue deployment. It shifts all the old ones out, all the new ones are in, and then, boom, I'm locked. I'm in business. Easy-peasy.

MIKE: [laughs]

KYLE: Easy-peasy.

WILL: Easy-peasy, lemon squeezy.

MIKE: [laughs] If the engineers have to think about it, then DevOps still has some more work to do, right? Where do you see DevOps going in the next few years, Kyle?

KYLE: I've kind of been trying to think about that and, like, how does AI come into play? And then, we're really kind of headed towards this platform engineering type role; at least, that's what I've been seeing and what we end up delivering. And I think we're going to see more systems-based deliverables. I think we're going to continue to grow that out. And I think the next thing is, like, I think you're seeing right now a lot of specialization in the sense of clouds and which cloud, you know, you’re AWS certified; you’re GCP certified, that type of thing.

I would expect that in the next few years, it's not going to matter. They're just going to be one of the tools that gets used in your multi-cloud rotation, and you're going to need to know all of them. You're going to need to know how to use all of them, how to have them interacting and networking between all of them, and how to have either Kubernetes or whatever else it is, how all of that's going to play together between the multiple cloud environments. Because right now, we're kind of eking into multi-region, and [inaudible 33:19] understand that there's teams that are good at multi-region and stuff like that.

I think it's going to go cloud-based, and then how much of that is handled by AI and stuff like that I'm not sure. I'm not sure how much is going to assist us in our world. I've seen hit-and-misses from my perspective as some of the tooling...AI is good at stuff that's known, I guess, is what I'm getting at. And the DevOps world is moving so rapidly that by the time it's documented, it's outdated is what I've referred to it as.

And so, even some of the stuff that, like, I'll ask an AI about, it's like, I have no idea what this tech is called. I've never seen this tech. And it's like, so it can't help me with it. I'm having to use the resources online, other resources online to do that. But yeah, so that's a very long-winded way of me saying I have guesses, but I don't really know. But I see it going towards we're managing at a higher and higher level as we move on.

MIKE: Well, AI, like you said, known things. You can definitely teach a system to understand things like load. So, you could...one thing that machine learning can do, and it doesn't necessarily have to be that sophisticated machine learning, is do things like auto-scaling or failovers. And that seems like something that could definitely be automated and probably better than what we're doing today.

KYLE: Probably, yeah. Like you're saying, it's just a system that's able to take in all the metrics, all the logs, all traces, all the observability information, and is able to make decisions based upon what's coming in and scale up your environment. Yeah, I can see that happening if we get AI to a point where it's able to turn over like that.

WILL: I'd say what AI is great for, writing Nginx config files.

MIKE: [laughs]

WILL: I'm so bad at Nginx config files. I was so bad at that for so long, and I would always get it done eventually, but, like, [inaudible 35:36]

KYLE: What I’ve found AI to be the best, for me, is I hate writing RegEx expressions. [laughter] Just throw it into the AI. It gets me...I'm like, thank you. This is the best tool for that.

[laughter]

WILL: Oh, dude, yeah. And it's so bad, too, because sometimes if you switch languages around, like, everyone they get it, like, just a little bit different, and I’m like, I don't need to know. Just what's the space, man? What’s the space? Backslashes, percentages, whatever, just what is it? Nginx config files, though, man.

MIKE: I'm [inaudible 36:19] ngix config files. And you got, like, some plugins and stuff in there, and you got a...oh man. You configure your security certificate [laughs], some dark magic there [laughs].

WILL: So, let me ask you this, right, so, like, okay, like, DevOps, right? To the degree you can, right? What do you see, like, the business has a need, like, an unmet need from DevOps? What are you guys thinking about implementing? What's the need that you wish you could fulfill or maybe you're getting pressure to, like, can you deliver this thing for me?

KYLE: Like, specifically right now, what are we getting pressure on?

WILL: Well, I mean, to the degree you can. Like, what's on your mind in terms of they really want this, and either I don't know how to do it, or I'm trying to free up resources to do it, or something like that. Like, where are you seeing like, okay, I'm getting heat; the next step for me is this thing?

KYLE: Me specifically, it would be observability, just because that's where I'm focusing in on, and so cutting costs on logging and then the other metrics and traces in that atmosphere. But the company as a whole, auto-scaling, being able to react to load. Pod horizontal auto-scaling is something that we're really looking towards getting nailed down and how we go about that. Do we go about that in a standard load fashion? Do we respond to load? Do we respond to our Ruby apps? Do we respond to Puma threads? And do we have differentiators between our different languages on what we're responding to for that?

Then another one is RDS, Postgres specifically. How do we scale that? How do we make that multi-region? How do we have that single point of data? And then, how do we have that available in multiple regions for DR purposes or multi-region access? And then, on top of that...I feel like I just went brain down on what my last point was [laughs], but yeah.

WILL: I mean, it seems like efficiency, right, and cost management, like, that’s, like, [inaudible 38:47] everybody talks about auto-scaling. What they're talking about is they're talking about efficiency, right?

KYLE: Yes.

WILL: If I need 20 boxes, let's get 20 boxes. But if I only need 10, let's keep it at 10. Man, multi-region, that is such a bear of a problem. Can I ask, there are some people who need multi-region for sure. But, in general, really? Multi-region, really? If California slides into the ocean, we got to stay up, and it's a reasonable expectation because there, you really need to start thinking about is this juice worth the squeeze.

Because, on one hand, like, yeah, okay, you've got a tremendously much more complicated system. But you also, like, you're introducing all kinds of permanent all the time performance taxes, all the time, because those things got to sync up. And they have to, you know what I mean, keeping the East Coast and the West Coast on the same page, that's a long hop. That's a long...

KYLE: And you're saying that, and one of the things that we do keep in mind, or not necessarily battle with, but we do keep in mind and we monitor, is that replication latency because it's a real thing. You're replicating clear across several states, several time zones, and it's like, it’s there. Your data is not available right away. They call it real-time data, but it's not. There's a latency sink to that. And then, we kind of think of the data as a single source, but we're replicating the data in several data lakes even. And then, do you do that multi-region, too? Do you need a multi-region data lake, which is basically a copy of your data anyways, at least for us? And how do you keep that available?

I kind of tend to agree with you on some of your point, which is, is the cost worth it? Do we really concern ourselves with it? Because it's been a running joke on my team, you know, California, Oregon, if something happens to them, is being able to fail over to a new region our biggest concern? There's going to be so much else that's down. Just because we have our stuff in Ohio, Oregon blew up. What the heck? Is everything going to work still?

MIKE: You don't have to have so much an explosion as some contractor with a [inaudible 41:27] who didn't put up the flags [laughs].

WILL: Yeah, I remember back when AWS was pretty new. It was not unheard of, but, like, once a year, US-East-1? Not today, boys. Not today.

MIKE: [laughs]

WILL: Check back after lunch. We'll see what we got for ya. And Amazon never went down, but [inaudible 41:54]

[laughter]

KYLE: Should have done multi-region Amazon [inaudible 41:57]

WILL: Nope.

MIKE: [laughs]

WILL: I shouldn’t have gone US-East-1. I'm just saying.

MIKE: [laughs]. It's always US-East-1.

WILL: It's always US-East-1. Always. Always. It's always the one, man. Ohio is fine. Portland is nice. Virginia? I don't think so. NSA bots or whatever, I don't know. I blame the CIA. It's got to be their fault. But something's happening.

MIKE: [laughs] So, we've talked a lot about the amazing complexity that goes on in DevOps world today: layers, managing, well, you talked about observability. And it comes down to better management, right, better management of complex systems. And, clearly, the industry feels it's important because we've got a whole field, right, that's dedicated to managing these complex systems and trying to optimize cost efficiency is what we're saying. Is there any final things you'd like to say about DevOps? Say, you know, “This is what you should know. This has become a thing. You should know this.”

KYLE: I would just say I was sitting here thinking we focused a lot on where everything lands and how that's being managed. A huge part of DevOps is the entire CI/CD flow. And that on its own is enough of another podcast, but yeah, that's something else that we need to keep in mind when considering DevOps, for sure, is we need to know about how to build everything through that CI/CD flow and validate it and get it out because there's different pipelines. There's different approval processes, and that's its own beast.

WILL: Oh, man. So, I'm working on mobile apps, right? So, mobile app builds have to go through a macOS machine, and we've got a closet full of Mac Studios screaming, just screaming [laughter] for mercy all day and all night. And it's a closet in the office, and if that closet somebody kicks the plug out of the wall, everything stops. And people will drive down there in the middle of the night, where it's like, ah, I got to go. It's ridiculous, truly ridiculous, but it's got to be done.

MIKE: And all of our layers of abstraction are pushed away, and we see reality [laughs], the closet.

WILL: Yeah, the closet. The closet is just a bunch of wire shelves, like, a box fan, I think [laughter]. I think literally a box fan. They put a box fan on the ground and then a box fan out. So, it’s shooting up, and then shooting out [laughter]. God help you if one of those fans goes out, too, because they'll start throttling [laughter]. And this is a multi-billion-dollar enterprise running off a Costco box fan, lying on the ground [laughter]. It's not on the ground. I think it's on a milk crate for airflow, but anyway.

[laughter]

KYLE: To be fair, that ecosystem is not as easy to build on, and it is very expensive to rent virtual machines on. So, the idea that you guys have machines in a closet like that it's probably pretty common.

WILL: Oh, absolutely. It's a pain in the ass. So stupid. But Daddy Apple wants what Daddy Apple wants. It's what it is.

MIKE: Which is probably a good conclusion here. We like the abstraction. We don't want to have to think about that closet. We want to not think about that closet. We want to think about that closet as little as possible. And so, we have Kyle [laughter].

KYLE: [laughs] And so, we have Kyle.

MIKE: Army of fellow DevOps people who take care of that for us.

WILL: There's always a closet.

MIKE: [laughs] It exists.

WILL: [inaudible 46:05] closet. Maybe it's a virtual closet.

MIKE: [laughs] But it exists.

WILL: There’s always a closet.

MIKE: [laughs]

WILL: And a bunch of blinking lights and box fans.

MIKE: Thank you for listening. Hopefully, you got some glimpse as to the value that DevOps provides and how that whole field has evolved. And until next time on the Acima Development Podcast.