Scary Code

About this Episode

On this episode of the Acima Development podcast, the crew leans into a Halloween “code horror” theme, using real-world stories to explore the scariest things they’ve seen in software. Mike opens with a literal horror from homeownership: a drain pipe so clogged with roots it had become a pipe-shaped root sculpture, a perfect metaphor for an ancient 3,000+ line Rails controller crammed with overlapping workflows, unclear entry points, and so much tangled logic that a junior engineer couldn’t ship a single change in months. That sets the stage for an episode about legacy systems, complexity, and the way neglected codebases slowly turn into haunted houses you’re afraid to enter.

From there, the hosts trade progressively more terrifying war stories from their careers. Dave recalls a nightmare installer project where he had to write Pascal inside InstallShield to find and kill a running Rails process on Windows using SQL against the process table—an absurdly convoluted solution that nevertheless shipped and worked. Justin shares high-stakes fintech deployments where downtime cost millions per minute, quarterly release windows created brutal pressure, and failed rollouts meant rolling back three months of work. Kyle talks about discovering hard-coded secrets and shared keys scattered across repos, unrotated for years, then being told to fix and rotate them all “by end of day” with almost no historical context. Mike adds tales of a reporting system literally built on SQL injection, “fixed” by an enormous hand-rolled SQL builder that was later thrown away, and a credit card gateway acquisition where an injection flaw had already been exploited to steal over a million dollars.

The horror then zooms out to systemic and operational failures: clickstream data sold by ad blockers that can easily de-anonymize users, HIPAA-reportable incidents that nearly trigger federal oversight, and outsourcing critical code to poorly vetted contractors only to see entire codebases appear on the dark web. They dig into how floating point differences between systems can change financial reports by a few dollars, how time zones and users changing their device clocks can break “simple” expiry logic, and how massive vulnerability scans can surface tens of thousands of “critical” issues across hundreds of repos. Add in AWS us-east-1 outages that turn disaster recovery plans into live-fire drills, layoffs that leave one engineer alone with a mysterious legacy system, and useless commit messages like “test” repeated 50 times, and you get a grim but funny campfire circle for engineers. They close by emphasizing the real moral behind the scares: every one of these stories carries a lesson about security, architecture, and process, so listeners can learn from others’ hot-stove moments instead of burning themselves the same way.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. With me, I have Justin, Dave.

DAVE: Save yourselves.

MIKE: And Kyle. And --

DAVE: I just realized this is not going to come out on Halloween.

MIKE: [inaudible 00:37]

DAVE: We are recording this on Halloween.

MIKE: Exactly [laughter]. Well, that's what I was about to say.

DAVE: Oh, sorry.

MIKE: You're going to probably hear this in January, but we're recording this on Halloween [chuckles], Halloween of 2025, and so we're in a spooky mood [chuckles] with the Halloween theme. We're going to run with that, and, as usual, I wanted to connect this at least a little bit to something tangible.

And what I was thinking of is, a couple of weeks ago, I was cleaning out a drain pipe. So, I've got a downspout from my house, and it goes into a perforated drain pipe that runs out to the edge...well, near the edge of the yard. So, when it rains, the water goes out near the edge of the yard rather than going down next to the house and going to the foundation.

The problem is water was not going down the pipe, and [chuckles] that wasn't good. We tried to figure out why and noticed every time it rains, water is coming out the top of the pipe...at the bottom of the downspout rather than going up the other end.

So, we ran a pipe snake up the end of the pipe, thought, okay, there's probably something in there, and gave it several tries. But one of the times I pulled, like, wow, that's really stuck in there, and I pulled it out. And I pulled out, like, a several-foot-long length of root that had fine roots shaped around the inside of a perforated pipe. It was [laughter] a very interesting pipe-shaped root. Oh, that's interesting, kind of creepy looking actually [chuckles].

This strange thing reminded me of an image I saw of some blood that somebody [inaudible 02:08] from a lung, you know, this very twisted intricate thing. And that still didn't even begin to get all the roots out of it. We were, like, oh, you know what, this is not going to work. Eventually, I just tore out the whole thing, threw it away, and replaced it with a new one because it was not going to work because that tangle of roots apparently had not been cared for for a long time.

Where it had been, we'd had a bush, a weigela bush, that had grown unreasonably large for its location. Why did it grow so well in that spot? The light's not very good. It's very dry. Well, now I know why it had done so well [chuckles]. It had filled that pipe with roots. And the bush isn't even there anymore, but apparently it left the roots behind, and it's gradually, like, silted in and was just totally clogged. Water was not going through.

I guess when you leave something to get tangled for that long, eventually, you end up with just this spaghetti that won't allow anything to be accomplished through that pipe. Not serving purpose anymore. Which leads me to an example [laughs].

Some years back, I was doing Ruby on Rails, and we had a Rails controller, and I don't remember the exact length, but I remember it was over 3,000 lines long in a single file. And there were multiple competing workflows all implemented in the single Rails controller. It just kind of did everything in that general area of the code. And, like I said, there are multiple competing ones. I think they're both still active depending on, like, what version. I don't even remember what exactly...how you got in one or the other.

And, you know, you’d go into the one and, in the frontend, it would send you to another. And you had to know the details of the template to know how which action got called in the controller because of course it had, like, a hundred methods. It wasn't clear at all which ones should be public, which ones shouldn't be public. I put a junior engineer on that once to try to accomplish something. After a couple of months, they hadn't got a single PR out, and, finally, we just gave up and had them work on something else.

JUSTIN: I've done that [laughs].

MIKE: I regretted sending them in there, into that dark cave, entangled and twisted, which is our introduction for our theme today. As we've already said, we're going to talk about the horrors that we have seen in code. And we've got some people who've been around for a while [laughs] and have seen some things. I started with my story, but, hopefully, I think we've got enough collective experience there to see all kinds of things that will scare you. So, what are some scary things that you all have seen in code? I've got a list, but I'll just sprinkle them in as we go.

DAVE: I think a lot of these they can go in interesting ways, right? For me, the most horrifying code stories aren't the ones where you go in and you find a mess that somebody else made, you know, some cavalier cowboy, I mean, those don't scare me. They just make me mad [laughs]. The code that makes me just shake my head in wonder is often code that succeeds, code that actually works.

So, I will tell a quick story. My story for this is my very first Rails programming job. I wanted to be a Rails programmer. I was doing a ton of PHP, which I hated. I was doing Perl and Python on the side. And I was getting into Ruby. I really liked Rails. And a guy called me up and said, "Hey, do you want to da da da...?" I'm, like, "I mean, if I can work in Rails, if can do it in Ruby, I will." And he's, like, "Sure".

So, he puts me on a project. It's a Windows .exe file. So, you have to install it with Windows with install.exe. And we wrap up the Rails OCI. That was the one-click installer. This was an .exe file that you could run way back when. And it would give you Ruby and Rails and everything you needed to run a Ruby on Rails site on Windows. This is, like, 2007 era. So, I mean, this is actually significant, right? I mean, this is...we're playing games with, you know, like, Tomcat Manager and all that good stuff, and this one didn't use Tomcat; I think it used Postgres. But you get the idea that it was managing the whole stack.

And I got given the job of updating the installer to chain either to add a gem...I think we wanted to change the Rails version. And the project was new enough－it had been out for a year or two－that we had never upgraded Rails. So, [vocalization] how hard can it be? I'm programming in Rails.

So, I will skip a long involved, headachy story. But what I ended up with was the installer is written using InstallShield. InstallShield is written in Delphi, which is Pascal. So, I wrote a Pascal program to go into this installer. That Pascal program then, when you run the installer...by the way, sorry, I left out the most important detail, which is when you run the reinstaller, it will fail if Rails.exe is already running. So, I have to go look for it and tell it to shut down. It's just, like, I'm going to, you know, click here to kill the Rails process and restart, right? That kind of thing. So, that's what I have to do.

And on Linux, you just PS and grep the table. On Windows, there's a thing called the system process table. This is 20 years ago, so I may have the names on these wrong. But it's a table, and you query it using a COM native call. So, I wrote that in Pascal. It goes out, talks to the system process table, and sends it a SQL query.

So, now I'm using Pascal to write SQL to query the process table. I find Rails; I shut it down. I'm able to do that with an API call from inside Delphi. And at long last, I finally get a piece of code, and I change a version number, save it, install it, wrap it up, test it, ship it. It works. It's horrible.

If you are using InstallShield right now today in 2025, I would not bet any amount of money that they have moved off of Delphi. I would not be surprised at all to find out that Pascal is still going strong.

MIKE: It's one way to write a Rails app.

DAVE: Mm-hmm. Mm-hmm.

JUSTIN: [laughs] So, I've got a scary story. And this was...I don't know if it is so much a scary, but it was, like, you know, I worked for a large fintech company, and the downtime was measured in millions of dollars per minute. So, we had a window of once every quarter where we would do our install. And the install would start at around 11 o'clock p.m., and it had to be wrapped up by 4:00 a.m. And so, you had one chance per quarter that was scheduled to do the install. And right around 3 o'clock, everybody starts to feel really nervous because things aren't going well.

And, you know, you're doing installs on different servers and in different locations and testing and everything. And 3:00 o'clock rolls around, things aren't going well, 3:30 rolls around. And the guy in charge is, like, "You really want to go forward with it because it represents three months of your life." But they're, like, "Sorry, guys, we're pulling the plug. Roll back." And then that just messes up a lot of schedules and everything else, and you have to do everything.

But it's, like, you know, you go in there, and you worry about it, and you don't even need to drink any caffeine that night because you're just so much on the edge, so...[laughs]

MIKE: And that could compound because now your next quarter deployment is going to have everything from that one plus everything from the next three months. And so, it's going to be twice as likely to fail.

JUSTIN: Yeah.

MIKE: Oh [laughter], that's not a good experience.

JUSTIN: No I --.

DAVE: Spooky. That was a good spooky story [laughter].

JUSTIN: You can feel that the palm is sweating right now.

MIKE: Oh, man. So, we've taken turns so far. Kyle, what do you got?

KYLE: I've got one that all of us might appreciate. That's when you've had a security incident at your company, and you start going through the repos for your codebase, and you find hard-coded keys and secrets. That can be scary, especially when you're finding keys that are shared across several services by, you know, several repos. And they're hard-coded in each repo, and they haven't been rotated in years. That's one that I've experienced and one that was scary to me.

JUSTIN: And you're, like, I have no idea how to rotate these or where the original keys came from or --[laughs]

KYLE: Yeah, no, and there's always a deadline, right? Get these rotated by end of day. And you have no idea the historical context behind them, who put them there, why they're there, and where they're stored to even be rotated. Like, the context, you're flying blind a lot of the time in these things, right? You don't know who it's associated with, what role, what access. It's, yeah --

DAVE: That’s awful.

JUSTIN: Ah, good times [laughs].

MIKE: So, a lot of security-oriented items here. I've got a few of those. I've got one that I think I've shared before, so maybe I won't...well, I'll maybe mention in passing. But I worked on a reporting system. I came in on it, right? I kind of came into an app, like, so I see you have some reporting. How is that done? And the entire system was completely reliant on SQL injection. That was its fundamental mechanism of operation.

It would get snippets of SQL from the frontend. They would be sent to the backend and just sent verbatim down into the database. Go for it. Here's how you get your reports [laughs]. And it had been running that way for years. And they’re like, yeah, that's how we do it.

And the funny thing is, I decided to do something about it. And the engineer who decided to do something about it came up with a solution. So, I've got to be careful here. The person who did this came up with a solution that worked and that met the constraints of the problem, which was fix it inside the code, and so that's what they did. They fixed it inside the code. And to solve it, they wrote their own SQL builder, which had dozens of files, which could have been its own massive project anywhere else [laughs], and took, I think, a couple of months to build and then connect to all the reports.

And I was given the review when that went out. And it went out in a single pull request that had about 100,000 lines, if I remember correctly [laughs]. And it took me multiple days just to scan through the code to just try to maybe catch anything. Like, you know what? I guess it's good [laughs]. So, I don't know if the cure was better than the disease.

One thing I learned from this, because, again, it was a brilliant solution in that it worked. It just, you know, the problem with that is our data was growing so quickly. So, within a year or two, we couldn't run reports against that database because you shouldn't be running reports against your production database because it doesn't work. And so, all of that work was thrown away within a year or two because [chuckles] there's no way it could possibly work because you're querying the wrong database. But there's a moral to this story. Do not build a reporting system on your production database [chuckles].

DAVE: I mean, that used to be a best practice, though, right?

MIKE: Sure.

DAVE: I mean, like, 20 years ago, hard drives were expensive, and RAM was worse. But, yeah, more and more, the data team keeps coming back and saying, "Stop being so scotch with memory and storage, and stop making us do so much compute because that hurts us."

MIKE: Well, and there are systems. It's a solved problem today. Reporting, go with your vendor. They can give you some pretty reports. You're good. And that way, you're not trashing your production database by throwing these huge queries at it that it's not designed for. And you'll be much happier.

Again, I've seen...so, I said I'd mention one because I think I've mentioned it before. Another project that was handed to me [laughs] I was working at a company where they acquired...I don't know if it's quite the right word. They took over a company going bankrupt or failing. They had, like, one engineer left, and he was about to quit because they weren't paying him. And it was a credit card processing gateway, not the kind of company you want to be underfunded.

And, like, okay, and we kind of figured it out, figured out what was going on, and I got the lay of land. Turns out that before they handed it to us, without us knowing, maybe them not knowing...they probably had no idea because they weren't even looking at it.

They had left a SQL injection flaw in the code, and it had been exploited. And by the time we discovered it, over a million dollars had been stolen from their customers.

DAVE: Wow.

MIKE: It didn't go over really well. I did not go over very well at all. That was not the best acquisition I've taken part in. Also, it has led me to be very sensitive about injections [laughs] in the code.

DAVE: It's one of those meetings you start by going in and saying, "Okay, guys, this message is going to make you want to shoot the messenger and everybody else in the room, but I'm the messenger, and don't shoot."

MIKE: And I believe that I was the one who found the flaw after we knew that there was a problem. It didn't take me that long to find it, actually, because I knew what to look for. This was in VBScript, you know, old ASP [chuckles], way back in the day, and just sanitizing your parameters wasn't always a thing, you know, it was kind of opt-in. There’s another spooky one.

DAVE: That brings back memories.

MIKE: Yeah. We're kind of going around the table here. Dave, I think you're next up.

DAVE: I don't know if I told the Toyota story here. I told it so many times on Rogues that Josh would roll his eyes, and here we go again, every time I'd tell it over there. I think I’ve told it here.

MIKE: [inaudible 16:39] rolled his eyes [inaudible 16:39] the vehicle.

DAVE: [inaudible 16:41] the vehicle. Yeah, exactly. That was where the bug report began with, "Fortunately, no one was killed, but..." Actually, I have a better one. This one was genuinely terrifying.

In 2016, I went to DEF CON. I took some time off work. And so, I wanted to go to DEF CON forever. It's a security conference in Las Vegas. I wanted to go forever. I went down there, had a great time, got scared to death in one talk because they start going through how AdBlock is selling everyone up the river. And they claim to be anonymizing your data. So, their talk was this is how much of a click stream we need from you to de-anonymize you, and they put up, like, five links.

On Twitter, if you visit your profile page, you cannot visit the profile page unless you are that user. So, if you visit a user, like, the profile settings, the edit profile, if you are on user/settings, we know who you are at that point. You’ve just told us your username, that sort of thing.

And the moral of this story is, even if you are on HTTPS, the URL that you send and the query string that is attached to it is in the clear. And AdBlock and all these other people...the reason you have AdBlock on your computer for free is because they're selling our click streams. And if you've got $100,000, which is what they were about to use for their experiment, they're like, well, let's try it out, you know, let us try it out.

For a hundred grand, you can buy a database that is accurate to the click stream of everybody that has AdBlock Plus installed, and it's in near real time. I mean, we're talking seconds from you clicking on a link to people around the world knowing that you clicked on that link.

I came back to work and panicked because I'm like, "Guys, we are sending this information over a query string. There's this piece of data right here. It's in the clear." And I'm going to stop telling the results of that story at this point. But the end result is, if you get more than 400 records exploited, the HIPAA laws means you have to voluntarily submit yourself to this very angry...nah, nah, nah angry is not the word, very strict government body for deploying your code. Imagine having to go to a federal regulator and ask for permission to deploy your software every single time.

MIKE: Oh boy.

DAVE: Right? If you're on the OCI, the Office of One Click Installers...

MIKE: [chuckles]

DAVE: The Office of Civil something. Anyway, it’s civil rights, OCR, there we go, Office of Civil Rights. If you have a HIPAA violation, they get involved. And we managed to keep that number under 400, but there were a lot of sleepless nights.

MIKE: Scary [laughs].

DAVE: Very.

MIKE: Justin, I think you're up next.

JUSTIN: Yeah, my next scary story is not so much in code. It's more of a situation, another fintech firm. This one, a bright-eyed and bushy-tailed MBA came in as a director and thought of ways to save money. And they were like, "Oh, we'll save money. We've got to do these projects, but we're going to save money by contracting out the work to China."

Now, this was back in the mid-2000s, or something like that, and so China wasn't quite as belligerent. But the thing about it was that they kept on not being successful with any actual work being done there. And then they got alerted, like, our department got alerted because the entire codebase got published on the dark web. And they think that they tracked it down to some of those Chinese contractors.

And it was just like, you know, that money that they saved, some incremental amount, by contracting out this very important source code, was that any savings was totally obliterated by this situation that they now found themselves in. So, yeah, moral of the story, really, really examine who you are contracting out your source code. You know, the most important thing to your company, the thing that makes you money, be very careful who you send that to.

MIKE: Kind of like putting your crown jewels in a room with no security camera, and easy access from the street in Paris.

DAVE: Yeah. Or letting AI look at your codebase. Oh, did I say that out loud? [laughter]

JUSTIN: And one result of that that was really frustrating for the rest of us is everybody lost their individual laptops. We had to log on to VDIs from then on. And back then, VDIs, you think they suck now? They really sucked back then [laughter]. So, it was like, you know, it was a bad experience.

MIKE: Kyle.

KYLE: The one that I had to deal with a while back was, I don't know if this falls into scary as much as it falls into the frustrating that Dave here brought up, but I had a coworker that left the company and had recently migrated a project from one version to the next. And I was tasked with debugging and getting the service up and running.

Now, my breadcrumb of trails was about 50 commits to the repo with either test or other commit messages, which were not very useful but most of them being test. And I had to go into each commit message individually to figure out how we got from step one to step two while we were making this transition from one version to another, to be able to backtrack the decision that was made because I couldn't just go and talk to him about what had been done. But yeah, that's my scary story, is commit messages that are next to useless, and many of them.

MIKE: And I'm guessing they didn't rebase before merging, so...[chuckles]

KYLE: No.

MIKE: No order.

KYLE: No, no.

MIKE: [laughs] Oh, I think that makes it my turn. Which one should I do next? I've got a couple of them. You mentioned the one about China, Justin.

I did once run into a situation where there was a contractor that didn't generally show their face on camera very much, which was interesting. But it turned out that they had been subcontracting out their work to a number of other engineers overseas. So, the person we were working with was actually an unknown number of people who were all working as a collective overseas for the supposedly onshore engineer [chuckles]. And we found out about that one because of the code being leaked on the dark web, not a great thing [chuckles]. One of the subcontractors thought that they'd get a little greedy and make some more money in another way [chuckles]. It didn't work out so well [chuckles].

Another one I was thinking of is...so maybe this is enough. I can put three letters together that will make everybody shudder: P-H-P [laughs]. Not so bad today. I guess a few people use it. Facebook still uses PHP, don't they? They wrote their own compiler, not interpreter. I think they wrote a compiler. And so, you know, they made it work.

But 20 years ago, yeah, 20 years ago, I'll say, in PHP, they had a library for connecting to MySQL, which was the popular default database at the time. And they had a standard library for connecting to it. And their standard library, by default, did not parameterize any queries. And so, the standard default behavior out of the box was to do exactly the wrong thing and have a SQL injection for every single query that you ran [chuckles]. And that was, out of the box, this tool that I think at the time was the most widely distributed web platform on the planet and eminently hackable. It was a scary time [laughter].

JUSTIN: You remember 20 years ago writing all the...I don't know if you guys are front-end engineers, but you had to write some sort of crazy nested if statements about which versions of browsers you supported and whether or not the JavaScript you were writing would actually work on those browsers.

MIKE: Oh man, that’s right.

JUSTIN: And then as time went on, you kept on having to support Internet Explorer 6.

MIKE: Internet Explorer 6. That was the one. It was Internet Explorer 6. That was everybody's bane [laughter].

JUSTIN: Oh man. All the way down was like, if Internet Explorer 6, you know, do the very most simple thing or display a message saying, you know, "Your browser is not supported. Click here to go upgrade."

MIKE: Right. Didn't even Microsoft have a funeral when they finally killed that up?

JUSTIN: [laughs] Yeah, I think so.

MIKE: I remember seeing...I remember seeing, like, they had a picture of, like, a cake with a tombstone [laughs].

DAVE: I don't know that one, but I know that when they released the Zune, they made a coffin or a casket-sized iPhone and had...or iPod and had a procession of getting rid of it. I’m like, what? It turned out to not age well.

MIKE: It didn't, no [laughter].

JUSTIN: No [laughs].

MIKE: I don’t think we even remember what the Zune is [laughs].

DAVE: Mm-hmm. Mm-hmm. Oh.

MIKE: It was brown.

DAVE: Yeah. I wrote a video game years and years ago, and Old Man Murray reviewed it. And it's an abusive review site, hilariously so. They reviewed it as this game comes in a red cartridge (This is Nintendo 64.), but it ought to be a dull brown because this game is a silicon turd. So, anything that comes in a brown package makes me giggle.

MIKE: [laughs]

DAVE: So, do you want to hear a scary story that will keep you awake tonight, because it's not solved and it cannot ever be solved? What if I told you that multiplication was not deterministic?

We're programmers. We like things to be discrete, right? X times Y should equal Z. You will run into this if...Okay, I'm going to bring it back to the sanity world, and then say floating point arithmetic. Okay, so you're already approximating.

So, the short version of the story, the reason it's not deterministic is because the standard for how you do multiplication or other...Any operation that could change the base, the size of the radix or the...Yeah, anyway, anything that affects the offset, multiplication, division, primarily, addition will only do it if it overflows the most significant digit. But multiplication almost always is capable if it's...

Anyway, you will see this in the wild if you are trying to get your customers off of Microsoft Excel and onto Google Sheets in Ruby, because Excel and Sheets round currency differently. And when we moved from Redshift to Snowflake, we ran into this. We had financial reports that were using an easily sufficient amount of floating point to come up with something honest, legal, and reasonable. When we moved to Snowflake, people's commission paychecks changed by like $5 or $6.

And I don't know if you know this, but when you're receiving money, you check the amount, and you notice when it's different. And we had people going, these reports have to...And even worse, you imported the last how many years out of Redshift? If I run this report in Snowflake on that year, I need the same answer. If I can't get the same answer, I have to keep Redshift so that I can run that year's report on Redshift. So, Zack, our data guru, he figured out how to make Snowflake multiply the same way that Redshift did.

JUSTIN: [laughs]

DAVE: And that is amazeballs. That just blows me away.

JUSTIN: So, which one was right, though? There's got to be a discrete answer here.

DAVE: They both comply with the IEEE floating point standard, and there's a great, big...I ought to frame it, actually, as, like, a horror. Like, put it next to the movie posters for Jaws and Psycho, a printout of the IEEE standard saying [laughter], "There is no deterministic way to do this."

KYLE: But isn't that a lot... Sorry, that comes down to a lot because we use base-10, right? If we switch to something like base-12, a lot of the deterministic issues that we run into end up actually being solved.

DAVE: They would come out differently, yeah.

MIKE: But not... It helps that what base-12 gives you is the ability to cleanly divide by 2 and 3, right, and some multiples of those, right? And with base-10, you get 2 and 5 instead of 2 and 3. And so, you get multiples of 2.

DAVE: Trade-offs.

MIKE: Yeah, you got multiples of 2 and multiples of 3 with your 12. But you still...if you take other prime numbers, just walk up your prime numbers, and you'll run into the same problems pretty quickly. So, hit 5 with your 12, well, now you've got the same problem, or 7, or 13, or just anything that is not divisible by your 2 or your 3. You run into it, and it's not going to divide cleanly. Now you've got repeating decimals, and you're going to have to do some rounding.

DAVE: Yeah. The short explanation of why this happens is, in floating point, you pick how many 1s and 0s, how many bits are going to be the fractional part. And if you don't pick very many, you've got a very imprecise number. If you pick way too many, you have a lot of 0s and wasted space. When you multiply 2 numbers, you've got this thing that's going to potentially double in size, bit-wise. But it has to go into the same space.

And so, the question becomes, if you multiply these 2 numbers, and you've got 12...like, say you're using 6 digits for precision, and your new number is going to have up to 12 digits of precision after the decimal point. Which one do you keep? And, you know, which direction...how many bits... when you put this number in the new target place, do you put it in the same 6-point thing, or do you change it around? And yeah.

KYLE: It makes me go back to my math class in college where our professor spent 15 minutes proving that .999 repeating constantly is actually 1, and how it converges to 1, and there's no difference even though they look different.

MIKE: That's Zeno’s paradox.

DAVE: Yeah, Archer's paradox, yeah. Fantastic.

MIKE: So, Justin --

JUSTIN: Well, I've got one more, and then I've got to take off.

MIKE: Okay.

JUSTIN: And this one's not really that scary. Well, I guess it is, I don't know. When you get a codebase with X number of repos, like, we'll call it 200 and 300 repos, and then you're like, oh, I'm an application security engineer. I’m going to go in and run my vulnerability scanner on all these codebases. And then you start it running, and you have a running total of the vulnerabilities found: criticals, highs, mediums, lows. And you come back, like, 20 minutes later, and the criticals are in the thousands or tens of thousands [laughs]. And you're like, oh my God [laughs]. What? We might as well just burn this down now [laughter].

But you look at that, and then you, like, start to categorize and such. And you're like, okay, you know, there's this many repos, but only, like, five of them are actually running in production or something like that. And so, you can cut things down quite a bit. But whenever you get that feeling of, you know, oh, my vulnerability scanner is running, and you see the total just incrementing up, and you're like, has this ever been run [laughter]?

MIKE: No, probably not [laughter].

DAVE: I hope the answer is no, because if the answer is yes, I need some violence in my life.

JUSTIN: You're like, why did the last one quit [laughter]?

KYLE: And then the scary--

DAVE: Why is this file called the last report, you know [laughter]? It's the previous three guys. It was the last thing they did here. You're next [laughter].

KYLE: And the scary part is you report that, and there's other business-critical initiatives going on to where, you know, even though it has a thousand critical alerts, we can't fix that before we go to production [laughter]. We're going out anyways.

JUSTIN: Going out anyway.

MIKE: We'll be fine [laughs].

DAVE: What could go wrong?

MIKE: Well, thank you, Justin, for joining us and contributing some stories. Kyle, do you have any more for the list?

KYLE: I've got one that's programming-adjacent and may or may not be relevant to recent news. That's, when you're an engineer, you come to the office, sit down to start your day, only to find out that we won't call us-east-1, but us-east-1 is down [laughter]. That is one of the scariest times for multiple times over my career that I have had. When you've made decisions and a lot of aging companies, we'll call them, they have picked that zone to be their zone. And it happens in other zones, too, but it's notorious for us-east-1.

You deal with a lot of backlash, a lot of failovers, and you are testing that DR strategy that you’ve tried to implement. You are now testing it live [laughter], and you are hoping for the best. And stress levels have never been higher, especially when databases are involved, so...[laughter]

MIKE: I'm going to add on that one. That's not only your systems. Because if any of the partners that you have, any of the vendors that we use for software services have anything in that availability zone and do not have the appropriate disaster recovery redundancy, then you're down. And if any of their providers have that, then you go down. You can fan out as far as you want, you know, it's not a single point of failure dx. It's an exponential point of failure --

DAVE: You can't take credit cards if your CC service is down, but it doesn't matter because your OAuth is down, too, and nobody can log in anyway.

[laughter]

MIKE: So, I'm going to say something a little bit adjacent as well, back from my bag of stories. I may have mentioned this before. But more than once, I have been on a team where everybody else got laid off [laughs]. That's a scary situation. The first time it happened to me, I was actually one of the people that got laid off. But then my boss quit because he's like, no, I'm not doing this, and they hired me back on. Maybe I should have learned some things. I've learned some things since then. I took the job back, and because he'd left, I didn't have any training whatsoever. He may have given me, like, 20 minutes, like, "Yeah, these are the things I do to keep things running. Good luck, [laughs]" and he left.

And so, I had to take this software application that had been running for years, I don't know how many years, and which I had never done the administration for, except kind of tangentially, and hope I got it right. I tell you, you learn a lot when you get put in that situation. And you also make some serious mistakes [chuckles] that you regret later. That's part of where the learning comes from, is from the things you do wrong [chuckles]. Luckily, no customer funds were lost, but customers may have been frustrated by my [chuckles] actions at that time.

DAVE: It's one of those 100 bad days gives you 100 good stories.

MIKE: Yes, absolutely [laughs]. You got any more you'd like to share, Dave?

DAVE: One tiny piece of...I don't know if hope is the right word, but the point in my career when I realized I'm not going to slay this dragon, so I just made peace with the dragon. When I was an early programmer, I had to learn how to do recursion, and recursion was...it broke my brain. And I worked at it, worked at it, and I eventually got it. As I advanced as a programmer, a little...actually, prior to this, pointers. Trying to understand how pointers worked were very, very difficult. Then, like, mid-expertise, I ran into problems with recursion and getting over that.

As a senior programmer, I went up against time zones. I thought, you got to be able to solve this [laughter]. It cannot be solved for part of the same reason that the floating point...well, a little bit, the floating point number also has this problem, which is that it backs onto humans, like, in real time, meaning that you can get every time zone locked in, and tomorrow you can find out that some jurisdiction has changed time zones. It doesn't happen very often, but it does actually happen. So, if it backs onto a human that is dynamic, it's not solvable. So, stay in UTC, kids.

MIKE: Well, I heard about a problem the other day that picking a time zone wouldn't help either. They were comparing a timestamp against the local time to see if a build had expired, and this was on a mobile device. And they kept on getting weird behavior, where people weren't reinstalling it, even though the build should have expired. Not really thinking that people on their mobile devices will regularly reset their time in order to get extra points in a game. Because there are a lot of those games that force you to wait, if you've got the free version, until you can get your reward again or, you know, be able to play again.

And so, people change their time by hours, 8 hours, 12 hours, days, all the time on their mobile app. And if you're trusting the system time, you're in big trouble.

I saw another one. There was a...another time zone story. There was a user where any time they looked at a resource that was shared with some others...I'm being ambiguous here to spare some parties. Any time they looked at the resource, everybody else got signed out, including them.

DAVE: Hmm [laughs]. It's awesome. You have to sign out REST resource? I mean, that would do it. No, that would only be one person, even if it was.

MIKE: It was a nice just get sort of standard page, look at the details. It turns out that this user's time had been gradually drifting forward for a year, and finally went over the five-minute expiration window that users had to look at this resource. And the way the code was written is it just said, "Hey, if you get a message coming back that this resource is locked, or that it's even being looked at, if you see that that five-minute time frame has expired, then sign out." And it signs everybody out because you're only supposed to have one person looking at it at a time or locking it at a time. And --

DAVE: Right. That's how read locks work. We just throw everyone out of the building, yeah.

MIKE: You throw everyone out of the building, and that's how it works. The problem is their time had gone over that five-minute window. So, if they ever looked...and this user's, like...and their job was to assist other people. They were, like, a training person. And so, any time they'd go in to help somebody with their work, everybody would get kicked out [laughs]. And so, they were poison to anything they touched.

DAVE: Was the session time cumulative, like, you only get five minutes to look at this resource in your entire life?

MIKE: No, you get the...but the lock was set the time somebody claimed it. The lock was set the time somebody claimed it. And any event, so there was a WebSocket, and any event that came back would have the timestamp on it. And so, even though it wasn't a locking event, if somebody just looked at it, they'd see, oh, somebody else has joined the chat, basically, right, and started looking at it. It'd see, oh, this has expired. Better kick everybody out.

DAVE: [chuckles] So, everybody on the system has software running on their computer saying, "Should I kill my own process?"

MIKE: That’s right. Every single person.

DAVE: That's fantastic.

MIKE: [laughs]

DAVE: That's fantastic. I love that. Oh my gosh, people are the worst. I love it.

MIKE: Because you think time, you know, everybody's on the same time. It should work, right? And it did most of the time.

DAVE: But it really doesn't. Kyle, the reason I said the time zones is like floating point is because outside of floating point, we still have this problem. Prior to the existence of computers, we had accountants. And so, there is a thing called banker's rounding. And for anybody that doesn't know, if you do a multiplication, division, you land exactly on one half, exactly on .5. In banker's rounding, you round to the even number. So, if your number is odd, you've got to round up, and if it's even, you've got to round down. And it's weirdly consistent.

And everybody does it when they're doing it in finances, and I haven't found a programming library that supports it. It's like, why does everybody do this and nobody has the code for it? You'll figure.

KYLE: I didn't know that. I've never thought about a library not existing for that.

MIKE: That's interesting because they keep it fair. Because just randomly, you've got to make sure that it balances out.

DAVE: Exactly. Exactly. Because the accountants basically got to have a fair way of...yeah, if I take a little bit off of your penny, then later you can take a little bit off of my penny. Yep.

KYLE: Now I want a library that actually does a round and just, you know, oh, there's a five round up or down.

MIKE: Deliberately non-deterministic rounding. Nice. So, you can get your books will balance a little bit different every time.

DAVE: So, yeah, half of three cents is two.

KYLE: That explains why my paycheck is, you know, one cent different week by week [laughter]. It might actually be that. I don't know.

DAVE: Well, Dave's pretty odd. We can just round his stuff down. Wait [laughter].

KYLE: And then you --

MIKE: Well, we've done a survey of spooky stories. I hope you've enjoyed as a listener and hopefully learned something. Generally, all these things have a lesson, right [laughs]? You touch the hot stove. Hopefully, you don't do it again. And if you've watched somebody else touch the hot stove like the people in this call [chuckles], you don't have to make the same mistake. You're probably not listening to this on Halloween, but you can reminisce and hopefully enjoy.

And until next time on the Acima Development Podcast.