S2E05 - Understanding and Working With Tech Debt

Péter (2): Hey everyone, welcome to the retrospective, the engineering

leadership podcast where we talk about topics for engineering manager that

they face in their day to day jobs.

me here is Jeremy and I'm, I'm Peter.

Let's get into it.

Jeremy: Hey Peter, it's great to be here today and I'm really enjoying

Season 2 and the topics we're covering.

Péter (2): Same.

Jeremy: So for the, for the listeners, just the way that we've been planning

this season is that we've been taking turns bringing a topic each.

So this week it's, it's your topic.

So, what are you bringing today for us to talk about?

Péter (2): I have a nice, beautiful surprise for you,

which is called TechDept.

think it's a favorite of a lot of engineers and engineering managers

and probably business leaders also, not necessarily in a good way.

We're not going to solve TechDept in 30 minutes, but what we can do is share

our combined experience on the field.

And, uh, yeah, let's talk about TechDept.

Yeah.

Jeremy: um, partly because in a lot of companies where there are,

I think we're in, you know, we, we hear the end of zero interest.

We hear a lot of focus on, companies that are having to do more with less.

There's a, there's a high pressure, especially in startup land and a

lot of pressure to ship more in a very competitive environment,

certainly that's something that we're, I'm experiencing in my job.

And, um, at the same time we have to manage this mystical

thing called tech debt.

And, uh, so I think it's a very timely topic for the, for the

period of time we're in right now.

Péter (2): Yeah, and another aspect why I think it's super

timely is AI assisted coding.

I'm pretty sure that in every companies there are a lot of code checked in

the code base that was written by or with the huge assistance of AI.

if it doesn't work so well, that's going to increase your tech debt.

Big way, and maybe not so obvious ways, uh, initially.

Jeremy: Actually any interesting anecdote, Google said that 25

percent of code at Google is now written by AI and, um, where are you?

Péter (2): like 25 percent of what, like, uh, yeah, sure.

The line breaks, the prints, the, the, the returns, the functions,

there is a lot of text in code that's, that could be easily automatized.

Jeremy: I think it's auto complete.

I think it's just fancy auto complete.

Um, and we've seen that.

Péter (2): underplay, it's super

Jeremy: Yeah,

Péter (2): but yeah,

Jeremy: yeah, we've seen that we're using an AI, , plugin, called Codium

and it gives us stats and it says it's done 40 percent of coding for us.

But I think most of it is just a fancy auto complete.

That you're going to be doing anyway, it's kind of like, autocomplete in, Grammarly

or in, I don't know, in Gmail when it's completing sentence this that you would

have written anyway, that's, yeah,

Péter (2): Yeah.

Saving time and as a side effect might increase your tech debt.

Jeremy: exactly because unintended things creep in, but, yeah, so maybe if you can

help us level set things a little bit, Peter, how would you define, tech debt?

Péter (2): Yeah.

Yeah.

I actually thought a lot about that and I came up with a long, uh, definition,

long sentence, but, uh, I will unpack it.

So my definition of tech debt is our best decisions, negative effect.

on the quality of our systems, limiting our options today.

There's three aspects to that, past decisions.

I wanted to capture that, uh, with tech, you created tech debt.

Tech debt is, is, is something that you live with today, but it was your past

decisions that resulted in some tech debt.

I want to start here by explaining that,

Jeremy: Um,

Péter (2): code health in, in, in a lot of different ways.

And systems can mean anything, technical, your code, your infrastructure, maybe

even your processes, CI, CD pipelines,

Jeremy: Um, Um,

Péter (2): options.

You can have, uh, you cannot cut costs so easily.

You cannot pivot to a different feature.

You cannot re architecture your system to today's needs so easily.

So it's basically less less options you have in in in what you're doing.

Jeremy: Yeah, I really like that definition, Peter, because yeah, we, we

do need to on purpose take decisions.

And in fact, tech debt can accumulate from very sensible decisions where we

choose to, be pragmatic architectures, not overbuilding for the future.

And then over time as things scale and grow, we need to constantly adjust.

For that.

Péter (2): that you don't want to miss and you cannot have, have it all.

It's always a compromise.

So you will need to compromise on the quality and increase

the tech depth of your systems.

I like the mortgage analogy taking out a big loan to be able to buy a

house where you can live rent free.

Oftentimes it's a good decision, arguably, and, it limits your options today.

You, you cannot, maybe you cannot buy a new car for a year now.

Maybe you cannot go on a big vacation for a while because you keep on have to

pay back, uh, this monthly installments.

So yeah, tech deft is not necessarily bad.

important thing is it should be conscious.

The result of conscious Decisions and not just happenstance and accidents.

Jeremy: Yeah, yeah, I'm thinking about this also from explaining

it to someone in the business.

Think your mortgage analogy can work really well.

I think another way to describe it is, you know, you used an example earlier

where we We try, cut some corners to try and get something out the door fast.

I was thinking about this one where you have a plumber that, comes and fits a new

sink and they don't actually have all the pipes, so they fit something in cardboard.

That's very weird analogy, but we do do this in tech, um, but if

it's some cardboard pipes and some tape stuff up and that can work.

Potentially temporarily, um, maybe not for too long, but if you increase

the pressure or if you let it run for too long, then that's going to fail.

And oftentimes that's the kind of tech debt that we, we do.

We build something just enough to ship, but then that cardboard needs to be

replaced, and put in real pipes and so on.

And that's again, again, the kind of classical approach

to tech debt that we have.

You know,

Péter (2): Yeah.

That's, that's, that's a good analogy.

Cause cause Using cardboard piping, whatever that is, it's, it's still

better than having no pipes at all and having water coming off

your sink to your feet and, and

Jeremy: yeah, or potentially a sink that doesn't work for a bit,

but you really need it, you know?

Péter (2): yeah, yeah, exactly.

Jeremy: Yeah.

Okay.

Have you come across any other good ways to explain this to the business?

Because I think one of the big challenges with tech debt is just

actually explaining the concept to business stakeholders and even users.

Honestly, sometimes we have to explain to users why things are taking longer.

Péter (2): any approach works that connect the limited options that I

was talking about in the definition to the business, business dictionary.

We have limited options in quality, stakeholders will feel the pain when

it takes two days to get back from an incident as a result of some tech

debt in your processes on pipelines or infrastructure, for example.

Stakeholders don't care about, the code health, the duplications, the redundancy,

everything that makes code problematic.

But they do care about quality.

They do care about costs.

They do care about, uh, user satisfaction.

You as an engineering manager, the listener of this podcast, your

job is to translate between these stakeholders and engineering, and make

the connection the impact of that in in the business goals and outcomes.

Jeremy: Yeah, I really like that.

Just connect the lack of options you have today and the negative effect that

past decisions had on the quality and explain that using different analogies.

Péter (2): Yeah.

Yeah.

Jeremy: Yeah.

Péter (2): If the business is in a big cost cutting exercise.

You can tie your tech debt initiative to that and focus on the infrastructure.

Maybe, uh, where you can decommission some old services by moving them to

two newer ones and save some money.

It's, it's, it's not a fight.

It's a discussion between, between the business and the engineering side.

Jeremy: Yeah, I agree.

So how as a team, when you're a team, how do you tell when

you have too much tech debt?

How do you measure, how do you detect it?

Like, what's, what's your, what are your thoughts on, on that?

Péter (2): Yeah.

I really like that you use the word measure because talking about data

is, uh, much more productive than talking about feelings and beliefs.

I'm pretty sure that there are a lot of engineers who are frustrated

about tech debt with the code they work with, but some of those people

will probably struggle if they have to point to some numbers, what they

mean by tech that, uh, exactly.

So fortunately we have a lot of, good metrics in the industry that

can be used and satisfy the need to connect this assessment to business.

What, what we were talking about, metrics can, can come as hand a pretty big

industry standard 2024, when we are.

Talking about this, especially metrics like the lead time for changes or

or failure rate or something that can tell a lot about the system.

The time to restore service.

I really like this metric because there are a lot of things that can impact this

metric and a lot of those are tied to tech depths in various parts of your

system's infrastructure code or processes.

So you can use Dora metrics, you can be more direct and measure some

quality metrics number of bugs or number of new bugs creating the last

three months, response times, of your systems, like more performance metrics.

It really depends on the organization.

What are the goals of your organization?

What is the business trying to achieve and finding those metrics and , bringing

in some depth, uh, aspects can, can help.

Um,

Jeremy: I think the other part for me is, is toil, Not fun stuff that you have to do

and I have a really great example of this.

There was a infrastructure engineer, in one of the companies I worked

at, where we had a very manual process to deploy the application.

And he recorded a video of what the deployment process was, and it was

over 300 clicks and, manual things taken to get a whole thing live.

And obviously that was to explain what was needed to be done and to

show very vividly the tech debt.

And obviously we invested heavily in automation and, we got to the point

where it was just click a button.

The whole thing was done automatically.

So

Péter (2): I really love that story.

Uh, the tool this engineer was using recording themselves doing this actual

process, it's very powerful and yeah, I don't need to say it, but, uh, those

are 300 ways this process can break.

I'm pretty sure.

There were a few occasions when some of those broken.

Jeremy: all the time, because how do you remember and that process was

already an improvement because the original one, when that company was

acquired, the first time it was all deployed from one person's laptop.

So at least

Péter (2): wow.

Jeremy: the process that he videoed was being able to be

run from multiple places, but,

Péter (2): you virtualize that guy's laptop?

Jeremy: No, but, uh, yeah, I mean, you just invest.

Yeah, I think sometimes it's really, really obvious.

It's like the elephant, um, and you know, sometimes it's, it's less obvious.

So depending on the situation you have different kind of ways

of checking and tracking it.

Péter (2): It's important.

You say sometimes it's really obvious.

Of course it is for an engineer.

And I would say even for a non engineered leader seeing that video of

the 300 steps that it takes to deploy the application, it's very obvious.

They will say, Oh, my God, I feel your pain.

I promise you we are done with this sales cycle.

We are done with the next product launch.

You can, but they don't Actually feel the impact on the business because

what they are walking away from the demonstration is that, Oh, poor Joe.

He has to go through 300 steps every day.

That's not the impact.

The impact is all the ways this can break and, and all the ways you will

struggle to release a new feature when, when something is changing this process.

And that's what the business feels that the users don't get the new feature.

Features take longer.

Features are buggy incidents are happening.

So I would say even when it's obvious.

Think about the users of your systems and the business stakeholders and

make it obvious for them the pain, what the option limitations are.

Jeremy: And actually, in this case, the users were being impacted because things

would go wrong in that process, as you say, and there were regularly fires.

So that was easy.

It was an easy sell.

Péter (2): yeah, yeah.

Okay.

Jeremy: Those are the obvious situations, but what about in a situation where

it's, it's less obvious and there's more of a disagreement about tech debt.

How do you come to an agreement then inside a team or between a team and

its stakeholders that you need to invest on tech debt or that it's

acceptable to leave it as it is.

Péter (2): That's a really good point because because we were talking about

convincing the business, but oftentimes you need to convince your engineering

peers that we should address this type that and not that type that.

What you can do is I like to think in teams and not in individuals.

So in this case, I would suggest a practice something like attack

that tour like every other sprint or some regular occurrences.

You pick a different engineer and they hold a short 15 minute walkthrough of

the part of the code base or the part of the processor or infrastructure

that they are most concerned about.

They explain just like this 300 step video.

That's a perfect example for

Jeremy: Silence.

Péter (2): base.

Maybe we could address this together.

So elevating the understanding and the knowledge of the team, with these

regular tech debt tours, I think it can be a good first step to, to create

alignment on the engineering side, that this is the most impactful or most

urgent thing that we want to address.

In the next time, and then you can tie to the business goals and

impact.

Jeremy: no, for me, I think it's really important when we talk about the team here

that the product manager is involved in that because, , often I think engineers

can quickly see and agree on on things when it's made more visible like that.

But product manager it's probably caring less about the tech debt

and they still have a lot of the concerns of the business.

And , it's super important that they understand the impact on the

team to be able to protect the team when you take a decision to work

on tech debt and, um, they need to be part of that, group as well.

Péter (2): I totally agree.

And thanks for bringing this up.

I'm a bit biased because I was leading platform engineering teams most recently.

And I didn't have product managers in my teams.

I had a very good product partner on the level, but it's super important

what you're saying to include the product side in this alignment off of

seeing the problem, because oftentimes when priorities are decided, you're

not even going to be in that room.

It's the product manager of the team who's going to have to fight for for

the engineering side also to make the pain visible and get some leadership

buying and stakeholder buying to address.

Some of the tech devs, so definitely include the PM in these discussions.

Jeremy: Yeah.

And I think, um, going back to the point you made earlier about

data and measuring and so on.

The data bit is really important.

In the past, , we've done things like measuring the types of work or the

signals of not having invested on tech debt, like whether it's an increase in

bugs, , and some of the other metrics that you mentioned earlier, but I

think those are, those are important signals as part of this discussion about

aligning on, on tech debt and what to do.

Péter (2): they can be very powerful because they bring the discussion to

principles to how we want to operate.

For example, if you manage to track where you're spending your time, is it feature

development, unplanned work, and then the organization can have an ideal and

say, okay, I want to spend numbers 80 percent of time on feature development

and 20 percent or everything else.

And you come back with metrics and say in the last two quarters, we could only spend

60 percent of time on feature development.

And that's the impact of the tech that we are having.

So we would like to double down on that in the next quarter so we can get this amount

of time closer to the ideal 80 percent that we set as a goal as an organization.

Again, you're bringing the discussion.

to talk about interpreting data and impacting data and not about

philosophies like code should be beautiful and, uh, don't know, or,

of thing approaches that are not useful in business discussions.

Jeremy: Exactly.

And I will say the classic thing that a product manager or a business person,

uh, will do is say, listen, I just need you to deliver these features.

And, uh,

I promise you, I just need it for this next six months because it's super

crucial for our business right now.

And then we can do some of those other projects that you have that you talked

about, but no, we can't do those.

And we need to do this.

And then six months later.

Business, still in a crunch and the platform starts to burn

more, but you need to continue.

So that starts to make me think about, what are good strategies

for managing tech debt?

Are there different approaches?

What would you recommend, Peter?

Peter.

Silence.

Péter (2): but your, your house is not on fire.

So this is, this is a critical stage because you need to maintain this

and maybe gradually improve this.

One approach I find useful in these situations is pairing tech

dev work with feature development.

It's basically, uh, you know, there is this, I think it's from

the Boy Scouts that, leave your camping site cleaner than you found.

I think it applies to code and everything as an engineering team touch.

Like, uh, you develop a feature about, I don't know, uh, newsletters, then take a

look at how email sending is implemented and maybe refactor it slightly.

To, to make it healthier, do some small refactors that are

adapting it better to today's needs compared to yesterday's needs.

So pairing this work with feature development has two big benefits.

One is that you're in the context and mindset of that current feature.

You don't

to dig through documentation.

What is it that we are doing here?

Oftentimes.

is your documentation and nobody knows why we are doing things the

way we are doing it, but that's how it's implemented in the code.

If you're touching a feature like this that has a lot of historical parts

implemented, you can discuss with your product manager and remove some of those

while doing the feature development.

Jeremy: You

Péter (2): bit longer for, for the completion of the

feature, because you keep

Jeremy: Silence.

Péter (2): a lot of discipline, but it can be very efficient.

Jeremy: Yeah, I think the discipline is to let the scope creep of the

tech debt refactoring overtake too much that the situation.

Um, but I mean, I've, I've seen a really good example of that

recently where a staff engineer was working in the code base.

They found.

Some part of it that had a lot of toil to update versions.

And so they improved how we used the renovate bot to

better do that in the future.

classic fixing debt as you go approach.

Péter (2): Yeah, now you mentioned risks and it's very important.

Getting carried away on and refactoring stuff that are

not necessarily in the scope.

And, and then you didn't deliver the feature, but you're already

two weeks late and the other risk I would say is premature optimization.

It's when you think that, oh, there is this new feature newsletter sending,

maybe I should have a separate service that handles emails and maybe I should

have it done with an auto scaling microservices infrastructure and then you

start working on those and then Next week, the product decision is that actually,

which is going to use a third party for newsletters because it scales better

and all those work, all those premature optimizations was a waste of time.

So

Jeremy: Exactly.

Péter (2): Only solve today's problems.

But still paired the tech depth work with feature development.

Jeremy: It's really funny because I really like this example.

You just used the newsletter, sending thing because that's a

really good pragmatic decision.

It should be made like that.

Most situations.

Um, but the funny thing is that might become tech debt later

when you realize that, um, Yeah.

I know we're not going to use that external service and we are going,

we're going to need to scale it more.

And that's perfect.

You know, that, then that's a good moment to work on, on changing it.

So that's just a good example of how

choice now.

And then it's just a loop, you improve later.

You'll need to improve it later.

Péter (2): Yeah, you cannot solve tomorrow's problems because you

don't know tomorrow's context.

This is exactly why you need to deliver stuff quickly to get the

validation and feedback from your users.

And those information will affect your decisions tomorrow.

And maybe you will double down on in house built infrastructure.

Maybe you will pivot to external.

don't know.

Nobody knows.

That's the point.

You just solve today's problems.

Jeremy: Exactly.

Now those are all the kind of like the small ongoing things.

Péter (2): the big one.

Jeremy: Yeah.

Sometimes you have something really big, like that example of

the manual deployment process or others, how do you handle those?

Péter (2): I read it in, I think in one of Will Larson's books is that

the only solid, reliable way in these situations is a migration to a new system.

There is a point where your old code base, your old system is just

so Hard to work with so ridden with tech debt that it's easier to throw

everything out and start from scratch.

Now, this is not how you should do it.

And we can talk about how you should migrate to a new system.

But the goal should be this.

You leave behind the old mess because you save a lot of time trying to

discover why is it working like that?

What are the features that are still needed?

What is not?

It's, it's easier to move on to a new system from, from scratch.

Jeremy: So I have two questions when you say that one is how do I

know that I need to migrate because I've seen cases where people have

migrated or doing a rewrite when they absolutely did not need to do it.

Okay.

Péter (2): wrote their ideal to do application or commenting system

and everything and you don't, yeah, I don't like the rewrite word, but

To answer your question, um, this is why we have good engineers.

And, I hear myself that this is not a satisfactory answer, but, uh, there

are points where you can see that the, the, the system needs to move on, maybe

actually, maybe the way I can answer this is the technique of how I think

you should do this kind of migrations.

You know, the strangler fig pattern, uh, those of us who are

listening, Might not be familiar.

It's the concept is coming from biology, uh, about some kind of plants.

The point of it is that you isolate parts in your system.

You have a big, complex, interconnected system and you isolate a part

in it that has less connections, ideally with the rest of the system.

You make those connections very clear and, uh, this isolation, within this

isolated part of the system, you do some refactoring that the rest of

the system doesn't see happening.

So if you take this approach, what you get is that, you can

basically work in a live system.

You avoid this big boom migration when you say that, okay, on Monday we start a new

system, we shot down the old one and there is two weeks off of fires everywhere.

You have a much more scalable and reliable migration process.

You also increase.

confidence in the quality of the system, because isolating those connections

between the system you selected and the rest of it, the best way to do it is

to write a lot of tests to ensure that your system is still returning the stuff

that the rest of it is, is requiring.

And once you're done with that, you have those tests in your CI CD

pipelines every time you make a change.

And then maybe you don't even need to decide if it's just a small

refactoring, or if it's an entire rewrite, because it's rarely very binary.

Oftentimes it's a scale.

So maybe you can do like you isolated one system, you rewrote

it in the modern, uh, way that you want to isolate another rewrote it.

And you reach a point where 80 percent of your system is, is, uh, called modern.

I don't want to get into deeper.

What is that?

But you see what I mean?

And 20 percent is still the old one.

And maybe you're fine with that.

Like maybe that 20%, it doesn't worth the effort to address the

majority of your system is fine.

And maybe you will decommission it in two years or, or whatever.

So actually, I think this is a good way to look at this instead of rewrite everything

from scratch, you take parts isolated and migrate those little parts one by one.

Keep on evaluating if.

You still need to pay attention to tag that or you reach the

point where it's okay to leave it.

Jeremy: Yeah, I think, I can probably list some bad reasons

why you shouldn't do it.

I have some examples.

There was one stage we had, um, A PHP, um, backend with a,

a modern JavaScript front end.

The, the PHP backend was actually very well written.

Structured a lot of tests, thousands of tests, things

were fast, CI was really good.

And the new , CTO.

Who knew Python, but didn't know PHP , he didn't land and there were challenges

with that PHP backend, but, uh, he decided that he wanted to re write, the backend

in Python and, that's a total reskilling of it and then, it was really funny

was the CTO decided that they didn't like view, which the front end was in.

So we did the front end and react., and, uh, you know those are the

wrong reasons to rewrite things.

I mean, There could be some valid business reasons to do that, but in those cases,

I really felt like, having known that system and worked in it and then seeing

those decisions after, um, I lived with things that I felt were suboptimal,

but I focused on the business value.

Whereas, those rewrites generated very little value for

the business, and obviously distracted the team significantly.

Péter (2): This is why it's very useful to back to our earlier point

is that you talk about tech that you should talk about the options.

It's limiting you and the business outcomes and how it's painful and

hurtful for the business to live with this tech that every day.

In your example, maybe I'm a Overassuming.

But, uh, I think what your team was optimizing for is the comfort

of the CTO with the tech stack.

And that's not something you spend months and months off developers time

and engineering time to optimize for.

And switching Both front end and back end, the wall stack, it's

super risky and super expensive.

And sometimes that's a good solution.

Like your product was pivoting a lot and you're doing a very

differently, then maybe that's a good opportunity to change that also.

Jeremy: I do think there has to be a real business value

justification for changing text.

Péter (2): be everything that should drive these decisions.

Cause ultimately the goal of the company is to make money, everything

else is just a means to those and

Jeremy: Exactly.

So thinking more , tactically , our podcast is really targeting

engineering managers.

Um, what.

Péter (2): there.

Jeremy: Yeah.

What advice or, and, how would you recommend engineers decide, what

tech debt to tackle in the sprint they're planning at the moment, or,

you know, the, the next month ahead, how should they prioritize that?

Péter (2): yeah, that's, I, I like that lot of stuff that you

don't mention in this question, which is implied, which is great.

Like, uh, is a healthy process of every sprint tackling some of the tech debt.

And this is what I want to make explicit that that's a way for a

sustainable tech debt management.

And the other thing that you're implying is that there is.

Engineering understanding of tasks that we need to tackle.

So, so this is a great point and now comes the big question.

Okay.

Which one of the fires do we turn our attention to?

And, it's basically a decision making exercise and there are

a lot of tools that can work.

One very simple one that I can explain and I think is adaptable.

Well, here is the Eisenhower matrix for decision making, you know, it's a

two by two, um, Grid on the horizontal axes, you have urgency on the vertical,

you have, uh, impact, and then you can group your tasks into these

four quadrants, you're going to have urgent, but not impactful, tasks.

You're going to have impactful, but not urgent tasks, and

then you're going to have.

and impactful and not urgent, not impactful.

My point is that you should have a portfolio approach

because the big risk is.

you want to avoid that you always pay attention to the urgent

stuff because you feel the pain now and you always deprioritize

the impactful non urgent ones.

Jeremy: Okay.

Péter (2): have some new API commands.

And we're going to have to.

deprecate some of the current ones that you're using.

It's not urgent at all.

You have a year to think about this, but it's going to be super impactful

if you don't meet these deadlines and you're not ready with your API by then.

So my advice is a portfolio approach.

Make sure that for example, every quarter you tackle some of the.

Impactful tasks that you have and then balance it with the urgent ones.

Also, this framework helps to realize that there are some tabs that are

not urgent and not impactful, I don't know, code styling, tabs

versus spaces, this kind of stuff.

Maybe you can just defer them to a hackathon or, some, backlog , of

we might address one day.

Uh, not, not put too much effort in it.

Jeremy: I like that.

And I think there's this kind of balancing approach that you described,

between features and tech debt and , choosing which ones to do, and

there's another approach, which is, like the kind of Maslow's, , hierarchy

of needs , pyramid approach, which.

Pairs well with kind of a zero bugs approach, which is essentially on the

bottom of your foundation is things like security and compliance and so on.

So anytime that you have something that, you know, is not secure or

updating a library or whatever, you do those, all of those things first,

and then you do things that affect reliability and stability performance.

And then it's only when your pyramid is stable, and then you do the top parts,

the, the delivering value to the users, as a new functionality or functionality

that helps the business grow.

And, I think, there's no one right approach.

I do have a personal preference for the kind of zero bugs, um, pyramid approach.

I really.

hate carrying stuff in the backlog, maybe I also hate backlogs,, uh,

there's another topic for another day, but managing bugs and so on.

But, I really think , if you're say you're going to do it , shouldn't keep

it on the long finger, as we say in Ireland, it's like you keep it somewhere

far away, reminding yourself that you need to do it, but you never get to it.

Um, so I really like, um, this kind of.

Pyramid prioritization approach, but I think

Péter (2): talk a bit about this, like, you're working with a system that has a

lot of, uh, ancient stuff, a lot of things that maybe you will decommission later,

You have 359 bugs in your ticket systems.

How do you implement this system where your foundation is always in a good place.

And before that solid, you don't do anything else on the upper levels.

Jeremy: So the ideal scenario for this is when you're building

like, in a startup, you start like this and you continue like this

and you're able to maintain the pyramid in a stable way.

Right?

Um, that's that's how the best , approach is, but , then in the situation you're

describing, which is a lot more legacy and debt that then you need to see this

as the direction you want to end up in, and you need to work your way towards

that, making those migrations and all the different choices, towards that, like

the scenario I was in where I use that example about the manual deployments.

And so we carved out time.

To get to deployments to be automated and, this is like an 18 months, two year

process to get to a different, more stable part and different bits of your system

will arrive at different points in time into that place where you can operate

with a zero bugs approach in that area.

Um, you also are going to have to have a tolerance for more known issues.

Potentially and just have more documented, like just the whole

way that you approach things.

You know, we talked about gardening and thinking in the future and stuff.

That's what you want senior people to do is map a route to this place.

What is not acceptable is staying in that,

um, rocky tech debt situation with lots of fires constantly happening.

Péter (2): Yeah, I like that you brought up the startup approach because they

seem like they have the ruxury of a greenfield project and they can set all

those foundation levels and pay attention.

But oftentimes startups.

say that our only goal is to survive till we can validate or

MVP or have a product market fit and everything else is secondary.

And I don't care about uptime.

I don't care about anything.

So those works against the foundation.

I think how you can resolve this.

This conflict is.

on making the foundation very minimal, what are the few handful of

key differentiators that you don't compromise on, for example, in a security

company, the protection of your users, data, whatever product experiments

you're doing in a desperate attempt to get some funding or validation.

You don't compromise that.

That's the foundation.

And if there are problems on that area, you throw away

everything and fix those first.

So I think a good way to implement this pyramid approach, which I really like

is to spend some time and don't get too optimistic about what you put on the

lower levels and just the key stuff.

Jeremy: Yeah, obviously, in that startup place the danger that

people do in those startup scenarios as they build too many things.

So I think you should outsource a lot to sass.

Péter (2): Yes.

Jeremy: And not try and build and run on.

I've written a blog post that about don't use Kubernetes in those

situations, unless there's a specific reason why you really need to use it.

Use, I don't know, fly.

io or one of these other platform as a service offerings, minimize what

you do, to focus on the value that you can really add, remove, outsource

all the undifferentiated heavy lifting at some point in the future.

If you're successful, you might need to bring some of those in house to

actually start to differentiate.

This is where a lot of tech debt is accumulated in companies

because they overbuild, in the early stages and they make their

surface area of the product too big.

Péter (2): Yeah,

Jeremy: Um, Uh, Uh,

Péter (2): founder shared their learnings.

It was amazing it was very honest and very good.

They were saying the same stuff.

Like You can focus on your core differentiator, be very

comfortable with manual work.

Also, like it was some, uh, sports race video app, whatever the founder

was manually editing videos for their subscribers, because they didn't want to

invest to any kind of automation there too early and the concept they brought

in was Similar to minimum viable product is the MVB, the minimum viable business.

And, and this is what the startup focuses on only to, to get the validation, try out

all the experiments and see what sticks.

Investing in a solid architecture, in this period is a waste of

time, 99 percent of the time.

Because you don't know what your

going to do.

Jeremy: So this is it.

Uh,

Péter (2): we went a bit

Jeremy: no, but I think it's really interesting because this loops

back to what you said at the very beginning describing tech debt and,

as we summarize things, I really like how you said past decisions.

And we talked about how the earlier and better the decisions you can make

the quality will be better and you'll have more options in the future.

And, um, so I'd, I'd love for you to kind of wrap up this whole thing now going

down to the engineering manager level.

Péter (2): Okay.

So, so takeaways, don't talk about tech debt in the engineering context, only talk

about it in the business context, make the business pain visible to stakeholders.

Understand that tech debt is okay.

Some level of tech debt, learn how to live with it.

have a plan how you're going to address it and how you're going to get out of it.

Um, data, if you're talking about data, things you measure, time spent at various

areas incident numbers, everything you're arguing about where you want

your team and your organization to be.

And not about what you believe is good code or bad code a healthy culture.

You should have disagreements in your teams, but you should resolve those

disagreements on the engineering side about what is really a tech debt and

what are your plans to address it.

And finally, some balanced approach to prioritization.

Don't.

Go into half a year, we stop every feature development kind of

refactorings because those very rarely work out well in my experience.

Jeremy: Yeah, just a tip, assign if you know you're doing it well, the product

manager is the one that prioritizes working on tech debt, because you've

made it visible enough the business is recognizing the need to invest in it.

Péter (2): I love that.

You have a good relationship with your PM and you manage to convince

them that it's going to hurt them if we don't address this like that, or

you manage to find a way to pair it with the feature work, then you won.

you can do it.

Yeah.

Jeremy: Yeah, brilliant.

Great topic, Peter.

Uh, thanks very much.

And to our listeners, thank you.

Would love some feedback and, , really appreciate some of the messages we've had

in from the, some of the last episodes.

Of course it would be great if you, fed, the algorithms, to help

us , be more visible, but more than anything, it would really

appreciate if you found this valuable.

That you would share it with others that you think might , appreciate

, what we're talking about today.

Péter (2): Yeah, thank you very much and see you or talk to you in two weeks.

Jeremy: Awesome.

Yeah.

Péter (2): Bye.