Péter (2): Hey everyone, welcome to the retrospective, the engineering
leadership podcast where we talk about topics for engineering manager that
they face in their day to day jobs.
me here is Jeremy and I'm, I'm Peter.
Let's get into it.
Jeremy: Hey Peter, it's great to be here today and I'm really enjoying
Season 2 and the topics we're covering.
Péter (2): Same.
Jeremy: So for the, for the listeners, just the way that we've been planning
this season is that we've been taking turns bringing a topic each.
So this week it's, it's your topic.
So, what are you bringing today for us to talk about?
Péter (2): I have a nice, beautiful surprise for you,
which is called TechDept.
think it's a favorite of a lot of engineers and engineering managers
and probably business leaders also, not necessarily in a good way.
We're not going to solve TechDept in 30 minutes, but what we can do is share
our combined experience on the field.
And, uh, yeah, let's talk about TechDept.
Yeah.
Jeremy: um, partly because in a lot of companies where there are,
I think we're in, you know, we, we hear the end of zero interest.
We hear a lot of focus on, companies that are having to do more with less.
There's a, there's a high pressure, especially in startup land and a
lot of pressure to ship more in a very competitive environment,
certainly that's something that we're, I'm experiencing in my job.
And, um, at the same time we have to manage this mystical
thing called tech debt.
And, uh, so I think it's a very timely topic for the, for the
period of time we're in right now.
Péter (2): Yeah, and another aspect why I think it's super
timely is AI assisted coding.
I'm pretty sure that in every companies there are a lot of code checked in
the code base that was written by or with the huge assistance of AI.
if it doesn't work so well, that's going to increase your tech debt.
Big way, and maybe not so obvious ways, uh, initially.
Jeremy: Actually any interesting anecdote, Google said that 25
percent of code at Google is now written by AI and, um, where are you?
Péter (2): like 25 percent of what, like, uh, yeah, sure.
The line breaks, the prints, the, the, the returns, the functions,
there is a lot of text in code that's, that could be easily automatized.
Jeremy: I think it's auto complete.
I think it's just fancy auto complete.
Um, and we've seen that.
Péter (2): underplay, it's super
Jeremy: Yeah,
Péter (2): but yeah,
Jeremy: yeah, we've seen that we're using an AI, , plugin, called Codium
and it gives us stats and it says it's done 40 percent of coding for us.
But I think most of it is just a fancy auto complete.
That you're going to be doing anyway, it's kind of like, autocomplete in, Grammarly
or in, I don't know, in Gmail when it's completing sentence this that you would
have written anyway, that's, yeah,
Péter (2): Yeah.
Saving time and as a side effect might increase your tech debt.
Jeremy: exactly because unintended things creep in, but, yeah, so maybe if you can
help us level set things a little bit, Peter, how would you define, tech debt?
Péter (2): Yeah.
Yeah.
I actually thought a lot about that and I came up with a long, uh, definition,
long sentence, but, uh, I will unpack it.
So my definition of tech debt is our best decisions, negative effect.
on the quality of our systems, limiting our options today.
There's three aspects to that, past decisions.
I wanted to capture that, uh, with tech, you created tech debt.
Tech debt is, is, is something that you live with today, but it was your past
decisions that resulted in some tech debt.
I want to start here by explaining that,
Jeremy: Um,
Péter (2): code health in, in, in a lot of different ways.
And systems can mean anything, technical, your code, your infrastructure, maybe
even your processes, CI, CD pipelines,
Jeremy: Um, Um,
Péter (2): options.
You can have, uh, you cannot cut costs so easily.
You cannot pivot to a different feature.
You cannot re architecture your system to today's needs so easily.
So it's basically less less options you have in in in what you're doing.
Jeremy: Yeah, I really like that definition, Peter, because yeah, we, we
do need to on purpose take decisions.
And in fact, tech debt can accumulate from very sensible decisions where we
choose to, be pragmatic architectures, not overbuilding for the future.
And then over time as things scale and grow, we need to constantly adjust.
For that.
Péter (2): that you don't want to miss and you cannot have, have it all.
It's always a compromise.
So you will need to compromise on the quality and increase
the tech depth of your systems.
I like the mortgage analogy taking out a big loan to be able to buy a
house where you can live rent free.
Oftentimes it's a good decision, arguably, and, it limits your options today.
You, you cannot, maybe you cannot buy a new car for a year now.
Maybe you cannot go on a big vacation for a while because you keep on have to
pay back, uh, this monthly installments.
So yeah, tech deft is not necessarily bad.
important thing is it should be conscious.
The result of conscious Decisions and not just happenstance and accidents.
Jeremy: Yeah, yeah, I'm thinking about this also from explaining
it to someone in the business.
Think your mortgage analogy can work really well.
I think another way to describe it is, you know, you used an example earlier
where we We try, cut some corners to try and get something out the door fast.
I was thinking about this one where you have a plumber that, comes and fits a new
sink and they don't actually have all the pipes, so they fit something in cardboard.
That's very weird analogy, but we do do this in tech, um, but if
it's some cardboard pipes and some tape stuff up and that can work.
Potentially temporarily, um, maybe not for too long, but if you increase
the pressure or if you let it run for too long, then that's going to fail.
And oftentimes that's the kind of tech debt that we, we do.
We build something just enough to ship, but then that cardboard needs to be
replaced, and put in real pipes and so on.
And that's again, again, the kind of classical approach
to tech debt that we have.
You know,
Péter (2): Yeah.
That's, that's, that's a good analogy.
Cause cause Using cardboard piping, whatever that is, it's, it's still
better than having no pipes at all and having water coming off
your sink to your feet and, and
Jeremy: yeah, or potentially a sink that doesn't work for a bit,
but you really need it, you know?
Péter (2): yeah, yeah, exactly.
Jeremy: Yeah.
Okay.
Have you come across any other good ways to explain this to the business?
Because I think one of the big challenges with tech debt is just
actually explaining the concept to business stakeholders and even users.
Honestly, sometimes we have to explain to users why things are taking longer.
Péter (2): any approach works that connect the limited options that I
was talking about in the definition to the business, business dictionary.
We have limited options in quality, stakeholders will feel the pain when
it takes two days to get back from an incident as a result of some tech
debt in your processes on pipelines or infrastructure, for example.
Stakeholders don't care about, the code health, the duplications, the redundancy,
everything that makes code problematic.
But they do care about quality.
They do care about costs.
They do care about, uh, user satisfaction.
You as an engineering manager, the listener of this podcast, your
job is to translate between these stakeholders and engineering, and make
the connection the impact of that in in the business goals and outcomes.
Jeremy: Yeah, I really like that.
Just connect the lack of options you have today and the negative effect that
past decisions had on the quality and explain that using different analogies.
Péter (2): Yeah.
Yeah.
Jeremy: Yeah.
Péter (2): If the business is in a big cost cutting exercise.
You can tie your tech debt initiative to that and focus on the infrastructure.
Maybe, uh, where you can decommission some old services by moving them to
two newer ones and save some money.
It's, it's, it's not a fight.
It's a discussion between, between the business and the engineering side.
Jeremy: Yeah, I agree.
So how as a team, when you're a team, how do you tell when
you have too much tech debt?
How do you measure, how do you detect it?
Like, what's, what's your, what are your thoughts on, on that?
Péter (2): Yeah.
I really like that you use the word measure because talking about data
is, uh, much more productive than talking about feelings and beliefs.
I'm pretty sure that there are a lot of engineers who are frustrated
about tech debt with the code they work with, but some of those people
will probably struggle if they have to point to some numbers, what they
mean by tech that, uh, exactly.
So fortunately we have a lot of, good metrics in the industry that
can be used and satisfy the need to connect this assessment to business.
What, what we were talking about, metrics can, can come as hand a pretty big
industry standard 2024, when we are.
Talking about this, especially metrics like the lead time for changes or
or failure rate or something that can tell a lot about the system.
The time to restore service.
I really like this metric because there are a lot of things that can impact this
metric and a lot of those are tied to tech depths in various parts of your
system's infrastructure code or processes.
So you can use Dora metrics, you can be more direct and measure some
quality metrics number of bugs or number of new bugs creating the last
three months, response times, of your systems, like more performance metrics.
It really depends on the organization.
What are the goals of your organization?
What is the business trying to achieve and finding those metrics and , bringing
in some depth, uh, aspects can, can help.
Um,
Jeremy: I think the other part for me is, is toil, Not fun stuff that you have to do
and I have a really great example of this.
There was a infrastructure engineer, in one of the companies I worked
at, where we had a very manual process to deploy the application.
And he recorded a video of what the deployment process was, and it was
over 300 clicks and, manual things taken to get a whole thing live.
And obviously that was to explain what was needed to be done and to
show very vividly the tech debt.
And obviously we invested heavily in automation and, we got to the point
where it was just click a button.
The whole thing was done automatically.
So
Péter (2): I really love that story.
Uh, the tool this engineer was using recording themselves doing this actual
process, it's very powerful and yeah, I don't need to say it, but, uh, those
are 300 ways this process can break.
I'm pretty sure.
There were a few occasions when some of those broken.
Jeremy: all the time, because how do you remember and that process was
already an improvement because the original one, when that company was
acquired, the first time it was all deployed from one person's laptop.
So at least
Péter (2): wow.
Jeremy: the process that he videoed was being able to be
run from multiple places, but,
Péter (2): you virtualize that guy's laptop?
Jeremy: No, but, uh, yeah, I mean, you just invest.
Yeah, I think sometimes it's really, really obvious.
It's like the elephant, um, and you know, sometimes it's, it's less obvious.
So depending on the situation you have different kind of ways
of checking and tracking it.
Péter (2): It's important.
You say sometimes it's really obvious.
Of course it is for an engineer.
And I would say even for a non engineered leader seeing that video of
the 300 steps that it takes to deploy the application, it's very obvious.
They will say, Oh, my God, I feel your pain.
I promise you we are done with this sales cycle.
We are done with the next product launch.
You can, but they don't Actually feel the impact on the business because
what they are walking away from the demonstration is that, Oh, poor Joe.
He has to go through 300 steps every day.
That's not the impact.
The impact is all the ways this can break and, and all the ways you will
struggle to release a new feature when, when something is changing this process.
And that's what the business feels that the users don't get the new feature.
Features take longer.
Features are buggy incidents are happening.
So I would say even when it's obvious.
Think about the users of your systems and the business stakeholders and
make it obvious for them the pain, what the option limitations are.
Jeremy: And actually, in this case, the users were being impacted because things
would go wrong in that process, as you say, and there were regularly fires.
So that was easy.
It was an easy sell.
Péter (2): yeah, yeah.
Okay.
Jeremy: Those are the obvious situations, but what about in a situation where
it's, it's less obvious and there's more of a disagreement about tech debt.
How do you come to an agreement then inside a team or between a team and
its stakeholders that you need to invest on tech debt or that it's
acceptable to leave it as it is.
Péter (2): That's a really good point because because we were talking about
convincing the business, but oftentimes you need to convince your engineering
peers that we should address this type that and not that type that.
What you can do is I like to think in teams and not in individuals.
So in this case, I would suggest a practice something like attack
that tour like every other sprint or some regular occurrences.
You pick a different engineer and they hold a short 15 minute walkthrough of
the part of the code base or the part of the processor or infrastructure
that they are most concerned about.
They explain just like this 300 step video.
That's a perfect example for
Jeremy: Silence.
Péter (2): base.
Maybe we could address this together.
So elevating the understanding and the knowledge of the team, with these
regular tech debt tours, I think it can be a good first step to, to create
alignment on the engineering side, that this is the most impactful or most
urgent thing that we want to address.
In the next time, and then you can tie to the business goals and
impact.
Jeremy: no, for me, I think it's really important when we talk about the team here
that the product manager is involved in that because, , often I think engineers
can quickly see and agree on on things when it's made more visible like that.
But product manager it's probably caring less about the tech debt
and they still have a lot of the concerns of the business.
And , it's super important that they understand the impact on the
team to be able to protect the team when you take a decision to work
on tech debt and, um, they need to be part of that, group as well.
Péter (2): I totally agree.
And thanks for bringing this up.
I'm a bit biased because I was leading platform engineering teams most recently.
And I didn't have product managers in my teams.
I had a very good product partner on the level, but it's super important
what you're saying to include the product side in this alignment off of
seeing the problem, because oftentimes when priorities are decided, you're
not even going to be in that room.
It's the product manager of the team who's going to have to fight for for
the engineering side also to make the pain visible and get some leadership
buying and stakeholder buying to address.
Some of the tech devs, so definitely include the PM in these discussions.
Jeremy: Yeah.
And I think, um, going back to the point you made earlier about
data and measuring and so on.
The data bit is really important.
In the past, , we've done things like measuring the types of work or the
signals of not having invested on tech debt, like whether it's an increase in
bugs, , and some of the other metrics that you mentioned earlier, but I
think those are, those are important signals as part of this discussion about
aligning on, on tech debt and what to do.
Péter (2): they can be very powerful because they bring the discussion to
principles to how we want to operate.
For example, if you manage to track where you're spending your time, is it feature
development, unplanned work, and then the organization can have an ideal and
say, okay, I want to spend numbers 80 percent of time on feature development
and 20 percent or everything else.
And you come back with metrics and say in the last two quarters, we could only spend
60 percent of time on feature development.
And that's the impact of the tech that we are having.
So we would like to double down on that in the next quarter so we can get this amount
of time closer to the ideal 80 percent that we set as a goal as an organization.
Again, you're bringing the discussion.
to talk about interpreting data and impacting data and not about
philosophies like code should be beautiful and, uh, don't know, or,
of thing approaches that are not useful in business discussions.
Jeremy: Exactly.
And I will say the classic thing that a product manager or a business person,
uh, will do is say, listen, I just need you to deliver these features.
And, uh,
I promise you, I just need it for this next six months because it's super
crucial for our business right now.
And then we can do some of those other projects that you have that you talked
about, but no, we can't do those.
And we need to do this.
And then six months later.
Business, still in a crunch and the platform starts to burn
more, but you need to continue.
So that starts to make me think about, what are good strategies
for managing tech debt?
Are there different approaches?
What would you recommend, Peter?
Peter.
Silence.
Péter (2): but your, your house is not on fire.
So this is, this is a critical stage because you need to maintain this
and maybe gradually improve this.
One approach I find useful in these situations is pairing tech
dev work with feature development.
It's basically, uh, you know, there is this, I think it's from
the Boy Scouts that, leave your camping site cleaner than you found.
I think it applies to code and everything as an engineering team touch.
Like, uh, you develop a feature about, I don't know, uh, newsletters, then take a
look at how email sending is implemented and maybe refactor it slightly.
To, to make it healthier, do some small refactors that are
adapting it better to today's needs compared to yesterday's needs.
So pairing this work with feature development has two big benefits.
One is that you're in the context and mindset of that current feature.
You don't
to dig through documentation.
What is it that we are doing here?
Oftentimes.
is your documentation and nobody knows why we are doing things the
way we are doing it, but that's how it's implemented in the code.
If you're touching a feature like this that has a lot of historical parts
implemented, you can discuss with your product manager and remove some of those
while doing the feature development.
Jeremy: You
Péter (2): bit longer for, for the completion of the
feature, because you keep
Jeremy: Silence.
Péter (2): a lot of discipline, but it can be very efficient.
Jeremy: Yeah, I think the discipline is to let the scope creep of the
tech debt refactoring overtake too much that the situation.
Um, but I mean, I've, I've seen a really good example of that
recently where a staff engineer was working in the code base.
They found.
Some part of it that had a lot of toil to update versions.
And so they improved how we used the renovate bot to
better do that in the future.
classic fixing debt as you go approach.
Péter (2): Yeah, now you mentioned risks and it's very important.
Getting carried away on and refactoring stuff that are
not necessarily in the scope.
And, and then you didn't deliver the feature, but you're already
two weeks late and the other risk I would say is premature optimization.
It's when you think that, oh, there is this new feature newsletter sending,
maybe I should have a separate service that handles emails and maybe I should
have it done with an auto scaling microservices infrastructure and then you
start working on those and then Next week, the product decision is that actually,
which is going to use a third party for newsletters because it scales better
and all those work, all those premature optimizations was a waste of time.
So
Jeremy: Exactly.
Péter (2): Only solve today's problems.
But still paired the tech depth work with feature development.
Jeremy: It's really funny because I really like this example.
You just used the newsletter, sending thing because that's a
really good pragmatic decision.
It should be made like that.
Most situations.
Um, but the funny thing is that might become tech debt later
when you realize that, um, Yeah.
I know we're not going to use that external service and we are going,
we're going to need to scale it more.
And that's perfect.
You know, that, then that's a good moment to work on, on changing it.
So that's just a good example of how
choice now.
And then it's just a loop, you improve later.
You'll need to improve it later.
Péter (2): Yeah, you cannot solve tomorrow's problems because you
don't know tomorrow's context.
This is exactly why you need to deliver stuff quickly to get the
validation and feedback from your users.
And those information will affect your decisions tomorrow.
And maybe you will double down on in house built infrastructure.
Maybe you will pivot to external.
don't know.
Nobody knows.
That's the point.
You just solve today's problems.
Jeremy: Exactly.
Now those are all the kind of like the small ongoing things.
Péter (2): the big one.
Jeremy: Yeah.
Sometimes you have something really big, like that example of
the manual deployment process or others, how do you handle those?
Péter (2): I read it in, I think in one of Will Larson's books is that
the only solid, reliable way in these situations is a migration to a new system.
There is a point where your old code base, your old system is just
so Hard to work with so ridden with tech debt that it's easier to throw
everything out and start from scratch.
Now, this is not how you should do it.
And we can talk about how you should migrate to a new system.
But the goal should be this.
You leave behind the old mess because you save a lot of time trying to
discover why is it working like that?
What are the features that are still needed?
What is not?
It's, it's easier to move on to a new system from, from scratch.
Jeremy: So I have two questions when you say that one is how do I
know that I need to migrate because I've seen cases where people have
migrated or doing a rewrite when they absolutely did not need to do it.
Okay.
Péter (2): wrote their ideal to do application or commenting system
and everything and you don't, yeah, I don't like the rewrite word, but
To answer your question, um, this is why we have good engineers.
And, I hear myself that this is not a satisfactory answer, but, uh, there
are points where you can see that the, the, the system needs to move on, maybe
actually, maybe the way I can answer this is the technique of how I think
you should do this kind of migrations.
You know, the strangler fig pattern, uh, those of us who are
listening, Might not be familiar.
It's the concept is coming from biology, uh, about some kind of plants.
The point of it is that you isolate parts in your system.
You have a big, complex, interconnected system and you isolate a part
in it that has less connections, ideally with the rest of the system.
You make those connections very clear and, uh, this isolation, within this
isolated part of the system, you do some refactoring that the rest of
the system doesn't see happening.
So if you take this approach, what you get is that, you can
basically work in a live system.
You avoid this big boom migration when you say that, okay, on Monday we start a new
system, we shot down the old one and there is two weeks off of fires everywhere.
You have a much more scalable and reliable migration process.
You also increase.
confidence in the quality of the system, because isolating those connections
between the system you selected and the rest of it, the best way to do it is
to write a lot of tests to ensure that your system is still returning the stuff
that the rest of it is, is requiring.
And once you're done with that, you have those tests in your CI CD
pipelines every time you make a change.
And then maybe you don't even need to decide if it's just a small
refactoring, or if it's an entire rewrite, because it's rarely very binary.
Oftentimes it's a scale.
So maybe you can do like you isolated one system, you rewrote
it in the modern, uh, way that you want to isolate another rewrote it.
And you reach a point where 80 percent of your system is, is, uh, called modern.
I don't want to get into deeper.
What is that?
But you see what I mean?
And 20 percent is still the old one.
And maybe you're fine with that.
Like maybe that 20%, it doesn't worth the effort to address the
majority of your system is fine.
And maybe you will decommission it in two years or, or whatever.
So actually, I think this is a good way to look at this instead of rewrite everything
from scratch, you take parts isolated and migrate those little parts one by one.
Keep on evaluating if.
You still need to pay attention to tag that or you reach the
point where it's okay to leave it.
Jeremy: Yeah, I think, I can probably list some bad reasons
why you shouldn't do it.
I have some examples.
There was one stage we had, um, A PHP, um, backend with a,
a modern JavaScript front end.
The, the PHP backend was actually very well written.
Structured a lot of tests, thousands of tests, things
were fast, CI was really good.
And the new , CTO.
Who knew Python, but didn't know PHP , he didn't land and there were challenges
with that PHP backend, but, uh, he decided that he wanted to re write, the backend
in Python and, that's a total reskilling of it and then, it was really funny
was the CTO decided that they didn't like view, which the front end was in.
So we did the front end and react., and, uh, you know those are the
wrong reasons to rewrite things.
I mean, There could be some valid business reasons to do that, but in those cases,
I really felt like, having known that system and worked in it and then seeing
those decisions after, um, I lived with things that I felt were suboptimal,
but I focused on the business value.
Whereas, those rewrites generated very little value for
the business, and obviously distracted the team significantly.
Péter (2): This is why it's very useful to back to our earlier point
is that you talk about tech that you should talk about the options.
It's limiting you and the business outcomes and how it's painful and
hurtful for the business to live with this tech that every day.
In your example, maybe I'm a Overassuming.
But, uh, I think what your team was optimizing for is the comfort
of the CTO with the tech stack.
And that's not something you spend months and months off developers time
and engineering time to optimize for.
And switching Both front end and back end, the wall stack, it's
super risky and super expensive.
And sometimes that's a good solution.
Like your product was pivoting a lot and you're doing a very
differently, then maybe that's a good opportunity to change that also.
Jeremy: I do think there has to be a real business value
justification for changing text.
Péter (2): be everything that should drive these decisions.
Cause ultimately the goal of the company is to make money, everything
else is just a means to those and
Jeremy: Exactly.
So thinking more , tactically , our podcast is really targeting
engineering managers.
Um, what.
Péter (2): there.
Jeremy: Yeah.
What advice or, and, how would you recommend engineers decide, what
tech debt to tackle in the sprint they're planning at the moment, or,
you know, the, the next month ahead, how should they prioritize that?
Péter (2): yeah, that's, I, I like that lot of stuff that you
don't mention in this question, which is implied, which is great.
Like, uh, is a healthy process of every sprint tackling some of the tech debt.
And this is what I want to make explicit that that's a way for a
sustainable tech debt management.
And the other thing that you're implying is that there is.
Engineering understanding of tasks that we need to tackle.
So, so this is a great point and now comes the big question.
Okay.
Which one of the fires do we turn our attention to?
And, it's basically a decision making exercise and there are
a lot of tools that can work.
One very simple one that I can explain and I think is adaptable.
Well, here is the Eisenhower matrix for decision making, you know, it's a
two by two, um, Grid on the horizontal axes, you have urgency on the vertical,
you have, uh, impact, and then you can group your tasks into these
four quadrants, you're going to have urgent, but not impactful, tasks.
You're going to have impactful, but not urgent tasks, and
then you're going to have.
and impactful and not urgent, not impactful.
My point is that you should have a portfolio approach
because the big risk is.
you want to avoid that you always pay attention to the urgent
stuff because you feel the pain now and you always deprioritize
the impactful non urgent ones.
Jeremy: Okay.
Péter (2): have some new API commands.
And we're going to have to.
deprecate some of the current ones that you're using.
It's not urgent at all.
You have a year to think about this, but it's going to be super impactful
if you don't meet these deadlines and you're not ready with your API by then.
So my advice is a portfolio approach.
Make sure that for example, every quarter you tackle some of the.
Impactful tasks that you have and then balance it with the urgent ones.
Also, this framework helps to realize that there are some tabs that are
not urgent and not impactful, I don't know, code styling, tabs
versus spaces, this kind of stuff.
Maybe you can just defer them to a hackathon or, some, backlog , of
we might address one day.
Uh, not, not put too much effort in it.
Jeremy: I like that.
And I think there's this kind of balancing approach that you described,
between features and tech debt and , choosing which ones to do, and
there's another approach, which is, like the kind of Maslow's, , hierarchy
of needs , pyramid approach, which.
Pairs well with kind of a zero bugs approach, which is essentially on the
bottom of your foundation is things like security and compliance and so on.
So anytime that you have something that, you know, is not secure or
updating a library or whatever, you do those, all of those things first,
and then you do things that affect reliability and stability performance.
And then it's only when your pyramid is stable, and then you do the top parts,
the, the delivering value to the users, as a new functionality or functionality
that helps the business grow.
And, I think, there's no one right approach.
I do have a personal preference for the kind of zero bugs, um, pyramid approach.
I really.
hate carrying stuff in the backlog, maybe I also hate backlogs,, uh,
there's another topic for another day, but managing bugs and so on.
But, I really think , if you're say you're going to do it , shouldn't keep
it on the long finger, as we say in Ireland, it's like you keep it somewhere
far away, reminding yourself that you need to do it, but you never get to it.
Um, so I really like, um, this kind of.
Pyramid prioritization approach, but I think
Péter (2): talk a bit about this, like, you're working with a system that has a
lot of, uh, ancient stuff, a lot of things that maybe you will decommission later,
You have 359 bugs in your ticket systems.
How do you implement this system where your foundation is always in a good place.
And before that solid, you don't do anything else on the upper levels.
Jeremy: So the ideal scenario for this is when you're building
like, in a startup, you start like this and you continue like this
and you're able to maintain the pyramid in a stable way.
Right?
Um, that's that's how the best , approach is, but , then in the situation you're
describing, which is a lot more legacy and debt that then you need to see this
as the direction you want to end up in, and you need to work your way towards
that, making those migrations and all the different choices, towards that, like
the scenario I was in where I use that example about the manual deployments.
And so we carved out time.
To get to deployments to be automated and, this is like an 18 months, two year
process to get to a different, more stable part and different bits of your system
will arrive at different points in time into that place where you can operate
with a zero bugs approach in that area.
Um, you also are going to have to have a tolerance for more known issues.
Potentially and just have more documented, like just the whole
way that you approach things.
You know, we talked about gardening and thinking in the future and stuff.
That's what you want senior people to do is map a route to this place.
What is not acceptable is staying in that,
um, rocky tech debt situation with lots of fires constantly happening.
Péter (2): Yeah, I like that you brought up the startup approach because they
seem like they have the ruxury of a greenfield project and they can set all
those foundation levels and pay attention.
But oftentimes startups.
say that our only goal is to survive till we can validate or
MVP or have a product market fit and everything else is secondary.
And I don't care about uptime.
I don't care about anything.
So those works against the foundation.
I think how you can resolve this.
This conflict is.
on making the foundation very minimal, what are the few handful of
key differentiators that you don't compromise on, for example, in a security
company, the protection of your users, data, whatever product experiments
you're doing in a desperate attempt to get some funding or validation.
You don't compromise that.
That's the foundation.
And if there are problems on that area, you throw away
everything and fix those first.
So I think a good way to implement this pyramid approach, which I really like
is to spend some time and don't get too optimistic about what you put on the
lower levels and just the key stuff.
Jeremy: Yeah, obviously, in that startup place the danger that
people do in those startup scenarios as they build too many things.
So I think you should outsource a lot to sass.
Péter (2): Yes.
Jeremy: And not try and build and run on.
I've written a blog post that about don't use Kubernetes in those
situations, unless there's a specific reason why you really need to use it.
Use, I don't know, fly.
io or one of these other platform as a service offerings, minimize what
you do, to focus on the value that you can really add, remove, outsource
all the undifferentiated heavy lifting at some point in the future.
If you're successful, you might need to bring some of those in house to
actually start to differentiate.
This is where a lot of tech debt is accumulated in companies
because they overbuild, in the early stages and they make their
surface area of the product too big.
Péter (2): Yeah,
Jeremy: Um, Uh, Uh,
Péter (2): founder shared their learnings.
It was amazing it was very honest and very good.
They were saying the same stuff.
Like You can focus on your core differentiator, be very
comfortable with manual work.
Also, like it was some, uh, sports race video app, whatever the founder
was manually editing videos for their subscribers, because they didn't want to
invest to any kind of automation there too early and the concept they brought
in was Similar to minimum viable product is the MVB, the minimum viable business.
And, and this is what the startup focuses on only to, to get the validation, try out
all the experiments and see what sticks.
Investing in a solid architecture, in this period is a waste of
time, 99 percent of the time.
Because you don't know what your
going to do.
Jeremy: So this is it.
Uh,
Péter (2): we went a bit
Jeremy: no, but I think it's really interesting because this loops
back to what you said at the very beginning describing tech debt and,
as we summarize things, I really like how you said past decisions.
And we talked about how the earlier and better the decisions you can make
the quality will be better and you'll have more options in the future.
And, um, so I'd, I'd love for you to kind of wrap up this whole thing now going
down to the engineering manager level.
Péter (2): Okay.
So, so takeaways, don't talk about tech debt in the engineering context, only talk
about it in the business context, make the business pain visible to stakeholders.
Understand that tech debt is okay.
Some level of tech debt, learn how to live with it.
have a plan how you're going to address it and how you're going to get out of it.
Um, data, if you're talking about data, things you measure, time spent at various
areas incident numbers, everything you're arguing about where you want
your team and your organization to be.
And not about what you believe is good code or bad code a healthy culture.
You should have disagreements in your teams, but you should resolve those
disagreements on the engineering side about what is really a tech debt and
what are your plans to address it.
And finally, some balanced approach to prioritization.
Don't.
Go into half a year, we stop every feature development kind of
refactorings because those very rarely work out well in my experience.
Jeremy: Yeah, just a tip, assign if you know you're doing it well, the product
manager is the one that prioritizes working on tech debt, because you've
made it visible enough the business is recognizing the need to invest in it.
Péter (2): I love that.
You have a good relationship with your PM and you manage to convince
them that it's going to hurt them if we don't address this like that, or
you manage to find a way to pair it with the feature work, then you won.
you can do it.
Yeah.
Jeremy: Yeah, brilliant.
Great topic, Peter.
Uh, thanks very much.
And to our listeners, thank you.
Would love some feedback and, , really appreciate some of the messages we've had
in from the, some of the last episodes.
Of course it would be great if you, fed, the algorithms, to help
us , be more visible, but more than anything, it would really
appreciate if you found this valuable.
That you would share it with others that you think might , appreciate
, what we're talking about today.
Péter (2): Yeah, thank you very much and see you or talk to you in two weeks.
Jeremy: Awesome.
Yeah.
Péter (2): Bye.