Péter: Hey everyone.
Welcome to the Retrospective, the engineering leadership podcast, where
we discuss topics about engineering management, technical leadership, and
similar areas today, as usual, with me is Jeremy and I'm Peter Sasz.
Welcome.
Jeremy: Welcome everybody.
Péter: What is today's topic Jeremy?
Jeremy: Yeah, so today I wanted to talk about a challenge that, I think
every engineering team faces, which is constantly being interrupted, context
switching, and I wanted to talk about some of the approaches that people
take and suggest my preferred approach based on some of the experience
I've had, which is this concept of a dedicated firefighter role in the team
that rotates regularly and handles all the interruptions for the team.
So we're going to try and to cover a little bit about how to implement a
firefighter role some of the pitfalls that people experience when they try and
do it and the best ways to be successful.
Péter: Awesome.
This is really a good topic.
I saw this working very well and I saw the concrete impact it had
on the focus performance and even external perception of the team.
So I'm really curious.
Let's dig into this.
Tell me a bit about unplanned work and interruptions.
Why are there so bad?
We are a team we need to be able to handle this.
Right.
Jeremy: Yeah, Yeah exactly.
It's a reality, right?
So you teams get bugs you have support contacting you you have other teams
in the organization who need something from you, or they have a request for,
for you to do something And people like me generate lots of slack messages.
I do I joke because like obviously slack is there.
It's an ever present.
Maybe some people still have email and use email and work too.
I would also say, you know So it's just there's just constant buzz at work and
some of this is actually really good.
Ultimately interruptions are you know, when you have a support
request you're helping customers, You need to unblock other teams.
Someone in your team needs a PR reviewed, or an external team
needs someone to review something that those are all good things.
When you have all these messages sharing info and stuff.
It's about getting everybody up to the same level of knowledge so
we can all make better decisions.
There is a good aspect to all of it.
I think we always jump to the negative of interruptions and blocking me or
unplanned stuff that's coming in.
There's a benefit to it, but there's also a cost, right?
And I think that's what we focused on.
We always want more deep work, and we want to be in, flow and all the rest.
And, but I think it's really hard and I think it's getting increasingly harder.
I remember early in my career it was mainly email and emails
weren't that frequent, now Slack and instant messages are non-stop!
I remember when hip chat became really big that was like the first moment in
work where this whole thing kicked off.
I think there's one study that I've often heard quoted, which is that every switch
you make has an impact of 23 minutes.
There was another one which I think is a really interesting
study by UC Irvine that over half of all tasks get interrupted.
And I think they also say that Any task that gets interrupted isn't very
likely to be finished in the same day.
And then there's the mental impact, stuff not getting finished cognitive
overhead, interruptions are very bad.
Péter: Yeah.
Tell me more about that, that study because it's, it's super interesting.
I think intuitively we all know the negative effects of interruptions.
We feel it on ourselves especially if interruptions get reinterrupted again,
and then you just don't even know where you started, but it's great to hear
that there are studies about that.
Tell me more about this.
Jeremy: Yeah.
I don't know.
Just on a personal level, when, we work in manager mode, you have the maker mode
and manager mode and part of manager mode is, dealing with interrupts and having
a schedule that's like really fragmented.
But yeah, even then we still have to produce stuff.
And what this study I found really interesting was that it, it said that
they, they did a study of people working in knowledge work and they
found that people were switching tasks every three minutes, which is scary.
Yeah.
When I look at myself, I'm wondering if I'm doing the same things.
And I know this bit is definitely true.
We interrupt ourselves more than others interrupt us.
So
Péter: Oh,
Jeremy: we handle interrupts as a team, we're still our own worst enemy.
Péter: I think it's like I don't want to be very gloomy and everything, but
I think today's age with all these micro contents and micro interruptions from
my phones are, are contributing to this in a, in a negative way, like Yeah.
Yeah.
I can see on people that not just the addiction aspect,
but the attention span aspect.
Like if a video is longer, and by long I mean 40 seconds instead of 10 seconds,
they don't watch it till the end.
I see the effects on myself also.
So all these interruptions are super, super draining.
Jeremy: Yeah.
I wonder if we ran this, if they ran this study again,
would the results be even worse?
So yeah, I have to say that I'm, I find I've definitely noticed my
ability to focus on long form is getting harder because of the amount
of short form stuff that we're getting
hit with.
Péter: We interrupted ourselves.
Let's, let's get back to this.
Jeremy: No, it's okay.
But what's, well, we both like feeling guilty about whatever
stuff we look at, on our phones!
What was interesting was and it's fairly obvious in a way, but the
more complex the task, the more the bigger it costs to recover from it.
They said the afternoon interruptions were hit harder.
The more people you work with in a team, obviously, which is obvious,
the more interruptions you experience.
So that's definitely another case for the small team approach and
another obvious one, but the more projects you have in flight, so
the more work in progress you have.
The bigger the impact.
And then it was interesting also was if you're interrupting yourself,
like we said, if I'm switching and then going back, there's actually
a faster recovery time in terms of being able to be back on task.
Whereas if you have an external interruption, it was a longer recovery.
And that if you're interrupted if you're in flow in a deep task, you're less
likely to get back or maybe it says, you might never get back to peak focus.
And yeah, so this is the thing I mentioned earlier, but only
40 percent of interrupted tasks get completed the same day.
Péter: That's a problem because I guess if you add the lost time and different
focus for the night then it's just so much harder to get back into the focus
the next day and finish the task.
Yeah.
Jeremy: Yeah.
Even if you're interrupted in the day just the ability to get back to that
deep focus to finish out the task.
And here's, this is the final interesting bit.
This is how we're all coping, right?
This is what I find interesting.
We all work faster.
And we're getting more stressed because of it and I can feel it, like I feel
more stressed today with everything that the way that the world feels
like it's spinning faster because you, and you have to step up your pace.
and Yeah, the most productive hours are early morning.
And that's true.
I find I'm up early and I can get stuff done in a focused way that
I struggle later in the day to do.
Péter: I'm curious to know if, if, if this is because you have
less external interruptions because others are still sleeping.
Jeremy: Yeah, exactly but even then I see people who get into work early, that's
their productive moments before then there's a buzz in the office or whatever.
The biggest way that we interrupt ourselves is checking email or messaging
where I guess in this case, Slack.
And I know that's the case for myself.
What I find interesting about all of this is there's two parts.
There's like a personal productivity and interrupting yourself, but
there's also the bigger cost
on a team that's trying to get projects done.
And it's trying to get big tasks done and they have interruptions
coming into that team.
And it's a huge cost to actually the productivity of the team.
Péter: Yeah.
That's, that's what I wanted to ask you about.
Like, like, I think we talked a bit about the personal aspect.
Let's elevate it to the team aspect and see what, what, how does the
worst case scenario look like?
Describe a team that's constantly interrupted.
Jeremy: They have a lot of projects in flight.
They're constantly being interrupted.
They're in a, what I would call like a reactive mode and they're not in charge
of their timeline and delivery anymore.
Quality of what they do is decreasing because they have to ship something
because there's, more pressure but they never had their deadlines struggling
with tech debt because it's a spiral.
You get worse and worse out of control.
And so you make things worse and worse over time that I think the energy in
the team and the morale decreases.
And obviously your external stakeholders and customers of the team start
to lose trust with you because of, The way the team is functioning.
It's, it can be really dire.
Things can turn out really bad if teams don't find a way to deal with this.
Péter: It's, it sounds like really a lose, lose situation.
And it's, it's, it's very sad and ironic because it's coming out of
a need to be better and respond to all the interruptions on time.
But in reality, the result is negative.
So how, how did you see team solve this problem?
Cause as you say it drains team energy, morality it errodes, customer trust.
So, so they know that this is not a good.
Place to be at.
What, what are the common approaches you've seen?
Mm
Jeremy: And in, it can work in the early stage, small team.
It can work in an early stage startup as well.
But it's essentially, that you don't really have a process team
just decides, you triage stuff and you prioritize as it comes in.
And it's if you think about your time, you have many slices every time an interrupt
comes and you have a, you try and have a fast, flexible way to handle stuff.
And it can work.
And this is, it can work at a certain level.
Especially if a team is taking on board some of the common advice that you are
given for handling interruptions, which generally work at this stage, which is
having some kind of focus time in the team where and where the, you try and
avoid having too many topics in flight for members and for the team and having
some kind of Way that some kind of framework for communicating with the
outside world, maybe a specific channel.
Maybe some kind of okay this is a bug.
It's not too serious.
I'm going to put it in the backlog and we'll do it in the
next sprint kind of approach.
Like all of this, having some kind of lightweight framework.
This is where teams.
Most teams sit and are most, like that's like the, almost the default mode.
And I think that works, right?
I think it works for a certain stage.
But it, I think the wheels come off quite, can come off quite
quickly as teams become bigger.
So you have we talked about the connections between the team
Great.
More interrupts.
If a business becomes more successful or your product is growing, you have
more users, more customers, more support requests more things going on
or you're part of a busy organization where there's just a lot going on, I
think then what you end up happening is that your backlog explodes.
Your team is constantly context switching your sprint start to,
if you're using sprints, you might not be meeting your sprint goals.
Or you'll see in if you're in Kanban mode, your stuff will get your tasks
will get stuck not be advancing and then you'll be bunching up.
You'll fall into all sorts of traps of trying to still push more and
ultimately you're not predictable.
Your delivery is not predictable anymore because of all those interruptions
that are sidetracking you all the time.
Péter: Yeah, yeah, yeah, this is a good summary.
The way I see this ad hoc approaches or early stage approach is that
It focuses a bit more on the individual than on the team.
Like it has some, some good ideas and prescriptions for individuals the focus
time, the limiting task shifts, but it doesn't have a holistic approach
to the whole problem of interruption.
So, but,
Jeremy: love that framing actually, I think that makes a lot of sense,
Péter: and I, I, I think.
I mean, I don't know what you're going to come up with, but I
think later approaches, I hope that they are a bit more holistic
and acknowledging approaching the interruption problem better.
What else did you see working in bigger teams or more successful
teams that were handling this?
Jeremy: yeah so then I think there's this kind of approach it's been
more popularized by Basecamp and shape up which is the cool down or a
dedicated sprint approach, essentially.
You try and ignore all of the majority of your interrupts and then
you handle them in a batch mode.
I see different variations of this one day a week or, every sprint, we have a
cool down after it or, whatever the kind of like in, for example, in shape up,
they have a six week, Sprints and then a cool down and then another six week.
Obviously I think there's an aspect of this that starts to
go to a really good principle, which is timeboxing and batching
And so it's a great idea.
Teams get desperate and they say we've got loads of bugs, so
let's stick it into a cool down.
And I've seen that, several times now.
Oh, that's a cool down task.
We'll do that.
And then the cool down starts to, it works, but I think there's
some things that break down.
First of all, the cool down can get over filled.
And, I think the other part is this does not work.
Where you need to respond quickly to something in a timely way.
Péter: Yeah, you cannot say to all requests that thanks
for your, your request.
I will handle it next week.
Jeremy: Yeah, exactly.
Bugs become outdated, right?
If, I have a bug and then I push it out by even two weeks, how can I reproduce it?
And just that context switch also the thing about bugs is the longer you
leave it to fix you, the longer you are from when you originally wrote the
code and the harder it is to switch
back into the context of that.
So at least for me, I feel like bugs should be, we want to shift, we want to
triage and fix bugs as soon as possible and get and fix them as early, shift left,
but fix them as early in the process.
I think those cooldowns can overflow and obviously the user
experience of this is not great.
We talk a little bit about it.
Péter: I mean, it's if you talk about user experience like the customer
perception of the team and the service the team provides, arguably there
there is an improvement because at least the team is more predictable
than than the ad hoc handling.
So the stakeholder might be upset that that his problem is only going
to be handled in two weeks, but that's that's something they can plan with.
As opposed to the ad hoc when one request gets handled immediately
and the other one in three months.
Jeremy: That's exactly it.
So you are trading off on being more predictable on delivery for
the responsiveness and so on.
But I think, and so this can work, and it does work.
And where you'll see it's not working is you have these crises that interrupt.
And I think, that your team with this mode is also going to struggle on that
user serve, the user responsiveness
So I think it's very hard to run in this mode and avoid interruption still,
Péter: Yeah.
Jeremy: because you end up, what you end up with is a combination
of cool down and ad hoc
to operate properly.
Péter: Guess it, it depends on, on, on the discipline and how well you can keep it.
But to your point, like if, if production is down, then, then you cannot just
push it to the cooldown period.
So, so yeah, whatever discipline you keep and however strong you
push back, there are always things, interruptions that are creeping through.
Okay.
So thanks for these approaches.
Tell us a bit more about the firefighter role because it seems
like that's a good compromise.
Jeremy: Honestly, this is the pattern that I found works really well, especially
when you have a bit more of an established organization regular, support, regular
things flowing into the team, or you're
part of a bigger organization.
And no matter how much we try to remove dependencies between teams
we still have to communicate.
And or especially and this is also.
An important role, I would say, for platform teams that provide a service
internally to other teams in organization, where they have both a build and a run
and I, most teams have some run element.
So the firefighter is a, it's taking, instead of having team interrupts, it's
having a rotating role in the team who is dedicated To handling all of those
interrupts it allows everybody else to stay focused on their work and, The
firefighter is doing all the run tasks.
They're handling any kind of production incidents, customer support requests
requests from other teams and so on.
And they're like the first line support triage before.
We do decide, and sometimes you still have to interrupt the team,
but they're definitely, they handle the majority of those things.
And then this is a really important part of this role, which I think teams get
wrong and where they fail is they don't do this on top of their normal work.
you have to plan for that person to not be in, for the period of
their rotation, you need to plan for them to be not doing any work.
And the work that they should be doing beyond all of those support requests is.
learning what causes the team to have those requests and
preventing them in the future.
So it might be fixing bugs, but it's more identifying patterns and then figuring
out, Oh, we need some better monitoring to avoid this, or I need to create some
quick tooling or something self service.
I'm going to improve the docs because we getting these requests.
This piece of tech debt is really.
Impacting us every time I'm going to work on this and ideally those
are more interruptible tasks because you're doing you know, you're the one
that's supposed to be interrupted.
But I think, this is a person that can work in that more ad hoc mode, but just
that one individual with a prioritization.
So anytime that teams are struggling to deal with interruptions, this model
has worked well consistently for me.
Péter: Hmm.
That's that's interesting.
Let's talk a bit about the practical aspects.
Like how, how do you implement something like this in a team?
What are the things to pay attention to?
And then maybe you will answer before me posing some of the
questions about the risks.
Hmm.
Mm
Jeremy: so I think the rotation is something, so when you're implementing, I
think there's some patterns for success.
I think the minimum.
of a rotation for the role is one week, though often two weeks is what people
do because you have enough time to actually tackle some of the debt stuff.
And essentially you need long enough time to build some context.
But it can't be so long that you get frustrated or burnt
out from all of the interrupts.
If you working in sprints, it makes sense to wrap it, to have it aligned
with the sprints that your team is doing, having the same cadence as the sprints.
And I think it's important to have some kind of handover
between the people who are, who's finishing and doing the next one.
Providing, maybe some kind of status on, on ongoing stuff if they're going beyond.
There's also having some kind of clear criteria for that person to, that you
develop over time, to escalate into the team if you can't handle things yourself.
Also if you're being overwhelmed by a lot of Issues, then that's also an
important part, by the way, if teams are, if that one person is being overwhelmed
all the time and every time, then I think the team has an issue that goes
beyond needing a firefighter role.
I think at that point you need to, maybe reconsider what is triggering a needing
so much that more than that, it takes more than one person to constantly handle those
Péter: Yeah, I, I think in this case is the person who is, who
cannot handle it because there are just too many interrupts.
I think that person is going to have a very good idea of how to make an
impactful improvement in the systems and, and just, just get into a better stage.
Maybe it's documentation, maybe it's setting up some self serve system.
Maybe it's just tech debt.
Yeah.
Thank you.
Yeah, that's
Jeremy: if the team can't handle that in the course of that firefighter time
slot, then they need to prioritize those tasks as part of the actual sprint or
put it in their, prioritize it in their backlog to make it a whole team focus.
And I think just the last thing is having some kind of tracking of the
kind of requests coming in is helpful to be able to spot the patterns and to
look back and be a bit more data driven.
I'm trying to say this in a way that I'm not just saying create tickets because
Péter: what I wanted to ask
Jeremy: that can be really.
Can be really evil and over heavyweight and so on, but yeah, some kind of
way to, to just understand what, all the requests that you have.
Péter: Well, what I like working is that the teams keep open.
They're usually Slack or some internal chat tool.
They're open to requests.
They have a dedicated room and the firefighter is immediately responding
that, okay, I will start to take a look at it and says, while I'm looking at
it or starting to address, can you just make sure to create a ticket for this?
You can link this chat, whatever, just because communicates that the issue is
not held up until the person creates all the administration, but still puts the
responsibility on the original requester to create the ticket and enforces this
culture that that's the culture of documentation being, being database.
Jeremy: Yeah.
Maybe I would say, I think that there are even better ways to
handle that now than say, I've got this, but please create a ticket.
I still think create a ticket is a horrible thing to
Péter: Yeah, yeah, I hear it.
Jeremy: And.
Now you can, if you're using Slack and I think most people are or, most,
I hope most people are hope you're not condemned to teams, but I think
you can do this with teams too, but
Péter: I think some of our listeners are so
Jeremy: yeah, sorry for you.
But what you can do is you can have emojis that you can tag on an issue on a Slack
message that triggers a workflow that creates a ticket in whatever in Jira or
linear or whatever tool you're using.
And And it just keeps the thread of that message in sync with your ticketing.
And then afterwards, you can do whatever triage you want.
And maybe some of your emojis that you can add can help like triage that.
But saying create a ticket for me is it's just not something
that I'm really a fan of,
you know.
Péter: really a service attitude.
Jeremy: Exactly.
It's not user friendly.
It's not user focused.
It's not customer focused.
So many rant over, but that, that's, I wish that people would
focus more on making it easier.
And especially if you're dealing with customer support or sales, it's such
a barrier for them, especially sales.
When they have requests they're running in the field.
Trying to chase customers usually not very technical and we need to meet them where
they are and not put straight jackets and processes around them, but that's maybe,
Péter: And I often saw the frustration of someone reaching out with something,
Hey, I noticed the bug in your systems.
It doesn't hold me back.
It's not a problem for me.
I'm just helping you out by letting you know and receiving a feedback saying
that, okay, create a ticket for that.
It's really frustrating and teaches the person not to reach out next time.
Okay, let's, let's, let's get out of this rabbit hole of communication
culture and just continue.
I really like this approach of, of the firefighter because externally.
It shows up like a 24 seven support always available approach.
And that can be really good for, for, for everyone interacting with the team.
And it also creates focus time for the people who are not on on, on the
firefighting role and they can do some deep work, what are the common pitfalls?
Why, why, how do you see this failing or what are the things that we are
trading for this better service?
Jeremy: Yeah.
So some things that I've seen so first of all, the trap of making the firefighter
a permanent part of somebody's job,
either, you know, like the best engineer because they know everything,
which is awful because you need to push like junior members of the team.
They might need some support.
Yeah.
but this is how you learn.
And, we need to take the training wheels off, but maybe it's a pairing,
but there needs to be a point where the junior people in the team need
to be able to handle this role.
The other trap that I see, which I really dislike also is the manager of the team.
Fulfills the role.
I think if you're a hands on manager, it's perfectly acceptable for you
to have a rotation and you should be having a rotation as a firefighter,
just like everybody else in the team.
But what I don't think it's good is if you Permanently do it.
I think I just think it's a real anti pattern.
The other one that this is where this implementation always fails
down when people think of it, and they make the firefighters still
have work in the sprint or whatever.
And this is like the, this is why all, a lot of this fails, because
you just overwhelm someone, and their work is still late because of all
the reasons we talked about earlier.
And then the other ones we briefly talked about not having a plan when
you have multiple things kicking off that are overwhelming the
individual and not having a handover.
And so you stuff falls in the cracks when you rotate and you don't
have a handover of the in flight.
Topics.
I think then customer service suffers as well, or what's worse is you take
it with you into your next sprint.
So some, you have to, if you're going to take it in, you need to plan it as work in
the next sprint that you're going to do.
But it needs to be really explicit and not like a side thing because,
oh, I was helping that person during my firefighting rotation and now
I'm just going to continue somehow.
Those are all the danger.
Danger.
Yeah.
Péter: juniors on on firefighter.
I saw this working amazingly in one of the places I work that there was this
this junior engineer who was Not very experienced, and she was afraid of this
role, but she really shown in this, this, this place because she had very
good soft skills and do help people who are reaching out with requests.
And it was very motivating for her to, to, to see the impact of, of her help can do.
And she balanced very well when to ask for help.
Actually, I think this is one of the risks.
If you put junior people in this role that they don't recognize.
The point where they should ask for help and not try to figure themselves out.
So if you teach them to be comfortable with not knowing things and asking for
help and how to escalate efficiently, then this can be an amazing, motivating
and learning experience for them.
Yeah,
Jeremy: podcast episode we did about what makes a senior engineer, because
this is exactly the skills that this is part of that growing of a junior
person into a senior and the kind of the skillset that they need to learn
in order to be to improve and grow.
So, yeah.
Péter: quick question before we summarize and say goodbye.
You mentioned the importance of knowledge transfer and to make sure
that handover is happening efficiently.
And I agree because I think one of the values in this firefighter situations
is that the internal knowledge is increasing in the team if it's done well.
So I'm curious, what are some practical ways you found working about
this handover to make it efficient?
Jeremy: I think the first is to discuss this in the team retrospective.
And ideally if you're doing this around some kind of sprint cadence of two weeks,
then you're having a retrospective.
And so it's an important part of building it into the retrospective of the team.
And I think then it's maybe obviously sharing what you're learning
continuously in the daily and so on, that's an important part of it.
And finally, When you're handing over it's probably not just a one to one,
but maybe like with the manager, you have a three way, the outgoing and
the incoming and maybe the manager.
I'm not sure I would have the whole team involved in
Péter: that's what I'm thinking about.
It, I guess it
Jeremy: you, that's, you cover that in the retrospective.
Yeah.
Sorry.
I would just cover that in the retro.
Yeah.
Péter: yeah, because I'm thinking if, if you're talking about a nine person
team, then, then I think you're right.
It's a waste of time or, or not the best way to spend the time, but if
it's a very small team, like three people, four people, then, then,
then maybe you can combine this and just talk about briefly what happened
during the firefighter session.
So everyone learns.
Jeremy: Yeah.
And usually the, you need the firefighter, the bigger the team goes.
So maybe in a three or four person team, you're less likely to need it and you're
on top of stuff, but and you may not need to dedicate a full time firefighter also.
I still think that even in small teams, this pattern works way better
because you have one person who's the point of contact for that week.
Péter: Yeah.
Yeah.
And to those who say that, Oh, but in a four person team having a dedicated
firefighter, it means that we are running at 75 percent capacity.
That's not true.
That's really not true because that 25 percent is super valuable work and
it creates the stability, resilience and scalability for the team by not
just answering requests, but also the little things you were talking about.
We didn't go deep into it, but.
This person, when it's not interrupted and just waiting to help this person
can do a lot of very impactful tech that work, tidying up a documentation,
refactoring something, looking into an obscure bug that the team never had time.
So yeah, push back.
If somebody tries to oppose this,
Jeremy: Yeah, plus you're just making the invisible visible by
having that dedicated capacity.
And a lot of our work is just about making stuff visible and visualized.
And if the true capacity of the team is that, then that it's just much better.
Péter: all right, we are nearing our end.
Let's wrap it up.
What are the key takeaways, how to handle interruptions in a team?
Jeremy: Interruptions are good.
Interruptions are bad.
And we, what we were trying to achieve is not to eliminate all of them, but
to handle them more efficiently and learn from the interruptions by
constantly and regularly improving how the team works and a great pattern for
that when ad hoc handling of incoming requests starts to break down is to
have a dedicated person in the team.
In this case, I call it the firefighter.
And they shield the whole team.
You regularly rotate that role and you have some kind of handover . And what
makes this work well is if during that rotation, you actually are doing
improvements For how the team works or how the team interfaces with others.
Things like documentation, automation, self service.
And obviously the team is just learning from what that person is doing.
To share knowledge inside the team of the interruptions to be
able to make bigger improvements.
Péter: Awesome.
Yeah.
Nice wrap up.
Very practical.
Thank you, Jeremy.
And to our listeners, if you have someone in your personal or
professional circle whose team is struggling with interruptions,
send them over this episode.
And see you in two weeks when we're going to tackle another
engineering leadership problem.
Jeremy: Yeah.
Brilliant.
And we'll put some links to some great resources in the show notes as well.