Use the Right Lever for Successful Change

TLDR;

There are four levers you can use in making changes in your organization. Each of these levers is longer than the one before and provides greater leverage. Choosing the lever with the greatest leverage possible for a given problem improves your ability to solve that problem. These four levers of change are, in order of leverage provided; Strategy, Structure, People, and Process. Using a lever that is too short will be much less effective than using a longer lever. Choose the right lever to effect change.

The Past

Once upon a time, medieval aristocrats considered farmers and serfs to be lazy and stupid. They didn't try to better themselves, they didn't try to get ahead in life, and they didn't try to solve perennial problems. If you have issues in your organization, thoughts like this may have passed through your mind once or twice.

Unsurprisingly, medieval farmers were neither lazy nor stupid. They optimized for the environment in which they lived, and they did so brilliantly. That optimization was not apparent to those looking in from the outside. They focused on risk mitigation rather than production. There was no way to store value, so they invested in their relationship with peers. Even if they discovered a way to generate additional wealth, they were often legally restricted to the roles into which they were born.

Their environment, both social and technological, shaped their actions. If you wanted to change their actions, you need to change the social and technological environment in which they existed.

A Survey

Let me ask you a few questions. Take a moment and answer them and write down your answers. We are going to work through them at the end of this article.

• Do you often have features delayed because a specific developer is on vacation? Susan is on vacation. She owns area x of the codebase, so we have to wait until she gets back to finish that.
• Do you often have a release take days or weeks to get into production even after the team claimed it was 'done'? We were dev done, but we found many problems when we got it into the integration environment.
• Do you often have to rollback releases immediately after getting them into production? "There are a bunch of areas that we can't test, so we don't find them until things are in production."
• Have you not delivered anything significant to production in weeks or months? "The backend team finished the work, but the front end team is still working, and DevOps hasn't had a chance to look at it yet."
• Do initiatives look like they are doing fine right up until the final week or two before delivery before you get the terrible news that dates are going to slip by weeks or months? "We thought we still had plenty of time to catch up, but then this other thing happened, and now there is no way to deliver on time."
• Do you often hear 'If we just rewrite this <insert large system> it would solve our delivery problems' from your engineering leaders? "We messed up in X's design. Every time we touch it, we break it more. We need to rewrite it from scratch".
• Do you have the same two or three engineers jumping in to solve every production problem? "Joe jumped in again last night to fix the problem. It was great. None of the other engineers even had to wake up".
• Are you terrified of losing your 'star performers' because you believe that the wheels will come off the business if they leave? "Joe is the only one that knows everything. If we lose him, we are so screwed".

These types of problems often feel unsolvable. Worse, everything you try fails. You end up living with them. You continue to attack them, trying different solutions that are all variations on the same theme.  You may even go out and try to hire a savior, but somehow the problems remain. These problems are massive, and you are not using the right lever. Just like with those medieval farmers, you need to change the environment to change behavior.

The Four Levers

Give me a place to stand, and with a lever, I will move the whole world. -Archimedes

There is a set of four levers that you can use to shift your organization. Each of these levers is 'longer,'. Each provides greater leverage than the next. They consist of changes to  Strategy, Structure, People, and Process. Choosing the right lever, the right place to make changes, is critical to those changes' success.

Lever 4: Process

Unfortunately, the longer the lever, the harder it is to use. Process, the shortest of the levers, is the easiest to use. Changes to Processes tend to be uncontroversial. Those changes don't affect self-worth or livelihood. They are safe. They are also ineffective. That makes them a Honey Pot for leaders trying to improve the organization. You or someone in your organization thinks of a nice process improvement that would make everything better if people just followed it. You implement the Process, and nothing changes. Rinse and repeat ad-infinitum. You feel like you are 'doing something,' but you are not.

Think of Process as a refinement technique.  It can improve things that are already working, but it can't fix things that are not working.

Lever 3: People

Having the right people in the right places doing the right things is critical to success. The longer you leave the wrong people in place, the worse things are going to be. I once had a brilliant engineer working for me that had some social issues. At some point in his career, he decided that the best way to mentor is coworkers was to challenge them. He felt that forcing his coworkers to defend their thoughts helped them grow and encouraged them to think through things thoroughly. His coworkers felt attacked and dismissed. While the engineer was productive, he demoralized everyone around him and made them less productive.

There are two ways to get the right person in the right place. The first is to hire well. The second is to mentor and build up the person you have in place until they become the right person. In this particular case, this engineer didn't take counsel well and couldn't see the effect he was having on the organization. I invested a considerable amount of time into this individual, but I had to fire him in the end.

When a person is causing an issue, you will only solve that problem by changing the person's behavior or removing them from the role.

Lever 2: Structure

The Structure of the organization is the environment in which people operate. Get the Structure wrong, and you will destroy the organization's ability to deliver. The main feature of an incorrect structure is hard schedule dependencies between teams.  If team A has to deliver a feature and depend on team B to do something to deliver that feature, both will fail.  No one will be happy because no one will be successful, and they will always feel like someone else in the organization is at fault. While there will be lots of finger-pointing and blame, it's not the fault of any individual.  The Structure of the organization is at fault.

When there are problems between teams, look for hard schedule dependencies between those teams. A great example of this is between DevOps and Platform teams. The Platform teams say, 'DevOps is not paying us any attention. We have to wait for weeks on this change.' DevOps says, 'The team keeps creating crap software, this one minor problem causes it to fall over every few days, and they refuse to fix it.' The reality is quite different. The DevOps team supports the entire organization and is firefighting everywhere. Deploying the Platform team's feature is just one of many priorities, and probably not the highest.

On the other side of the coin, the DevOps team operates the software in production and depends on the Platform team to produce quality software. They have a problem that is causing the system to fall over every few days. The Platform team has its priorities.  They are getting lots of pressure to deliver feature X from the Product team. They don't prioritize the fix for DevOps. This is an entirely natural and human approach. People optimize to reduce the pain that they feel, not pain that others feel.

A process-based solution to this would be to figure out how to flow the work correctly to DevOps, prioritize the work in DevOps, inform the Platform team of their priorities and accurately communicate problems back, etc. It won't work. At the very least, it won't work well. The better solution is to restructure ownership so that the Platform team owns deploying and running their own software in production. The DevOps team can focus on tooling and consultation. That puts the incentives are in the right place. If the Platform team wants something to get into production, they prioritize it. If something is falling over in production and causing extra work, they fix it, and they weigh the value of fixing that problem vs. other ongoing priorities.

One response that leaders have to Structure problems is to move people and reform teams without actually removing any rigid schedule dependencies between teams. They move people from team A to team B or shift ownership areas without changing how teams relate to one another. They end up playing musical chairs with the organization. Playing musical chairs doesn't solve any problem. It fatigues the organization and makes you look like an idiot. Don't do it. Look for hard schedule dependencies and eliminate them.

Lever 1: Strategy

Effective leadership revolves around moving as much of the organization's decision-making to its lowest possible point in the organization. The people at the lowest point that can make the decision should do so. For example, if there are two ways to implement a feature, the engineer should choose between them. That engineer shouldn't need to ask a Principal Engineer for advice or direction. If a product requires an external service, the manager or director should contract that service. It shouldn't have to flow up to the VP of Engineering.

To do that successfully, those individuals need to understand the organization's Strategy. It's long-term and short-term goals. They need to have a deep understanding of the goals and restrictions of the organization. Without that information, they can't make those decisions effectively.

Most organizations define a strategy.  In those few organizations that do, they still fail at communicating the Strategy and supporting it. The symptoms of that suffering are that decisions move up the leadership hierarchy. With each move up, you get a significant increase in latency or throughput and a more unsatisfactory outcome.

You can't improve something that doesn't exist. To build a Strategy, you must:

1. You need to define it
2. You need to communicate it
3. You need to support it

Once you accomplish those three things, you can make changes to the Strategy that changes the company. Until you have those three things, strategy changes will be ineffective. You can't change something you don't have.

Answering the Questions

Do you often have features delayed because a specific developer is on vacation?

This is a Structure problem. A team owns code, not individuals. You should make sure that your engineers pick up code from around the codebase and encourage the idea of shared ownership. There are lots of process changes that can foster that sense of team ownership, but the hard law is, no single person owns any line of code.

Do you often have release take days or weeks to get into production even after the team claimed it was 'done'?

This is a Structure problem, and it's a nasty one. You have a DevOps or Infrastructure team that owns releasing software and running that software in production. That single ownership of all the production code led to a single shared environment for every stage, a single Integration, QA, and Prod Environment. You share those environments across all products and cause a combinatorial explosion in potential problems. The fact that the engineering teams have to work through the DevOps team exacerbates that problem. This problem is challenging to unwind. You need to split the environments to have a separate environment for each product or service, then give ownership of those environments to the team that owns that product or service.

Do you often have to rollback releases immediately after getting them into production?

This is a Strategy and People problem. Strategically, you have not placed the right value on quality of software. You have not invested enough in automated testing, and you haven't done a good enough job defining boundaries between systems. Your teams probably do not have a good handle on how to backward-compatible changes and may not have invested in the infrastructure to support it.

Have you not delivered anything significant to production in weeks or months?

This is a Structure problem. No single team owns the deliverable. The ownership is sprinkled across multiple teams, each with their challenges and incentives. So no team can push anything to production without collaboration from numerous other teams. Each of those teams has their schedule, requirements, etc. In the end, nothing makes it to production.

Do initiatives look like they are doing fine right up until the final week or two before delivery before you get the bad news that dates are going to slip by weeks or months?

This is a Strategy, Structure, and Process problem. Fundamentally, it's the same problem as the one above. Ownership is split among multiple teams with the same outcome. You also lack the tooling needed to highlight delivery problems early rather than later. You haven't shaped the culture such that identifying issues earlier rather than later is rewarded.

Do you often hear 'If we just rewrite this <insert large system> it would solve our delivery problems' from your engineering leaders?

This is a People problem. Your technical leaders either don't have the experience to understand that the risk involved or doesn't have the discipline to invest in iterative improvements to the system. They are probably also conflating Strategy and Structure issues with technical issues.

Do you have the same two or three engineers jumping in to solve every production problem?

This is a Structure and People problem. If one or two organization members are always jumping in, then several things could be wrong. You may have structured the team such that you only a small number of people have the knowledge they need to solve problems. It could be that your team members don't have a sense of ownership of the codebase and defer to those they feel do. It could be that on-call is structured so that the same people are always pulled in to solve problems.

Are you terrified of losing your 'star performers' because you believe that if they leave, the wheels will come off the business?

This is the same problem as the one above. While every individual brings a lot to the table, you should never get yourself into a situation where the business is dependent on an individual. The strength of an organization comes from the sum of its people, not any person. Losing one of these 'star performers' will change the organization but will not damage it. Don't be afraid, losing one of these performers if they are also toxic (which seems to happen, often), then work them out of the organization. There probably won't even be a hiccup when they leave.

Conclusion

Focusing on Process isn't going to help you when you have a problem that need more leverage. When solving a problem, think about the longest lever you can use to make the change and then use that lever.