The limits of OKRs

I

The first mistake I made while doing planning & writing engineering goals was that I was confusing the ends with the means. I made that error repeatedly. Part of the pitfall was due to a framework I was using to guide my thinking, called objectives & key results, or OKRs. I would phrase a typical result along the lines of

Increase by 2x the number of users of our library.

The next step after drawing a set of results is a negotiation step. As part of this, me and my customer try to agree that the results I propose are good, or otherwise how to improve them. Whenever I would negotiate a result such as the one above, I would get the question "why do we want to increase the number of users?" My answer would then be "that's how we know we're doing a good job," or something similar.

Like a salesman, my rationale was that more sales implies we're doing a good job, so the above formulation made sense to me.

I was confused. Specifically, I was confusing the means – growing the user base of our library – with the end – which I left unspecified. I did not state why we should pursue a goal, so my plans were hollow in a sense, devoid of value. It took me quite a bit of time for me to understand this gap and its implications.

This was merely ~4-5 months ago. At the same time, I was reading Doerr's book called "Measure What Matters," which I was using to deepen my understanding of OKRs. What I learned from this book, briefly, was that objectives answer the question "what is important?" while results are finer-grained milestones expressing "how do we work towards an objective?"

The OKR framework is simple in principle. It is anchored in the dual questions of what (objectives) & how (results). So it was that I was ignoring the why question, I wasn't bothered much about that part. Again, "why grow the number of users, towards what end?" This was a good question, but I assumed the answer was obvious, and this book was encouraging me in being superficial.

II

I grasped the importance of why by luck. Or maybe I was just ready for it.

This came in the form of an interesting story from Doerr's book. The story illustrates how a YouTube executive can fall into the same trap as I did. The difference is that the consequences of his confusion were more obvious to me compared to the consequences of my own error. In this story, the executive is trying to convince some of his colleagues that their plans should center on maximizing user engagement, i.e., watch time, instead of some other objective. I quote it in the picture below.

The youtuber's assumption that "our job was to keep people engaged" is the same as my assumption that my job was to "increase the number of users of our library". The scale and implications are different, but we are both confusing means with ends. Keeping people engaged is a means towards some end. Increasing the number of users of a library, likewise, is a means towards an unspecified end.

The story above illustrates what tech companies probably tell themselves when they decide that they must grow at all costs – even if that growth involves manipulation of user emotions. Both the end and the means are captured in the same metric. Another way to put it is that both the users and the service seem to be pursuing the same end, called happiness, and so the more a user employs their service (watching youtube endlessly, for instance) the happier they are “by definition”.

If we ask the question "why would it be important to increase watch time?" the only answer we know about is that "viewers are happier." But this answer is an assumption. The why question was in fact left unanswered. I can even imagine this kind of approach would be considered illegal sometime in the future. Just like casinos and drugs have regulations, so social platforms might bring regulation upon themselves by courting sketchy practices and claiming to know what makes users happy.

The story is doubly-interesting because it also shows a way in which the culture at YouTube probably deviates from the one at Google Search. Google/Alphabet wasn't always a power-hungry behemoth as it is today. Indeed, the same chapter in Doerr's book states:

"All other things being equal, our goal is to increase [video] watch time." For many folks at Google it smacked of heresy. Google Search was designed as a switchboard to route you off the site and out your best destination as quickly as possible. Maximizing watch time was antithetical to its purpose in life.

John Doerr, "Measure What Matters" (pg. 161), 2018. Portfolio/Penguin

Google Search still functions as a switchboard, fortunately. Their mission was beyond growth and "happiness", and I guess it was less about manipulating users and more about providing the best web search results possible. Why? So that users can find the web page they were looking for and get on with their life. If so, the folks at Google Search have not made the error of thinking that the search engine is an end in itself to try and capture all the attention around it.

From means to ends

I worked many hours to try and correct the myopic flaws in the OKRs I was writing. Our CEO suggested eventually that it may be confusing to think we're writing OKRs in the conventional sense of this framework. Instead of objectives and results, we shifted to using different terms, namely, results and outputs. These two concepts are not the same as objectives and results in the OKRs sense.

The results are "outcomes we create in the world," while outputs are "artifacts we produce". The way I understand it is that:

• outputs (in our method) and objectives (of OKRs)  serve similar purposes, by providing answers to the what question;
• the results in our method, however, have no correspondence in the OKR terminology. A result argues that a set of outputs are desirable, by answering the why question. We often say that a result is the outcome or consequences we intend to produce in the world around us. That is the end.

An example of a result would be:

Deliver the v1 implementation of IBC modules so that we unblock developers who want to build on IBC in Rust.

The approach we're using is all neatly documented, in a different perspective than I describe here, in our workflow guide. The change to results + outputs helped, though it was still confusing until I got used to it, which took a couple of iterations.

III

OKRs are very helpful, but of limited use on their own. This is fair: I doubt Doerr or Andrew Grove ever intended to say that OKRs are a sufficient solution to do planning in a modern engineering organization.

The takeaway is that the critical step after getting used to OKRs (or any other framework) is to gain an understanding of their limits. OKRs are a tool, so we ought to deploy it in a way that reflects our organization’s values. Doing that second step (refinement in accordance with values) is harder than the first step (adopting the tool).

Going back to the framing of results: What became important to me, once I crossed the threshold of why, was that we avoid being short-sighted and our plans capture not only what we consider success but also the consequences of that success. We achieve this by encoding the positive consequences we intend directly in each result, which is one of the most wholesome feelings you can from a spreadsheet.