Slava Akhmechet

I am a cofounder of RethinkDB — an open-source distributed database designed to help developers and operations teams work with unstructured data to build real-time applications. We're hiring.



The effective engineer’s handbook

10 Mar 2013

I wrote this guide for new RethinkDB employees sometime in early 2012. Re-reading it one year later, everything still rings true (though it’s much more wordy and self-aggrandizing than I’d like — I’ll edit it some day). I believe this advice can be helpful to everyone trying to do good work, not just people working at startups, so I’m publishing it as is.

EDIT: Daniel Gackle pointed out that the quotes by Plato and Aristotle aren’t real. I’m leaving the attributions since by now they’ve become a part of the long-term internet memory, and it’s not my place to fix cosmic errors of this magnitude.

One must have chaos within oneself, to give birth to a dancing star.
Friedrich Nietzsche

By now you must have received the official RethinkDB onboarding booklet. It is a document written by lawyers designed to have you meet minimal compliance to minimal standards. Of course, the actual standard requires relentless pursuit of quality – something no legal document could ever define or formalize. But just because something cannot be precisely defined does not mean it cannot be fruitfully discussed. Therefore this guide, as Zen Buddhists would say, is a finger pointing at the moon.

During the interview you solved complex algorithms problems and demonstrated a deep understanding of operating systems internals and development tools. As alien as this thought may be to you, you are one of the best software engineers in the world. But that’s only half the battle. It’s obvious you’re here because we hire based on intelligence. But our expectations of you aren’t based on that. They’re based on effectiveness. Evidence that there exists a stark difference between the two is all around us— the world is littered with brilliant musicians, artists, scientists, and engineers who never see the light of day because they aren’t effective in their work. You’ll learn a tremendous amount about software during your time here, but the most important lessons will be about how to be effective.

You may wonder why we don’t design our hiring process around effectiveness. The simple answer is that it’s impossible to test for. The only guide we have is to look at people’s previous achievements, but there is no guarantee that people that have displayed effectiveness before will continue to be effective in our environment. In fact, most people with stellar track records have likely already done their best work. They don’t want to get their hands dirty. I can’t tell you how many PhDs with fifteen years of industry experience we had to turn down because they couldn’t pass the FizzBuzz test. So we don’t look at pedigree at all. We test for intelligence instead, and work out effectiveness later.

What does it mean to be effective? There is an entire industry dedicated to selling resources on the subject. But you don’t need to read dozens of books to be effective. You can get 90% of the way there by paying attention to very few things. Here are the most important things we noticed during our tenure here.

Work on things that matter

Only one, but it’s always the right one.
Jose Capablanca, when asked how many moves ahead he looked while playing chess

The single biggest mistake brilliant engineers make is to work on things that don’t make the slightest bit of difference in the grand scheme of things. To illustrate how important this is, consider that you might spend sixteen hours a day in the office, but if you’re working on something that isn’t crucial to our success, it’s as if you didn’t show up that day to work at all. Of course days turn into weeks, and weeks turn into months. In most businesses it’s preposterous not to show up to work for months at a time, but is perfectly acceptable to spend months working on things that will make no difference whatsoever.

The reason why brilliant people make this mistake so often is that it’s easy to confuse the set of difficult problems with the set of useful ones. It’s like optimizing CPU cycles when you’re bound by a mechanical disk, except in the context of product features. Picking useful problems is a skill completely orthogonal to engineering. They don’t teach it in school, and they don’t teach it in most companies. But as a startup we don’t have the luxury to work on second order problems. We have no choice but to be ruthless about working only on things that really matter.

I’m not saying that it’s never worthwhile to work on fun problems that aren’t immediately useful, or that you shouldn’t occasionally spend a day or two working on a light, low priority feature. But it’s crucial not to fool yourself into thinking you’re solving important problems, when in fact you’re only satisfying your own curiosity.

We’re fortunate to be building a product full of hard, interesting problems. In fact, we designed RethinkDB with this in mind. But our primary motivation is to build a product that delights customers. That’s where we direct our curiosity and our efforts. Hard problems are the icing on the cake. At RethinkDB nobody will be breathing over your shoulder. If you don’t go out of your way to ask for direction, you likely won’t get any. Our management hierarchy is barely noticeable. We’ve all got more than enough work on our plates to tell you how to do yours. But you’re still expected to work on things that make the most impact, so it’s crucial that you quickly learn how to pick important problems.

The best way to learn is to keep asking us until our mental models converge. We didn’t start out being good at this. When I look back at our first release I’m surprised I ever thought anybody would want to use it. The work wasn’t wasted – we built incredible technology that we now build on top of, and we got tremendous education on what problems customers really want solved. But if we had to do it all over again, we could probably cut 25-50% of the work. Every startup begins this way. When a fresh team starts out building something new, it takes a while to develop a sense of taste for what a good product might look like. We’re still learning. With every release our software and our mental models are getting closer and closer to what customers are really looking for.

Today I can look at a customer’s LinkedIn profile and make a pretty good guess on what they’ll say about RethinkDB. Of course, product management is a crucial part of my job— I developed this understanding over three years and hundreds of face-to-face conversations. We don’t expect you to match it overnight, but we do expect you to demonstrate care, dedication, and aptitude for learning about the problems customers face. Software engineering is an applied science – it is useless in a vacuum.

Ship as quickly as possible, but no quicker

If you think you have things under control, you’re not going fast enough.
Mario Andretti, racecar driver

Today there are many opportunities to become wealthy by building companies that apply existing technology to antiquated business models. You can identify these companies by observing that to succeed they require next to no technical R&D effort. You can build very valuable products this way, but we don’t do that here (although we had more than a few opportunities to take that path). Companies we admire the most, the ones that have proud exhibits in the Computer History Museum, the ones whose prints you’ll find on our office walls, didn’t do it that way. Neither do we. Out of the set of valuable problems, we pick technically difficult ones because that’s what interests us, and because others don’t have the courage, the perseverance, or the expertise to work on them. Being unafraid to work on difficult technical problems is our secret weapon.

We don’t ship half-baked processes glued together with Perl scripts and call them a database. We care deeply about user experience. We tackle deep systems problems that everyone else ignores. But there is another side to this coin. Before we can get to the future, we need to survive in the world we live in today. This means we need to ship very quickly— faster than any of our competitors can. It may seem scary, even impossible. But it may also seem impossible that you’re one of the best developers in the world, yet all evidence points to this fact. All of us who have been with RethinkDB from its inception are getting close to logging ten thousand hours in the database field alone. Although it doesn’t feel that way, by Malcolm Gladwell’s assessment we’re experts in the field. To our surprise, this is corroborated every time we meet other people who are close to this field and probe their knowledge. After talking to enough of them, the problem of shipping quicker than everyone else doesn’t seem very scary to me at all.

We didn’t start out doing a good job at this either. In fact, we were probably the slowest to ship of everyone in this space. This was partially a conscious tradeoff – we wanted to build up the core technology, the architecture, and the internal tools to the point where we could be satisfied with the foundation. If you compare RethinkDB’s architecture with the architectures of competing products, you’ll instantly be able to tell the difference in craftsmanship and sophistication. And soon, these differences will become apparent to customers too.

But more importantly, we shipped slowly because we were learning about the constraints of the field, our own limitations, and the dynamics of working efficiently as a team. We’ve learned that shipping software faster than others and still maintaining higher quality from the customer’s point of view requires a tremendous amount of dedication, craftsmanship, and ingenuity. It requires making difficult choices in all things, big and small, that everyone else is unwilling or unable to make. Learning how to make these choices is the second most important skill you need to master.

The most common reason for being unable to ship high quality software quickly is over-engineering. This problem has been discussed to death in the software engineering community, but we still see it pop up all the time. It’s really the issue of working on things that matter, just on a smaller scale. It’s a very human quality to anticipate tomorrow’s problems and solve them today. The reason why it’s particularly dangerous in software engineering is that there is a mountain of empirical evidence that humans are very poor predictors of problems in software projects. Doing this well requires developing a domain-specific intuition, something you likely don’t have yet.

In my time here I almost never encountered opposition when I asked people to work on something, but I used to almost always encounter opposition when I asked them not to work on something. But in fact, people should do the exact opposite. When someone asks you to implement a feature or solve a problem, there is a good chance the problem exists only in their own mind. You ought to ask them to present evidence that the problem actually exists in the real world. The burden of proof is on them. But when someone tells you a problem isn’t worth solving because it likely only exists in your mind, they’re probably right. The burden of proof is on you to convince them otherwise. We fall in love with our own creations and with our ideas. It’s also a very human quality and a cognitive bias that’s very difficult to break. But learning to walk the tightrope that separates perfection and development speed is part of being effective, and you’ll be expected to do that here.

The second most common reason is lack of motivation. Most of the time it’s a natural fluctuation in productivity. There are days that I take off because otherwise I would have ended up just staring at the screen all day, and there are days when I just accept that I’ll have to force myself to get through a particularly unpleasant task because it needs to get done. There are other days I don’t get much done because I question whether the company is on the right track. But all of that is OK – ultimately I’m still happy to come to work because the ideals and the sense of purpose we’ve built up resonate with me, because I like and respect the people I work with, and because I have complete confidence in their ability to accomplish great things.

If you find yourself unable to be effective for long stretches of time, the reason is probably deeper than that. When I worked for the financial industry in New York, I was miserable. I couldn’t bring myself to care about the products they were building, I didn’t fit into their culture, and their ideals didn’t make any sense to me. Eventually I moved to Silicon Valley and started RethinkDB with Michael. It turns out I did myself and the companies I worked for a huge favor. In retrospect, I only wish I’d done it earlier.

If you managed to get a job here, you can get another one in almost any software company in the world. So if you find yourself unable to do high quality work quickly because you’re unhappy, you shouldn’t be afraid to speak up. If we can address the issues you bring up, we will. And if the differences are irreconcilable, you owe it to yourself to find a place where you can find fulfillment.

There is another reason commonly found in other companies – too much red tape that prevents people from getting work done. But that’s not worth discussing because it’s obvious, and because you won’t find any of that here.

Make people’s jobs easier

Be kind, for everyone you meet is fighting a hard battle.
Plato

Occasionally I walk around the office, asking people to explain in depth how they designed the components they’re working on. I always feel uneasy doing it because I feel like I’m interrupting real work from happening. When we started out, every time I did the walkthrough, I inevitably discovered that almost everyone had very different mental models of fairly important aspects of the system. Eventually each team member grew to be an information hub. I don’t do the walkthroughs nearly as often today because whenever I do, everyone tends to be on the same page. The effect on team cohesion and overall output has been enormous.

Learning to be on the same page with the team is a crucial skill for being effective. A few weeks into your time here you should have a good understanding of the company, the product, and its architecture. This means for the first two weeks you should spend at least a quarter of your time asking questions. Grab a different person every day and ask them what they’re working on, how they’re doing it, and why it’s important. Make it a daily ritual – I still find these conversations very valuable and stimulating. Beyond getting up to speed, talking to people about their work spurs new ideas, helps ensure everyone is on the same page, and improves the quality of our work.

When communication is going smoothly, product development is going smoothly. When communication is strained, product development is strained. Of course vice versa is also true. Communication and product development are both parts of one indivisible whole. If you’re attentive, you can tell how a company is doing by hanging around the office for a while. When things are going well, the office alternates between dead silence that occasionally breaks out into buzzing with energy and excitement. You can feel electrical charge in the air. If you’ve ever had the good fortune to witness an environment like this, you’ll recognize it immediately. When things are going poorly, there are constant arguments and irritation in people’s voice. The air is filled with a very different kind of charge. If you’ve ever seen an environment like that, you’ll recognize it too. Both are natural, necessary aspects of the creative process. You don’t see either of these in companies that have no sense of camaraderie and purpose.

Learning to be sensitive to how the rest of the team feels, and how your individual behavior affects everyone else is crucial to being effective. Software development is a team sport, and each person’s behavior has a multiplicative, not additive affect. It’s easy to disturb the delicate balance of a gelled team if you’re not paying attention. Coast when the team is excited, and you’ll quickly see the atmosphere change because people will lose faith. Show your irritation when the team is down, and you’ll only be adding fuel to the fire. Of course the opposite is also true. Finish a big feature when the team is down, and you’ll see everyone get excited. Smile when the team is tense, and everyone else will lighten up too.

If you aren’t an extrovert, we don’t expect you to act like one. But we do expect you to inspire the team with the quality of your work, your kindness, and your energy. Being effective means increasing the overall output of the team by much more than your individual contributions. It’s not enough to be good – you also have to make others around you better.

Wrapping things up

We are what we repeatedly do. Excellence, then, is not an act, but a habit.
Aristotle

Getting these things right requires a tremendous level of skill. You’re walking a tightrope in a multidimensional space, maintaining fine balance between being focused and burning out, shipping quickly and shipping quality, asserting yourself and damaging the team. We don’t expect you to get it right on day one. Occasionally we still slip and fall. Inevitably, you will too. That’s OK – we’ll pick you up and put you on the right track again. But it’s in mastering these details that great things are built. Learning how to do them well is an essential part of growing as an engineer because technical aptitude is only the beginning of excellence, not the end. We expect you to demonstrate dedication to learning these skills, receptiveness to feedback, and aptitude in mastering them quickly. It’s the most crucial aspect of building great software and is an essential part of being successful at RethinkDB.

Good luck!