The Speed Trap
"America is all about speed. Hot, nasty, badass speed."
- Eleanor Roosevelt, 1936
The pace of technology development seems to be ever accelerating, as multiple talented teams race for market share in a world increasingly dominated by software. Strategically, these teams are faced with needs that at times seem diametrically opposed to one and other. The need to release products that are simple, functional, and bullet-proof so as to capture and retain a valuable market, and the need to do it really, really fast.
At its heart, this is an opinion piece, but in it, I will discuss some key observations I've made in over 20 years working with very early stage companies. My own anecdotal evidence is bolstered by a common-sense reading of industry trends, so I feel comfortable making these broad statements about software development at early-stage technology companies.
Observation #1 - Software start-ups want and need to move very fast.
Nothing very controversial here. The desire to attack a market need quickly is driven by the advantages to be gained in being and staying first in your niche. In the world of software, any lead you can carve out over your competition is critical. The pace at which new technology is invented, and can be deployed, requires constant vigilance in your chosen market. Competition is fierce, and history teaches us that, with so many chasing the leader, being first mover never guarantees success. There are also significant advantages to having a shipping product when you visit the board-rooms of venture capital firms or of potential business partners. Although powerpoint slides and promises can still occasionally get funding, the advantage does go to the company with something real to show.
Observation #2 - The people developing software are human.
OK, I know I'm getting a bit silly, but stick with me. Humans make mistakes, humans working very fast make more mistakes than humans working slowly and methodically. The most evolved startups I've worked with accept and embrace this fact, a couple actual mottos from companies I've worked with are illustrative:
"Move fast and break things."
"Making mistakes is OK, moving slowly is not."
In both cases, the CEO's genuinely believed in these statements and strived to create a blame-free environment where, if mistakes were made, they were just as quickly corrected, and the company continued forward at a rapid pace. The people in the various teams, and in the company overall, collectively owned the successes and the failures. Though people who broke something might get a goofy "prize" or some good-natured ribbing, as long as the problem was corrected quickly, no harm was done and no blame was assigned.
Observation #3 - Its hard not to want to blame someone when your system is hard-down.
Really an extension of the "people are human" observation, when a company's source of revenue goes down, it can be difficult for those with the most vested interests to stay calm. As evolved as the company culture might be, as the fast pace continues and the systems become more complex, stability will begin to become a problem. As new people (and thus, new personalities) are added in various positions within the now fast-growing company, maintaining the egalitarian, open and tolerant culture becomes a bigger challenge, requiring an ever bigger commitment on the part of the founders and senior management.
Observation #4 - Moving fast for prolonged periods has measurable costs.
The companies I've observed prior to writing this have all tended to stay at speed continuously, some of them for years. This becomes evident in the form of several dangerous patterns that become progressively harder to correct:
In my experience, software teams that run at speed for years on end always reach a tipping point at which the accelerated pace ceases to move them objectively ahead of the competition, and actually becomes much more like borrowing current time-savings from some future-version of the company. Ultimately, a company must mature out of the run-like-crazy stage and there will be expectations of documented systems, repeatable processes, efficient on-boarding of new hires, project-transparency to management, and all manner of things that full-grown companies have to deal with. The longer the early-self of the company deviates from these expectations, the harder it is to correct the underlying issues later on.
Together, these observations form the nexus of the Speed Trap.
At this point one might be tempted to guess that the solution to the speed trap would be to slow down, but that would be incorrect. We've already established that moving fast is an existential necessity. Instead, I propose thinking of the kind of speed we're talking about in terms of an automotive track-racing metaphor. Just as a racing team does not field any old clunker in the Indy 500, so should a software start-up carefully prepare to go fast. In racing, tuning the engine and checking the safety equipment while still in the pit is the best way to quick lap times. During the race, knowing when to hit the brakes and come in for fuel and new tires is just as critical.
Let's run with this metaphor for a while and see how far it takes us...
This is the first step, to be done before a single line of code is written. It does not need to take months to accomplish, indeed, depending upon the core team size and the goal it can be done in as little as a week and usually less than a month. Doubts? Remember, we are talking about going fast here!
You need to spend a little time defining where you're going, this stage is like a driver taking practice laps and learning the track. Avoid analysis paralysis - people have been creating performant software on an Internet scale for a dozen years or more and much has been written on it. There are good general use system architecture patterns, emulate what works for your expected level of scale by connecting yourself to what other successful software companies are doing. The selection of technology stack is important, but not nearly as important as breaking a large, complex problem into smaller, solvable units - focus on this first! Forget "future-proofing" - it's an imaginary goal at this early stage.
Hint: Purpose-built services arranged in a decoupled, asynchronous, event-driven network of components is a great first start - more on this in future posts.
Use common third-party tooling, the most well-known and highest rated the chosen technology stack offers, resist the temptation to "roll your own."
It still surprises me how many teams miss the mark here. It is all too common for the teams infrastructure to be a hodgepodge of shell scripts and stuff cobbled-together in someone's spare time. Although this sort of infrastructure often works, it rarely scales and when new hires come in it can be very opaque to them, making it more difficult than it has to be to come up to speed.
Agile methods are designed to allow teams to respond quickly and easily to changing requirements. Hot, nasty, badass speed is not part of the dictionary definition of agility, so tweak the methods and don't be ideological. The teams I've seen running at high-speed usually use a mix of:
There is no ideal mix and all of these teams adjusted the method while running based on actual conditions and measurable feedback (i.e.: serious defects, customer compliments/complaints, missed vs. made deadlines) from product customers, company stakeholders, and the development teams themselves. The only commonality was that no team conformed completely to a particular method, agile or otherwise, but all put a fair amount of thought into a method. The industry a particular solution targets may drive the mix as well, for example, a new financial transaction platform would skew towards more established methods with much less wild-west and hanging-on.
The time is now to establish the company's overall tolerance of risk and acceptance of the inevitability of mistakes. It should be ingrained in the culture before a single line of code is written. I strongly advocate creating a value statement about this, as in observation #2.
Hire Fast People
I won't dwell too much on the topic of people. If you are assembling the core engineering team for a software start-up, you probably aren't going out of your way to hire lots of average people with little skill and even less independence and motivation. I am assuming a cohesive, proficient and motivated core team. I will make the simple statement that technology chops are table stakes, after you screen for mastery and skill, you should favor:
At this point, the team is set up for success, it now must be set free to achieve its objectives. This is the moment where you must start living your culture, embracing your method and trusting your infrastructure. When the driver is confident in their car, good things happen.
Tuning the engine to run at speed, then flooring it, is only half the racing strategy, you have to know when to make a pit stop. When running pedal-to-the-metal, it is often hard to be introspective. Its time to slow down and take stock if:
I've worked with companies that do everything else right but still fail to take this step until it too late to efficiently correct issues. Just as in racing, making a pit stop does not mean you're destined to loose the race, they are an integral and strategic part of the overall win strategy. The exact duration of such a stop is dependent upon it's objectives. It should be noted that continued maintenance and enhancement of the overall system does not actually completely stop, merely slows to make room for some important catch-up activities, which might include:
Question the Status Quo
Be brutally, but constructively, critical of the current state of the systems, methodology, and infrastructure. Imagine you are a member of the pit crew. What if you noticed a race-ending fuel leak in the engine, but were so concerned with the driver being angry about losing the race that you said nothing? What are the consequences? Of course, in most software development efforts, an oversight like this will not result in someone dying in giant ball of flame, but they can cost a company 10's or 100's of thousands of dollars down the line if not addressed.
Write stuff down
This is the most critical, but often the most mundane of tasks to accomplish during a strategic deceleration - if nothing else gets done but this one thing, it is worth it. Document undocumented systems, infrastructure, methods. Update all existing documents to reflect the current as-is state of the system.
Train the new guys
Almost as important as writing stuff down, if you have hired new team-members while running at full speed, most of their training has likely been on-the-job and done by reverse-engineering the existing systems on their own. Now is the time to formalize the recent-hires understanding of the system. Make sure everyone is on the same page, and you dramatically increase their ability to contribute.
Automate manual processes
Have manual processes crept into the daily lives of developers, QA or systems administrators? Time to automate!
Plan, Design, Refine
Now is a great time for design spikes, meet, talk, whiteboard. But don't get too caught up in architecting for the ages. Time-box these efforts so they don't become open-ended. You're about to launch into another speed-run, so it's best to develop guidelines for where you're going.
I must admit, as a software developer, I have on occasion, longed for the days when software development was a methodical and highly planned activity. As a business person, I have had to embrace the need for speed, and drag my software-developer self along for the ride. Most of the companies I've been closest to in my career did not do fast very efficiently or with great results. You learn a lot from failure, perhaps more than from success, and these experiences have coalesced into a recipe that I believe creates the best chance for successful high-speed software engineering.