Startups and Small Sample Sizes

Here's a great way to commit mail fraud: pick 1024 names out of the phonebook, and send each of those people a free letter describing your analysis of the stock market, chock full of details like flattening yield curves, oil futures, and federal reserve monetary policy. In half the letters, state that all of these factors, taken together, will make the stock market go up in the coming month. In the other half, conclude that the market will go down. In each letter, recommend that the reader go long or short the S&P 500 according to your prediction.

512 people will receive a letter that turns out to be wrong; forget about them. Repeat the process the following month, splitting the remaining group in half again, so that 256 people get the "go long" recommendation, and 256 get the "short the market" recommendation. Continue every month, discarding the people who got the wrong recommendation in the prior month.

At the end of ten months, there will be one person who has received an accurate prediction every month. Send them a letter saying, "you've benefited for my market-beating advice for almost a year now. To keep getting my newsletter, just send me a check for $5000."

Your final person ("the mark") has every reason to believe that you are infallible at predicting the direction of the stock market. Why, they'd have to be an idiot NOT to send you $5000 to continue getting such valuable market insight! You've never been wrong!

As outside observers, we can, of course, see that you've made 1024 distinct predictions, and you've been wrong in 1023 of them. But your target doesn't know that. They have the wrong sample.

So far, this has nothing to do with software. But reconsider the above scenario, replacing each of the targets with employees at each of 1024 different software startups, each with a 50% chance of surviving any given year. At the end of 10 years, we should expect that exactly one of our startups will still be around.

Is that one startup filled with people whose talent that is so far above the mean that it was destined to be the one to survive? Or were there a variety of random and semi-random factors at play, completely outside the control of the company, that helped them survive while the other guys failed?

I'm a very strong believer in the importance of a strong team, and I do believe that strong teams build successful companies. But all too often, people incorrectly make the reverse inference -- "If I'm at a successful startup, it's because I'm on a strong team. If the team is strong, it must because I am strong. In fact, my company has never made a mistake. I am personally, and in all ways, infallible! Wow, it is so great to be me!"

This phenomenom can easily get so far out of hand as to hurt your company. Entrenched gatekeepers from the early days of your company believe that their own judgement is solely responsible for the success of the company, and as such, why would they ever need to consider other opinions?

The best way to detect whether someone is truly a rainmaker is to watch how they do in their second startup. Just to pick on an old favorite: everybody remembers the whiz kid from UIUC who was going to destroy Microsoft with a web browser... but how many people remember LoudCloud? Broadcast.com was fantastic! But what ever happened to THAT guy? Oh, apparently he owns a basketball team...

Of course, if you're the late-joiner at a successful startup, you can't just wait around for everyone to leave and see how they do in their next venture (tempting though it may be to encourage them to head out on their own).

But you can at least come in with an understanding that people might be unwilling to entertain dissenting opinions for completely artificial reasons. When you see the early employees of your company making decisions without examing the arguments, that's your sign to push back, hard.

[Many people will read this post and ask the natural question, "well, does Google have this problem?" A year or two ago, we definitely had a large number of people walking around who were convinced of their own perfection. Fortunately, enough late-joiners have made enough critical contributions to the company that the problem has dissipated. There are certainly still pockets of this attitude, but it doesn't take long to figure out where they are and how to avoid them.]