The Better Letter: Randomness Rules
We tend dramatically to underestimate the role of randomness in the world.
If you like The Better Letter, please subscribe, share it, and forward it widely. It’s free, there are no ads, and I never sell or give away email addresses.
Thank you for reading.
Randomness Rules
Arkansas was one out away from the 2018 College World Series championship, leading Oregon State in the series and 3-2 in the ninth inning of the game when Cadyn Grenier lofted a foul pop down the right-field line. Three Razorbacks converged on the ball and were in position to make a routine play on it, only to watch it fall untouched to the ground in the midst of them. Had any one of them made the play, Arkansas would have been the national champion.
Nobody did.
Given “another lifeline,” Grenier hit an RBI single to tie the game before Trevor Larnach launched a two-run homer to give the Beavers a 5-3 lead and, ultimately, the game. “As soon as you see the ball drop, you know you have another life,” Grenier said. “That’s a gift.” The Beavers accepted the gift eagerly and went on win the championship the next day as Oregon State rode freshman pitcher Kevin Abel to a 5-0 win over Arkansas in the deciding game of the series. Abel threw a complete game shutout and retired the last 20 hitters he faced.
The highly unlikely happens pretty much all the time.
My hometown San Diego Padres missed out on the post-season in 2007 on account of a blown call on the last play of the do-or-die play-in game after being tied after 162 regular-season games.
Just last week, the Pads were down 8-0 to the Washington Nationals and future Hall-of-Famer Max Scherzer on the strength of a grand slam home run — his first Major League hit — by a just-called-up journeyman minor league pitcher from San Diego and a walk-off single from Trent Grisham.
As long-time Yankee lefthander Andy Pettitte said, “You gotta have luck, man. You gotta have things go right for you.”
We readily – routinely – underestimate the power and impact of randomness in and on our lives. In his book, The Drunkard’s Walk, Caltech physicist Leonard Mlodinow employs the idea of the “drunkard’s [random] walk” to compare “the paths molecules follow as they fly through space, incessantly bumping, and being bumped by, their sister molecules,” with “our lives, our paths from college to career, from single life to family life, from first hole of golf to eighteenth.”
Although countless random interactions seem to cancel each another out within large data sets, sometimes, “when pure luck occasionally leads to a lopsided preponderance of hits from some particular direction...a noticeable jiggle occurs.” When that happens, we notice the unlikely directional jiggle and build a carefully concocted story around it while ignoring the many, many random, counteracting collisions.
As Tversky and Kahneman have explained, “Chance is commonly viewed as a self-correcting process in which a deviation in one direction induces a deviation in the opposite direction to restore the equilibrium. In fact, deviations are not ‘corrected’ as a chance process unfolds, they are merely diluted.”
Such contingency explains why sports provide the world’s best reality show. The better team does not win every game.
The best team doesn’t always win the championship.
S*** happens.
Herb Brooks, coach of the 1980 U.S. Olympic hockey team of college kids that shocked the world, defeating the Soviet Union – the best hockey team anywhere (not just the best “amateur” team) on the way to winning the gold medal, played by Kurt Russell in the Miracle clip below, explains to his players how it can be that “this is your time.”
“One game. If we played ‘em ten times, they might win nine. But not this game. Not tonight.”
I, like millions upon millions of Americans, remember exactly where I was and what I was feeling when this happened.
Its power, its meaning, and its joy are wrapped in its improbability. In retrospect, it seems destined. That the U.S. team was “born for this.” The truth is, despite the power and greatness of Brooks’ speech, it was anything but.
As Stephen Jay Gould famously argued, were we able to recreate the experiment of life on Earth a million different times, nothing would ever be the same, because evolution relies upon randomness. Indeed, the essence of history is contingency.
Randomness rules.
In Major League Baseball, over a 162-game season, postseason teams lose roughly 40 percent of the time or a bit more. All-time great teams still lose about one out of every three games, all to inferior teams, demonstrating that winning baseball games involves a lot of luck. That’s why, over shorter stretches, it’s not unusual to see significant deviations from those percentages and noteworthy streaks. My Padres finished 14 games under .500 and 26 games out of first place in 1999 but still won 14 consecutive games during the season, providing fans with a lot of false hope.
Since mean reversion establishes that the expected value of the whole season is roughly 50:50 (or slightly above or below that level), a 60 percent winning percentage being really good means that there is a lot of randomness built into baseball outcomes. That idea makes intuitive sense – the difference between ball four and strike three, fair and foul, safe and out can be tantalizingly small (even if and when the umpire gets the call right). Strange things happen, too.
Sometimes hard drives get caught and poorly hit balls fall in. Players make uncharacteristic errors. Umpires make mistakes.
Great defense is both great skill on the part of the fielder and bad luck for the hitter.
Sometimes 17-year old fans inexplicably make baseball history and get in the way of what should have happened.
Luck matters. A lot. Yet, we tend dramatically to underestimate the role of randomness in the world.
The self-serving bias is our tendency to see the good stuff that happens as our doing (“we worked really hard and executed the game plan well”) while the bad stuff isn’t our fault (“It just wasn’t our night” or “we simply couldn’t catch a break” or “we would have won if the umpiring hadn’t been so awful”). Thus, desirable results are typically due to our skill and hard work — not luck — while lousy results are outside of our control and the offspring of being unlucky.
Two fine books undermine this outlook by (rightly) attributing a surprising amount of what happens to us — both good and bad – to luck. Michael Mauboussin’s The Success Equation seeks to untangle elements of luck and skill in sports, investing, and business. Ed Smith’s Luck considers a number of fields – international finance, war, sports, and even his own marriage – to examine how random chance influences the world around us. For example, Mauboussin describes the “paradox of skill” as follows: “As skill improves, performance becomes more consistent, and therefore luck becomes more important.” In investing, therefore (and for example), as the population of skilled investors has increased, the variation in skill has narrowed, making luck increasingly important to outcomes.
On account of the growth and development of the investment industry, John Bogle could quite consistently write his senior thesis at Princeton on the successes of active fund management and then go on to found Vanguard and become the primary developer and intellectual forefather of indexing. In other words, the ever-increasing aggregate skill (supplemented by massive computing power) of the investment world has come largely to cancel itself out.
Meanwhile, Smith argues that effort and repetition mean a great deal to athletic success, but that innate talent, which cannot be taught, means even more. Thus practice — even perfect practice — does not make perfect. Smith’s recounted experience as a cricketer whose career was ended due to the bad luck of injury and misdiagnosis resonated particularly for me since my younger son’s football career suffered a similar fate while he was in college.
Of course, randomness explains why the best team or player doesn’t always win, even though the best will tend to win more often. Being very good merely improves the odds of success. It doesn’t guarantee it. As Smith emphasizes, “[u]ncertainty is a pain to predict, but a joy to follow.”
Even without reading Mauboussin and Smith, we should all recognize that the outcomes in many activities in life combine elements of both skill and luck. Like baseball, investing is one of these. Understanding the relative contributions of luck and skill can help us assess past results and, more importantly, anticipate future results, a point to which Mauboussin pays particular attention.
The TradingMarkets/Playboy 2006 Stock Picking Contest was won by Playboy’s Miss May of 1998, Deanna Brooks. Her portfolio, which bet heavily on oil and gold stocks, gained 46.43 percent on the year and every stock in it provided double-digit returns. She liked Yamana Gold because “What girl doesn’t like a little bling? I’m hot for gold this year.…” It wasn’t her only nugget of sterling analysis. She also liked Petrobras because “oil is making money” and IBM because computers “aren’t going away.” She wasn’t the only Playmate to find a rich vein of success. A higher percentage of participating Playmates bested the S&P 500’s 2006 returns than active money managers. Think about that for a moment. Over the course of a full year, a bunch of Playmates outperformed a whopping majority of highly trained and experienced professionals with vast resources who spend all day every day trying to beat the market.
It’s easy to say that the Playmates got lucky, and they did. But we’d never expect a guy swimming laps at the YMCA to beat Michael Phelps across the pool, a girl off the street to beat a Grandmaster in chess, or an unschooled janitor to solve an insanely complex math problem amidst a spot of cleaning in the afternoon that the best and the brightest need years to figure out. Not even once.
If something like that actually were to happen, we’d treat is as a marvel (as the movie, Good Will Hunting, excerpted above, does), not just as a whimsical curiosity to be used for the purposes of garnering a bit of publicity and ogling attractive women.
It’s tempting simply to say that the contest is too small a sample size to be meaningful and move on. Had she stuck with investing, Miss May’s performance would miss and miss by a lot, probably sooner rather than later, as all investment performance tends to be mean-reverting. But we also know that sample size doesn’t mean much when little luck is involved. It doesn’t matter how many times I race Michael Phelps. The chances of my winning will always be vanishingly small — effectively zero.
These explanations are good as far as they go, but they hardly tell the entire story. Lady Luck is crucial to investment outcomes. There is no getting around it. Managing one’s portfolio so as to benefit the most from good luck and (even more importantly) to get hurt the least by bad luck are the keys to investment management. Doing so well is a remarkable skill, but not the sort of skill that’s commonly assumed, even (especially!) by professionals.
More to the point, if investment returns depend that heavily on luck and real investment skill is that elusive and rare, what should we do with our (or our clients’) money? For some answers, we turn to the world of…poker? That’s right — poker. Whether poker is deemed a game of chance or skill has significant legal implications for gamblers and those who earn a living from them. Moreover, there is a good deal of skill involved in playing poker well. However, for our purposes, since poker is an excellent means to evaluate probabilities under uncertainty and provides a great deal of data, it is an excellent means for investors to learn about the markets, which are also governed by a great deal of uncertainty, meaning that the best investors must deal with probabilities amidst uncertainty extraordinarily well.
Poker involves tremendous luck and tremendous skill. By way of comparison, consider chess, which involves little luck (white plays first) and tremendous skill, Tic-Tac-Toe, which involves little luck (who goes first) but little skill (such that a precocious first grader can quickly get entirely up to speed), and Chutes and Ladders, which involves tremendous luck and little skill. More specifically, the luck component is so high that the volatility of poker “returns” can make it maddeningly difficult for a better player to outperform even over substantial time periods (say many months of daily play, especially if the stakes are limited).
Looked at another way (as Nate Silver does in his book, The Signal and the Noise), a weaker player might be ahead of a stronger player after tens of thousands of hands. Silver estimates the likely outcomes for a very good (limit) Texas Hold ‘Em player over the course of 60,000 hands to range from up $275,000 to down $35,000 to 95 percent certainty. He also estimates that a player who has won $30,000 over his first 10,000 hands is still more likely than not to be a long-term loser (and short-term losers have mostly given up).
That’s a lot of luck!
In the markets, the average investor underperforms due to costs alone. Poker is similar on account of the house’s rake. Yet most investors — like most poker players and most people generally, due to optimism bias — think they are better (and often much better) than the norm. With poker players, the truth can be beaten into them as their losses mount. Since the markets are biased upward (most underperformers have positive returns overall), investors tend to remain delusional for a much longer time. Some never recover.
In a “quasi-experimental” study, researchers set out to examine these questions in poker. They got together a group of both expert and novice poker players to play fixed games, meaning that the players received hands that the researchers had set up – without the knowledge of the players – to test how things would go under various scenarios. The results revealed that while the cards dealt (luck) largely predicted the winner, skill was crucial to reducing losses when players were dealt a bad hand. That’s a true if unsurprising result as far as it goes. But the conclusion of the study (“that poker should be regarded as a game of chance”) is clearly overstated.
The study has been rightly criticized for not looking at enough data. It’s surely true that over the short term, luck dominates skill in poker. However, over longer and longer periods of time – a much larger database of hands – a slight skill advantage will result in a positive win rate because no player will have better cards in the aggregate. In other words, given enough time, luck cancels itself out.
As Silver argues in The Signal and the Noise, especially when the skill differential is not great, the interesting question is how long it will take for skill to win out. Another interesting question is why skill wins out. I suspect that – consistent with the study – the primary reason is that the expert player makes fewer mistakes. Science seeks the truth by uncovering and discarding what is false. What’s left is likely to be true. So, if you are a beginner, you’d better have beginner’s luck or you might be broke pretty quickly.
Luck can cut both ways, of course. Ben Roethlisberger was barely recruited before his senior year of high school because he didn’t play quarterback until then. His high school coach’s son, who was a year older, did (imagine that). As a consequence, he went to Miami of Ohio rather than a college power and was likely a more tenuous prospect as a result. Of course, he had less competition there than he would have had at – say – Ohio State, so maybe had he become a Buckeye he never would have made it at all.
And sometimes the luck is just bad. My younger son was an All-Freshman PAC-10 performer at Cal before a crushing injury changed everything for him.
After a big or revolutionary event, we tend to see it as having been inevitable. Such is the narrative fallacy. In this paper, ESSEC Business School’s Stoyan Sgourev notes that scholars of innovation typically focus upon the usual type of case, where incremental improvements rule the day. Sgourev moves past the typical to look at the unusual type of case, where there is a radical leap forward (equivalent to Thomas Kuhn’s paradigm shifts in science), as with Picasso and Les Demoiselles.
As Sgourev carefully argued, the Paris art market of Picasso’s time had recently become receptive to the commercial possibilities of risk-taking. Thus, artistic innovation was becoming commercially viable. Breaking with the past was then being encouraged for the first time. It would soon be demanded.
Most significantly for our purposes, Sgourev’s analysis of Cubism suggests that having an exceptional idea isn’t enough. For radical innovation really to take hold, market conditions have to be right, making its success a function of luck and timing as much as genius. Note that Van Gogh — no less a genius than Picasso — never sold a painting in his lifetime.
As noted above, we all like to think that our successes are earned and that only our failures are due to luck – bad luck. But the old expression – it’s better to be lucky than good – is at least partly true. That said, it’s best to be lucky *and* good. As a consequence, in all probabilistic fields (which is nearly all of them), the best performers dwell on process and diversify their bets. You should do the same.
Investing combines skill and luck, of course, but is more complicated because of the personal aspect (our psychology makes it hard for us to invest optimally), the difficulty in parsing out the market’s “psychology” the way a good poker player reads tells, and the much larger numbers of variables involved in ultimate outcomes. There is decidedly less market skill on display in the markets than poker skill at major tournaments.
This brings us back to the former Miss May, Deanna Brooks.
She won out over many professionals with lots of experience and vast resources who spent pretty much all day, every day studying the markets. That result provided still further evidence for what we should already know – market success (however defined), especially over the relatively short run, is more a matter of luck than of skill.
Investment performance data support this idea unequivocally. As Charley Ellis has shown, “research on the performance of institutional portfolios shows that after risk adjustment, 24% of funds fall significantly short of their chosen market benchmark and have negative alpha, 75% of funds roughly match the market and have zero alpha, and well under 1% achieve superior results after costs — a number not statistically significantly different from zero.”
As Silver emphasizes in The Signal and the Noise, we readily overestimate the degree of predictability in complex systems [and t]he experts we see in the media are much too sure of themselves (I wrote about this problem in our industry from a slightly different angle…). Much of what we attribute to skill is actually luck.
Plan accordingly.
Totally Worth It
This is the best thing I saw or read this week. The most interesting. The most noteworthy. The most remarkable. Stupidest candidates here, here, and here. The scariest. The sweetest. The most heinous. The funniest. The cutest. The most insightful. Irony is dead. The Iowa State Fair has 64 new food options this year, none of them healthy.
Please contact me via rpseawright [at] gmail [dot] com or on Twitter (@rpseawright) and let me know what you like, what you don’t like, what you’d like to see changed, and what you’d add. Don’t forget to subscribe and share.
Of course, the easiest way to share TBL is simply to forward it to a few dozen of your closest friends.
Benediction
This week’s benediction is “Presence of the Lord,” performed by all-time greats Eric Clapton and Steve Winwood.
Thanks for reading.
Issue 71 (July 16, 2021)