The Better Letter: Evidence-Based
Facts without interpretation are useless
Will Saletan argued this week that our current, hyper-partisan situation demands a new coalition: “a fact-based alliance that crosses party lines.” Today, the enemy isn’t any particular view or philosophy, it’s a commitment to being reality-based.
“I saw politics as a fight between left and right. I don’t see it that way anymore. Donald Trump’s presidency has exposed a bigger threat: an all-out attack on the principle that facts must be respected. We used to take that principle for granted; now we must defend it. Politics has become a fight between those who are willing to respect evidence and those who aren’t….
“In this fight, we need everyone who’s willing to play by the rules of deliberative democracy. So, for at least the next four years, that’s my commitment: If you believe in settling disputes by consulting evidence, I’m on your team.”
As Ulysses S. Grant wrote in 1861, “There are but two parties now, traitors and patriots and I want hereafter to be ranked with the latter.” Me, too.
However. It may be easy to fall in love…
…but it isn’t always easy to follow the evidence or, as has been said so often during this pandemic, to follow the science. In fact, it’s hard.
Our world is messy, highly complex, and generally proof resistant. At least as significantly, the heart wants what the heart wants. So, when we see some purported evidence or argument that confirms our priors, we ask if it can be true and rush to answer, “Yes!” And when we see some purported evidence or argument that contradicts our priors, we consider if it must be true and rush to answer, “No!”
The legal profession has been dealing with what good and relevant evidence is for centuries. According to the Federal Rules of Evidence (Rule 401): “Evidence is relevant if: (a) it has any tendency to make a fact more or less probable than it would be without the evidence, and (b) the fact is of consequence in determining the action.” That’s a really low bar, which explains why so much more than mere evidence is implicit within the rubric of evidence-based anything.
Therein lies the problem. Committing to an evidence-based approach is a great start – a necessary start even – to sound decision-making over the longer-term. But it’s not enough…not by a longshot. As philosophers would say, it’s necessary but not sufficient. Most fundamentally, that’s because:
The evidence almost always cuts in multiple directions;
We don’t see the evidence clearly or evaluate it well; and
We look for the wrong sorts of evidence.
This week’s TBL will have a look at each of those problems and then take on an example of how to go about examining evidence from the sports pages.
NOTE: I have gotten a bunch of requests to write about GameStop (I’m especially looking at you, Jarrett and David). Because everything is happening so fast, I’m afraid I’ll miss the market (literally and figuratively) if I wait until next week to discuss it. So watch your inboxes. A TBL special edition will be out as fast as I can get it done. To those of you who made the requests — thank you.
If you like The Better Letter, please subscribe, share it, and forward it widely. It’s free, there are no ads, and I never sell or give away email addresses.
Thanks for visiting.
Evidence Cuts Both Ways
As I have noted before, the first episode of Aaron Sorkin’s HBO series, The Newsroom, opened with television news anchor Will McAvoy (Jeff Daniels) sitting on a journalism school panel at Northwestern University and humiliating a doe-eyed college-sophomore about how “America is not the greatest country in the world anymore.”
Sorkin (via McAvoy) provided a long list of facts to support his claim and his alternative (and stronger) assertion, that “[t]here is absolutely no evidence to support the statement that we’re the greatest country in the world.” Sorkin’s list was culled pretty much straight out of The World Factbook from the CIA (2010 edition). As he has McAvoy recount, we were seventh in literacy, 27th in math, 22th in science, 49th in life expectancy, 178th in infant mortality (although this one is deceptive because the list counts down from highest to lowest), third in median household income, fourth in labor force, and fourth in exports.
Sorkin is careful to point out, however, that we do lead the world in incarceration rate, religiosity, and defense spending. He doesn’t mention it, but we’re also “number one” in obesity, divorce rate, drug usage (legal and illegal), murder, porn, national debt, and more.
One can even take issue with Sorkin’s claim that America used to be the greatest country on earth. That argument would likely begin by pointing out that despite our republic’s founding being based upon the idea that “all men are created equal” because “they are endowed by their Creator with certain unalienable Rights,” we still bought and sold people as chattel for nearly a century thereafter and women were not deemed so endowed for even longer. And then, after the Civil War provided a modicum of freedom to slaves, we still wouldn’t allow blacks to vote just because they were black for another century or more. We also waged genocide on native Americans and enforced immigration laws made up of quotas that allowed 100 times as much legal immigration from northern European nations as from, say, Mexico or Kenya.
The evidence Sorkin brings to bear about America’s lack of standing is strong. It suggests that our presumed national arrogance and home-country bias is delusional. Still, Sorkin’s primary claim about the lack of American exceptionalism is hardly the slam-dunk he assumes it to be and, contrary to his bold assertion, there is evidence – powerful evidence – that America really is the greatest country in the world.
The United States hosts almost 20 percent of the world’s migrants, making it a vastly more popular such destination than anywhere else on the globe. Even more importantly, the U.S. is by far the most desirable destination for the more than 750 million adults in the world who say they would like to migrate to another country permanently if they could. In fact, America is 3.5 times more desirable a destination for potential immigrants than any other country in the world.
Aaron Sorkin may not think that America is the greatest country in the world anymore, but people who want to escape the countries in which they live overwhelmingly think it is. Aren’t they in the best position to discern and decide which country is in fact the world’s best?
The most likely reason for that belief is the size, scope, and diversity of the American economy and the wide and deep access to it afforded generally. At 24 percent, the U.S. is responsible for a bigger share of the world’s GDP than any other country. By a lot. The U.S. stock markets account for more than half the world’s market capitalization while domestic bond markets are nearly 40 percent of the world total. Most importantly, overall, immigrants succeed here.
You may not agree if and when somebody asserts, perhaps without much aplomb, that America is the greatest country in the world. There are some good reasons for doing so. But there are plenty of reasons for agreeing, too.
The point, meanwhile, is the obvious conclusion that the available evidence doesn’t all point in one direction and, in this instance at least, the best available version of the truth, based upon the best available version of the facts and other evidence is uncertain and hardly dispositive. The evidence, as is usually the case, is conflicted and won’t interpret itself.
And that can be a very big problem.
Seeing as Through a Glass, Darkly
Nearly three-quarters of Republican voters say that Donald Trump was the legitimate winner of the 2020 election. Nearly half of Republicans say that, if a 2024 primary was held today, they’d vote for either Mr. Trump or Don Jr. Something like one-third of Republicans say they’d follow Mr. Trump to a new party. Professional Republicans play along, either duped or, much more likely, terrified of their base.
These are the worst offenders against reality in America today – by a lot – with the QAnon nonsense the very worst of the worst. Full. Stop.
But they are hardly alone. For example, the “Hands up; Don’t shoot!” movement is predicated on a lie – a lie often repeated long after it had been exposed by politicians who deemed it a helpful lie, including Joe Biden, Kamala Harris, and Elizabeth Warren (who remain anxious to declare Michael Brown a murder victim because they are terrified of their base). Various investigations, including one by the Obama Justice Department, concluded that Darren Wilson acted in self-defense and was justified in killing Mr. Brown.
As Jonathan Last has pointed out, it’s the political equivalent of the old saying about debt. When you owe a million dollars, the bank owns you. But when you owe a billion dollars, you own the bank. And it’s a much bigger and more bi-partisan problem than Saletan allows.
We’re all of us guilty: “None is righteous, no, not one.” Moreover, getting it right is hard: we readily believe a charlatan that affirms our priors over an expert who (rightly!) challenges them.
As I told Jason Zweig of The Wall Street Journal in a different context back in 2012: “There’s plenty of people who sell bad stuff knowingly, but I think the far bigger problem is inappropriate sales that are well-intended. I’ve seen people who sell bad stuff to their moms, because they thought it was the right thing.” We come to erroneous conclusions and yet are convinced that we are wholly justified, noble even, based upon our misreading, misunderstanding, and misapplication of the available evidence. We simply don’t see the evidence very clearly or evaluate it very effectively nearly often enough.
Proof Negative
As humans, we want deductive (definitive) proof, which is rarely available (except in closed systems, like math) but must settle for inductive (tentative) theories. Unfortunately, induction is the way science works and advances. That’s because science can never fully prove anything. It analyzes the available evidence and, when the force of the evidence is strong enough, it makes tentative conclusions, in an effort to ascertain the best available approximation of the truth.
These conclusions are always subject to modification or even outright rejection based upon further evidence gathering. The great value of evidence is not so much that it points toward the correct conclusion (even though it often does), but that it allows us the ability to show that some things are conclusively wrong. Never seeing a black swan among a million swans seen does not prove that all swans are white while seeing a single black swan (as in Australia) conclusively demonstrates that all swans are not white.
As much as we seek it, confirming evidence may add to the inductive case but doesn’t prove anything definitively. On the other hand, disconfirming evidence dispositively demonstrates what is false. Correlation is not causation and all that. Accordingly, disconfirming evidence is immensely (and far more) valuable. It allows us conclusively to eliminate a variety of ideas, approaches, or hypotheses. In other words, science progresses not via verification (which can only be inferred) but by falsification (which, if established and itself verified, provides relative certainty only as to what is not true).
That said, we tend to neglect the limits of induction and jump to overstated conclusions, especially when they are consistent with what we already think. Few papers get published establishing that something doesn’t work. Instead, we tend to spend the bulk of our time looking (and data-mining) for an approach that seems to work or even for evidence we can use to support our preconceived notions (see above). We should be spending much more of our time focused upon a search for disconfirming evidence for what we think (there are excellent behavioral reasons for doing so too).
As the great Charlie Munger famously said, “If you can get good at destroying your own wrong ideas, that is a great gift.” But we don’t often do that, as illustrated by a variation of the Wason selection task. Note that the test subjects were told that each of the cards illustrated below has a letter on one side and a number on the other.
Most people answer “A” — E and 4 — but that’s wrong. For the posited statement to be true, the E-card must have an even number on the other side of it and the 7-card must have a consonant on the other side. It doesn’t matter what’s on the other side of the 4-card. But we turn the 4-card over because we intuitively want confirming evidence. And we don’t think to turn over the 7-card because we tend not to look for disconfirming evidence, even when it would be “proof negative” establishing that a given hypothesis is incorrect. In a variety of test environments, fewer than 10 percent of people get the right answer to this type of question.
We should always be on the look-out for disconfirming evidence — proof negative — even though doing so is oh so counter-intuitive pretty much all the time. We routinely look for the wrong sorts of evidence.
Data-Driven
The root of this problem should be obvious: We’re human. On our best days, when wearing the right sort of spectacles, squinting and tilting our heads just so, we can be observant, efficient, loyal, assertive truth-tellers. However, on most days, all too much of the time, we’re delusional, lazy, partisan, arrogant confabulators. Evidence is what really matters, but with respect to persuading those we wish to persuade, confidence is at least as important as competence, and emotion matters even more.
So let’s work through a real-world problem – drawn straight from this week’s sports pages.
Trailing the fifth-seeded Tampa Bay the Buccaneers by eight points with 2:05 remaining and a Super Bowl berth on the line Sunday, the top-seeded Green Bay Packers faced a fourth-and-goal from the 8-yard line. A touchdown and two-point conversion could force overtime but Packers coach Matt LaFleur chose to kick a field goal to cut the lead to five.
To recap, the Packers needed a touchdown, were 8 yards from the end zone and were running out of time. And they had the league’s likely MVP, Aaron Rodgers, at quarterback. However, LaFleur sent the field goal unit onto the field (“Crosby makes this a five-point game again”) meaning … they still needed a touchdown, but now Tom Brady – the greatest QB ever – had the ball. Green Bay never got it back.
“The game is out there and it’s either play or get played,” as Omar said in The Wire. Let’s review the evidence to see if Green Bay played or got played. Well, we know they didn’t play; they kicked a field goal. Let’s look at whether LaFleur was right.
Strictly by the numbers, it was a close call. Had they gone for it, Green Bay needed a touchdown and a (difficult – roughly 50:50) two-point conversion to tie. Overtime would be a crapshoot. That’s unlikely but a field goal followed by a stop and a last-gasp TD was also unlikely. So, the numbers don’t clearly support either approach. Indeed, some models say kicking was the right call.
However, the data behind these sorts of calculations is drawn from the full universe of games and teams. On a stand-alone basis, Green Bay had the best red zone offense in NFL history this past season, scoring 48 touchdowns on 60 trips inside their opponents’ 20-yard-line. The Packers were the best in the NFL at going for it on fourth down this season – by a lot.
Here’s the bottom line. The Packers needed a touchdown and had Aaron freakin’ Rodgers. A field goal meant the Packer defense needed a quick stop and the Bucs had Tom freakin’ Brady.
My reading of the evidence says Aaron Rodgers got hosed. LaFleur and Green Bay got played. Your mileage may vary.
Totally Worth It
This is the best thing I read this week. It’s about love, loss, helping, grace, stories, the power of narrative, randomness, serendipity, vocation, and calling. Prepare for leaky eyes. This is the most persuasive thing I saw this week. This is the best political writing I saw. RIP, Hank Aaron.
This is spectacular.
This is lovely.
This is adorable.
Benediction
This week’s benediction features U2 and a gospel choir. I encourage you to sing along.
Contact me via rpseawright [at] gmail [dot] com or on Twitter (@rpseawright). Don’t forget to subscribe and share.
Thanks for reading.
Issue 49 (January 29, 2021)