The Signal and the Noise: Why So Many Predictions Fail--but Some Don't

This book is similar to Steven Levitt's  Freakonomics: A Rogue Economist Explores the Hidden Side of Everything (P.S.) , Nassim Taleb's  The Black Swan: Second Edition: The Impact of the Highly Improbable: With a new section: "On Robustness and Fragility" , and James Surowiecki's  The Wisdom of Crowds . All four books explore the intersection of data, human behavior, and outcomes. They explain how to quantify outcomes within the financial markets, professional sports or elections.This book is especially interesting because Nate Silver has honed firsthand his statistical skills onto numerous domains including professional poker, baseball performance forecasting (he developed one of the best software program to do that), political elections (his "fivethirtyeight" blog). And, when he is not a firsthand practitioner he is a first class investigator.The first seven chapters cover the errors and successes people have had in forecasting in various disciplines. Chapter eight is the most pedagogical, as the author explains the basics of Bayes Theorem that he considers as an overall solution to many of the errors we make in forecasting. The last five chapters focus on Bayesian thinking within various disciplines.Nate Silver's coverage of the credit rating agencies "Catastrophic failure of prediction" (first chapter title) is excellent. In a single sentence on page 13, he captures the cause of the financial crisis: "In advance of the financial crisis, the system was so highly leveraged that a single lax assumption in the credit rating agencies played a huge role in bringing down the whole global financial system." Silver states that the AAA rated CDOs were deemed to have a default rate of only 0.12%. The actual default rate was 28% or over 200 times greater! This was because the rating agencies missed out the correlation between mortgage default rates at different locations when a nationwide home price downturn hit (see figure 1.2 on page 28. Watch out that he mislabeled column 3 and 4 from the right). Silver assesses that overall leverage was too high during the housing bubble. Fannie Mae and Freddie Mac had a debt-to-equity leverage of 70-to-1. Lehman Brothers and other investment banks were leveraged over 30-to-1. Borrowers had often loan-to-value ratios of 100% on their homes. The volume of credit default swaps, MBS, CDOs represented 30 to 60 times the volume of home sales during the bubble years (fig. 1.5 page 35). Nate Silver summarizes the errors made. Investors trusted the rating agencies. The rating agencies assumed home prices would never decline on a nationwide basis because they never had since the Great Depression. Lenders and borrowers believed rising home prices would bail them out through refinancing. Policymakers believed the financial system had enough capital and was self-disciplined. And, economists completely missed the ensuing severe recession.Nate Silver focuses next on political predictions. This field of experts was so bad at predicting it motivated him to enter it by starting his fivethirtyeight blog. He documents their failings extensively. Within this chapter he refers to the theory of Philip Tetlock, professor of psychology and political science at Berkeley. Tetlock had surveyed predictions of experts in various fields. And, he categorized them within two archetypes: the hedgehogs and the foxes. The hedgehogs are dogmatic, rarely change their minds, and are very confident of their forecast. The foxes are just the opposite. They update their forecasts as often as new information warrants it. As a result, they make better forecasts.The chapter on baseball is one of the best because of Silver's extensive firsthand experience. He uncovers many concepts applicable to many sports such as the age-curve of baseball performance (pg. 81). All sports have a predetermined age-curve. Actually, every single aspects of life including life itself have predetermined age-curves. His description of what it takes to be a successful professional baseball player (pg. 97) has also surprisingly broad applications. The conclusion of the chapter is also fascinating. It describes baseball management as a competitive arms race of intelligence gathering to extract small competitive edges. And, that those competitive edges are short-lived. That's a very interesting application of the Efficient Market Hypothesis.The chapter on economists documents how inaccurate their forecasts are. The majority can't forecast a recession that has already started as they missed out on the three most recent ones (1990, 2001, 2007). In November 2007, the average economic forecast was 2.4% real GDP growth in 2008. Instead, real GDP shrank by -3.3%. Economists assigned only a 1-in-2000 chance of the economy shrinking that much. Yet, home prices were already declining. Foreclosures had picked up. Bear Stearns had gone belly up six months ago. Those were powerful signals the housing and financial markets were on the edge of a cliff. Also, economists are way too confident. The few times you can extract confidence intervals from the economic profession they are invariably way too narrow because they underestimate the error level within their forecasts (pg. 182). Nate Silver states that: "this property of overconfident prediction has been observed also in medical research, political science, finance, and psychology" (pg. 183). Despite our having so much more data and computer power at our hands, economic forecasting has not improved since 1968. This is because our underlying understanding of cause and effects has not changed much since.Chapter 8 introduces Bayes's Theorem. Here Nate Silver often refers to a very good book on the subject:  The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy  by Sharon Bertsch McGrayne.Chapter 9 and 10 about chess and poker are excellent. Kasparov was ultimately beaten by a computer bug. IBM Big Blue made a move late in the last game that did not make any sense (the team who programmed it confirmed it was due to a small programming bug). Kasparov who was in a vulnerable position could not figure out that move and in despair resigned the game and lost the series. The Pareto principle of prediction on page 312 and 314 and the ensuing economics of poker are really interesting. Poker winning are heavily dependent on the one worst player at a table. If he leaves, the winnings are a lot harder to reap.Chapter 11 on the Efficient Market Hypothesis (EMH) is excellent. Nate Silver states that the stock market is efficient most of the time, although it is never perfectly efficient (that would preclude a market). But, it can be wildly inefficient on few occasions associated with bubbles and crashes. Nate Silver demonstrates how both technical analysis and fundamental analysis do not beat the market over the long run. Fig 11.3 on page 340 shows no correlation between the performance of mutual funds over the 2002 to 2006 period vs over the 2007 to 2011 period. Past performance is no guarantee of future returns. Next, Silver refers to Robert Shiller in showing the market is not as efficient as the EMH entails. Shiller looked at the P/E ratio of the S&P 500 over a trailing 10 year period and looked at prospective returns. And, the longer the period contemplated the greater the negative correlation between trailing P/E levels and future average yearly returns. This suggests that the market can get overvalued. But, the return correction is not apparent until looking at average return over a 10 to 20 year period. Next, Nate Silver refers to the works of Richard Thaler and Daniel Kahneman in behavioral economics to outline how market traders are not perfectly rational. They suffer from herd mentality, overconfidence, and being overly emotional rendering their trading pro-cyclical.So, if the market is not so efficient, can you beat it? Probably not. On page 345, Nate Silver demonstrates how a hypothetical investor with perfect timing over a decade (1976-1986) would get killed by very small transaction costs. Even though this investor would handily beat the stock market before transaction costs, he would wipe out most of his capital after transaction costs. Silver next tests a prudent investment strategy over the 1970 to 2009 period. He assumes an investor is prudent and sells his position in the S&P 500 index whenever it had declined 25% from its peak and reinvests whenever it recovered 90% of its value. Such an investor would have earned only 2.6% per year vs close to 10% for a simple buy-and-hold strategy. Nate Silver does believe several hedge funds can beat the market. But, they have intellectual and technological resources that no retail investor and few mutual funds can match.Chapter 12 on climate change is really interesting. He differentiates between where scientists agree and disagree. They all agree that the greenhouse effect exists and keeps the Earth warmer than it would otherwise be; that temperatures have risen over the past century; that greenhouse gases have contributed to that trend; and that water vapor is by far the most potent greenhouse gas (not CO2 as the Media conveys). The majority of scientists agree that rising CO2 concentration does contribute to rising temperature. But, there is a debate regarding how much. Where the scientific community is more divergent is regarding climate models and projections. They acknowledge that Al Gore's  An Inconvenient Truth  deterministic apocalyptic message was way off base.Nate Silver explains why there is much uncertainty regarding climate models' projections. One uncertainty is figuring out CO2 levels 100 years down the road. Another uncertainty is getting the causal relationships right (there is a lot more than CO2 at play). Another uncertainty concerns whether those models are programmed correctly. Within the vast quantities of computer codes, are there a few bugs that contribute to generating erroneous forecasts? Nate Silver reviews the prediction of the IPCC's 1990 model and observes that temperatures have not risen as fast as the model predicted. Current temperatures are below the model's 95% confidence interval. This lead the IPCC to reduce their baseline temperature increase from 3 degree Celsius per century in 1990 to 1.8 degree in 1995. On page 407, Silver comes up with an interesting application of Bayes theorem applied to rising temperature predictions.The last chapter on terrorism is intriguing. Terrorist attacks follow a similar Power Law as earthquakes. The frequency of events declines exponentially with increase in intensity. More violent events are much rarer than lesser ones. But, the few major events dominate the data in human casualties. For instance, 9/11 represented more than half of the total fatalities from terror attacks in NATO countries since 1979. Thus, it is worth exploring means of mitigating the impact of such events.

I'm not the kind of person who loves every book of this style. I generally think that Malcolm Gladwell and Thomas Friedman just find cute ways to repeat things other people have already said. But I thought this book was completely different. Instead of finding cute new phrases to repeat old information, Nate Silver finds clear and concise ways to convey interesting, new information. I know that none of the mathematics he's conveying is new, but he does an insightful job of explaining how thinking in a Bayesian way can help in forecasting all sorts of world events and how actual outcomes can be used to improve our understanding of how the world works. Despite his non-academic background Nate Silver has made significant contributions to political forecasting. His forecasts are arguably better than and certainly on par with Princeton Professor (admittedly, not a professor of social sciences, so his professor job may not be very relevant to this conversation) Sam Wang. It's not just that Silver's numbers get closer to reality, but that he has a comprehensive approach that uses all information, appropriately weighted and with a wary eye towards overconfidence.Sometimes, I will say, that writing is a little grating ("...a stat head's wet dream!" - awkward). The chapter on baseball forecasting is pretty confusing if you don't already know a ton about baseball (I don't). And the chapter on Hurricane forecasting seems to miss the mark pretty badly. He has interesting things to say about the numbers around hurricane forecasts, but veers off in a crazy direction when offering a political interpretation of those numbers. He never mentions levees and makes one very brief, uncritical reference to FEMA. After explaining how great hurricane forecasts are these days he basically lays the blame for what happened in New Orleans on the victims. I guess he just wasn't thinking very hard about this part of the book, and neither was his editor. But overall it's a great read!

The book could a bit more of theoretical discussion, but it gives a reasonable introduction to probabilistic thinking and how it is used or mis(used) in daily life, with examples ranging from sports to financial markets.Given Nate's background it is not surprising that he focuses on forecasting, and how to develop a better framework for becoming a better forecaster, and also how a lack of probabilistic education and communication lead people astray, even leading to mistake that cost lives of thousand of people.My ultimate take of this book would be that we should consider more deeply the possible impacts of probability in our lives, and also that data does not speak for itself. Data need context and for gaining real insights it's necessary to apply critical thinking to it.

This book provides an excellent introduction to the world of forecasting and many different statistical concepts that are important to it. Every chapter uses different real world example from a different field (from earthquakes to poker) to explain a statistical concept in a way that is easy to understand but nonetheless fascinating. Usually an expert in that field is quoted and often visual examples are given. The use of these examples allows the reader to learn understand the concepts more easily but it also provides a fascinating insight into different real world uses of forecasting and statistics.Most books about statistics are not easy to read or understand, but this one is. And it still contains a lot of knowledge. One of the best books I've ever read.

Es un libro entretenido, hace visibles las desviaciones en las perspectivas que tenemos a futuro. En algunas áreas prácticamente "volamos a ciegas".

Must read for data folks.

将来予測をテーマに、地震予報、株式市場、経済、選挙などから始まり、ポーカー、ロッタリー、ＭＬＢ等と有りとあらゆる分野にわたって予測の面白さと難しさをメディア的見方と統計的見方を対比させながら説明する。予測に関する正しい見方、新しい見方、考え方の紹介は枚挙に尽きないが、評者は特に次の二点に注意を向けたい。一つはスパコン、メガデータ、ウェブ等は、天気予報を例外とし、予報の進歩には殆ど貢献してない。二つ目は日本軍の真珠湾攻撃は何故予測できなかったか？　そしてそこから発生する陰謀説。大人の知的好奇心を満足させてくれる一押しの良書です