Aviation Week & Space Technology released early this month its list of the Best Aerospace & Defense Companies for 2009. The methodology [1 June 2009, p. 65] behind the rankings requires some comment, as it constitutes one of the worst financial analyses I’ve ever seen. If you want all the details, click on that word above for Aviation Week’s own one-page elucidation of the approach. After reading that a few times, the rest of this will make more sense.
The first and foremost problem lies with the regression used to "validate", as it were, the approach. The logic appears circular. Aviation Week’s analyst seems to have regressed one set of accounting ratios on another set of ratios, and claimed a correlation coefficient of 99 percent. The magazine hasn’t published the list of the underlying ratios, but we might guess that the four response variables incorporated many of the supposedly independent variables. Rather than bragging about that ninety-nine, the editors should have recognized it as a red flag. The effect is at least one of collinearity, and the importance of avoiding it is taught in every introductory multivariate statistics class. It’s even possible that the analyst was essentially regressing X on X. The error is so embarrassing that a colleague and I each read the paragraph describing the approach several times, just to be certain that the writer really was saying something that silly.
The second problem is found in the egregious data-mining. Given the probably irredeemable collinearity of the method, the issue may be merely stylistic at this point, but it's still severe. Whoever piled on all these metrics has clearly missed the point of Occam's Razor. A model that relies on so many inputs is not just inelegant; it actually lacks explanatory power, because comparisons cannot be made with reasonable expectation as to the underlying differentiators.
The third problem is the lack of a clear and defensible objective function. Attempting to look forward, the analyst chose not to simply measure total shareholder returns—even though it seems that all the companies in his large-company population are publicly quoted. We might pause for a moment to think about the forty years of financial research that tell us that predicting future performance across companies is simply impossible. As Gene Fama put it in his blog back in April, “it's understandable, however, that hedge fund managers [and equities analysts] are immune to this evidence, since it's a threat to their existence.” But let's forget about that for the moment, and let this guy try. His objective function is a qualitative grab bag of four substantially overlapping measures—ROIC, Earnings "Momentum" (see below), Asset Management, and Financial Health—that he equally weights. He doesn't even take a stand on which ones might be more important.
The first problem, if it is indeed as severe as it appears, invalidates the method and impeaches the findings. The second problem would as well, even if a truly independent set of variables had been chosen for the model. The third problem would at least cause us to smirk strongly, even if we could bear to stay in the room. Yet there are still more problems.
One of the aforementioned four estimators is earnings momentum. Earnings are indeed classically easy to manipulate, at least for a while. Earnings momentum then becomes an addictive game for dishonest CFOs and day-traders alike. That's why everyone who's serious in finance looks at free cash flow, which is hard to manipulate, even if it requires a little work to calculate. Free cash flow, however, appears nowhere in the model.
I would mention that companies in the range of $3–10 billion in revenue are judged according to two sets of criteria: one for companies from $1–5 billion, and another for companies with more than $5 billion. This would be material were it the only problem. At this point, including it on the list of complaints is almost boring.
I wish that I were overstating just how bad this methodology really is. To produce something like this, one must have slept through or missed his introductory statistics, finance, accounting, and microeconomics classes. My best guess is that whoever is responsible was recently laid off from a job as a financial analyst at Freddie, Fannie, or a Norwegian village’s pension fund. I anticipate that Aviation Week will do better next year—as it’s a great magazine, it really must.
Comments