Thoughts from Dennis

I have an Uncle who holds some strong opinions. He wrote the following in an email, and I've got about 3 case studies in my piles illustrating the kind of bias in statistical analyses that are so common.

Below is his comments, my explaination, and a discussion of a spectacularly biased analysis from 1998.

+++++++++++++++++++++++++++++

An Uncle wrote:

I also have studies statistics and have use numbers to make a lot of decisions. It works!! But only when you use the numbers as numbers were meant to be used--without trying to prove a theory that you already have about the outcome. I have used numbers to determine a lot of things about costs of military systems and times to scrap systems instead of updating them and numbers only lie when political pressure comes to bear on the solution. For example, a system that should be scrapped will close a base in a powerful senator's state. I fear that the same kind of thing happens a lot when the data you need comes from sources that can easily be altered or subjectively changed as necessary to prove a point. Add to that, when analysts have as liberal a bias as you seem to have, the results can prove anything they say they do. For many years I felt that "Statistics" was a science but now I have concluded from what I see the liberal press trying to tell me about the world we live in, it has become worthless. It has about as much value to the future of this nation and the world as our education system and our legal system. In the real world the pendulum always swings so far in either direction and then comes back to reason. Talk to a liberal and if the swing is not going farther to the left, the world is going to end.

+++++

It's a fair criticism: many folks use skewed analyses to "prove" their point, regardless of their political orientation--even politically neutral folks. And it's not easy for casual observers to see the flaws in an argument, and unless you've got some insights into the processes that went into an analysis, it's hard to judge. As someone who makes a living from mathematics, I look for technical merits & fallacies, especially with an eye toward whether they are using appropriate techniques for the the data at hand.

And of course, often when all you have in front of you is a newspaper article, it is impossible to judge whether the analysis is competent or not; for that, I rely (as much as possible) on original articles from scientific journals, and (also if possible) other articles on the same topic, possibly reaching different conclusions.

Now for the rest of this post: back in '98 the Energy Information Agency published an estimate of the cost of the USA complying with the Kyota agreement on controlling CO2 emissions. As experts responsible for estimating the costs of energy costs of the nation given current conditions, they carried a lot of weight, and the estimated cost was whopping huge (though I don't remember what it was). One big problem though that was not noted: while they did a great job of say, estimating the cost of gasoline over a year given current supplies and trends, they had a history of being way off in longer term estimates. (For instance, they overestimated the cost of reducing sulfer emissions by a factor of 4!)

Below is an excellent analysis of the why the EIA CO2 study was so far off. It's a great case study of how analyses can be biased in subtle (or flagrent!) ways.

================================================

Date: Fri, 16 Oct 98 09:41:51 -0500
To:
Subject: [EENet]: Romm review of EIA Climate study
Reply-To: eenet@esa-online.org

Friends,

Several days ago an article from the Washington Post was posted to the list on the report from the Energy Information Administration on the projected economic costs of the Kyoto Protocol to the US. Below is an excellent critique by Dr. Joe Romm.
Jim Ball
________________________________________________________________________

FOR IMMEDIATE RELEASE CONTACT: Jon Coifman, 202/463-6670
WEDNESDAY, OCTOBER 14, 1998 Dr. Joe Romm, 202/483-1024
NEW GOV'T STUDY OF GLOBAL WARMING TREATY
RENDERED IRRELEVANT' BY MAJOR FLAWS

The Same Defective Methods by the Same Federal Agency Recently Overestimated Clean Air Act Costs by 400 Percent

A new 250-page study by the Department of Energy on the costs of meeting the global warming treaty is so riddled with economic flaws, analytical errors and outrageous policy assumptions that it is "rendered completely irrelevant," according to Dr. Joseph Romm, who recently left his post as DOE's top official in charge of energy efficiency and renewable energy technology.

The analysis was prepared by the Energy Information Administration (EIA), which is known equally among energy economists for both excellent short-term forecasting and grossly inaccurate long-term prognostications, according to Romm.

"The new study continues EIA's history of erroneous long-range predictions, including their four-fold overestimate on the costs of reducing sulfur dioxide pollution just a few years ago," he said.

According to Romm, the EIA study wildly overstates the carbon price required to reach various levels of emissions reductions, resulting in grossly inflated projections for increased energy prices and short-term, transitional impact on economic output. Even with its built-in bias, the new study found the U.S. could meet the Kyoto treaty terms with "no appreciable change in the long-term [GDP] growth rate."

"The EIA analysis expressly forbids the market from reacting intelligently." said Romm. "They overestimate the costs and underestimate the savings offered by both low-carbon fuels and key alternative technologies, including cogeneration, fuel cells, and advanced end-use efficiency."
The study yields grossly inflated cost estimates because the EIA has:

-- Ignored a mandate from Congress, failing to model the actual treaty rules;

-- Arbitrarily prohibited all energy users from responding intelligently to market signals;

-- Barred all proactive government policies;

-- Frozen monopoly utility regulation for two more decades;

-- Ignored existing laws like the Clean Air Act;

-- Ignore or inaccurately modeled key technologies;

-- Used an economic model it knew was highly inappropriate;

-- Did all this knowing these are exactly the same reasons it overestimated the cost of clean air laws just a few years ago.


GARBAGE IN, GARBAGE OUT:
Fatal Flaws Undermine EIA's Analysis

The problem with EIA's economic analysis lies in the assumptions used by agency modelers, assumptions that are even more pessimistic than those in the deliberately bleak forecasts constructed by treaty opponents.

FATAL FLAW #1:
EIA Prohibits Emissions Reductions from Starting Until 2005.

EIA makes the highly unlikely assumption that we will wait until just three years before the Kyoto mandate kicks in 2008 -- to even start reducing emissions, forcing the economy to absorb sharper, more drastic cuts than necessary. And the delay means the abrupt changes will start from a higher emissions baseline.

In reality, energy intensive industries can and will make sensible, long-term capital decisions that dramatically reduce greenhouse pollution long before 2005, but EIA number crunchers forbid them from doing so -- forcing the economy in their model to turn on a dime. EIA effectively overrules the many voluntary reductions underway by companies like British Petroleum and the U.S. steel industry, which have pledged 10 percent emissions cuts from 1990 levels by 2010.

FATAL FLAW #2:
EIA Ignores Explicit Congressional Instructions, Kyoto Treaty Provisions.

By expressly prohibiting either anticipatory action or emissions cuts in their model until after 2005, EIA ignores the explicit instructions of the House Science Committee Members who commissioned the report to "assume that the United States will begin to respond by 2005 to both reflect anticipatory actions by industry and consumers, and to meet the Kyoto Protocol's Article 3.2 requirement that ‘Each Party included in Annex I shall, by 2005, have made demonstrable progress achieving its commitments under this Protocol.' " [emphasis added]

FATAL FLAW #3:
EIA Assumes There Will Be No Policies to Reduce Costs of Transition.

EIA further assumes that Congress will not pass a single law, tax break or other policies to make it easier and cheaper to reduce emissions. In fact, EIA ignores key policies and trends already underway that will sharply reduce costs:

EIA freezes current utility monopolies until 2020, even though 17 states and the U.S. Congress are already well on the way to deregulation, even though EIA acknowledges "this may not be appropriate in the near future" Besides reducing baseline rates, competitive electricity pricing will open markets for cleaner, more efficient technology.

EIA ignores tougher Clean Air Act standards already being implemented for ozone and particulate pollution that will sharply curtail the artificial price advantage now enjoyed by obsolete, inefficient coal power plants over cleaner, more efficient alternatives (including gas, cogeneration and renewables).

FATAL FLAW #4: EIA Ignores Technology Changes Already in the Marketplace

EIA ignores or artificially limits technologies that every other major study has indicated would play a major role in reducing the impact of Kyoto:

COGENERATION: Widely seen as a major opportunity for greenhouse pollution cuts, since existing ‘cogen' technology cuts carbon emissions by almost half with zero increase in energy costs. Even if delivered coal prices (the price of coal plus new carbon tax or permit prices) rise 70 percent and electricity prices double, adoption of this proven money-saver rises only three to four percent from baseline by 2010 -- and then stays flat until 2020!

RENEWABLES: EIA artificially constrains renewable power; even when carbon and electricity prices soar through the roof in the model, renewable market share hardly budges. Utilities are the only sector allowed by the model to begin planning investments before 2005, yet event with "perfect foresight" that coal prices will jump 350 percent, renewable capacity is only 10 percent higher than the baseline in 2010.

FUEL CELLS: Even in its most extreme scenario -- a carbon tax of $300 per ton and a doubling of conventional electricity prices -- EIA assumes that today's rapidly-evolving fuel cell technology will play no part in the power generating market for the duration of their analysis.

FATAL FLAW #5: EIA Uses Calculating Tool Rated "Highly Inappropriate" to Figure Costs

After building this set of restrictive assumptions, EIA applied them to a commercial short-term forecasting tool known as the DRI model. This tool is very reliable for short-term analysis, but is notoriously flawed when it comes to long-range accuracy. As economists Dale Jorgenson and William Nordhaus noted in April, 1997, with specific respect to global warming policy:

"For longer term projections [the DRI model] is highly inappropriate and has some important limitations that are well-known to the community of economic forecasters. For example, the DRI long term projections are extremely sensitive to assumptions about energy prices. This feature is totally at odds with most empirical work ... "

FATAL FLAW #6:
EIA Repeats Admitted Errors that Ruined Previous Forecasts

EIA makes the exact same mistakes it made just a few years ago when overestimated the cost of sulfur pollution cuts by a factor of four. EIA's accounting of what went wrong then is conveniently included in the new Kyoto report. The explanation applies equally to the new analysis:

"There has also been downward pressure on . . . costs because generators have taken actions to comply with the SO2 limitations earlier than anticipated."

"There has been more fuel switching . . . than previously anticipated."

"Finally, technology improvements have lowered the costs of flue-gas desulfurization technologies, or scrubbers . . . The cost of SO2 compliance was overestimated to a large extent because compliance relied on scrubbing, a relatively new technology with which there was little experience."
A virtually identical set of errors work together to vastly inflate the cost estimates in the global warming treaty analysis.

"EIA's study is wholly irrelevant and has no bearing on the actual impacts of the of the Kyoto treaty on U.S. markets of our economic activity," said Romm. "EIA has simply shown what it will cost if we use a brain-dead approach to meet the treaty."

==========================================

posted by Dennis Sweitzer @ 10:40 AM