Blog Archive

11 March 2013

Global Warming's Terrifying New Math


Global Warming's Terrifying New Math

Three simple numbers that add up to global catastrophe - and that make clear who the real enemy is

reckoning illo
Illustration by Edel Rodriguez
If the pictures of those towering wildfires in Colorado haven't convinced you, or the size of your AC bill this summer, here are some hard numbers about climate change: June broke or tied 3,215 high-temperature records across the United States. That followed the warmest May on record for the Northern Hemisphere – the 327th consecutive month in which the temperature of the entire globe exceeded the 20th-century average, the odds of which occurring by simple chance were 3.7 x 10-99, a number considerably larger than the number of stars in the universe.
Meteorologists reported that this spring was the warmest ever recorded for our nation – in fact, it crushed the old record by so much that it represented the "largest temperature departure from average of any season on record." The same week, Saudi authorities reported that it had rained in Mecca despite a temperature of 109 degrees, the hottest downpour in the planet's history.
Not that our leaders seemed to notice. Last month the world's nations, meeting in Rio for the 20th-anniversary reprise of a massive 1992 environmental summit, accomplished nothing. Unlike George H.W. Bush, who flew in for the first conclave, Barack Obama didn't even attend. It was "a ghost of the glad, confident meeting 20 years ago," the British journalist George Monbiot wrote; no one paid it much attention, footsteps echoing through the halls "once thronged by multitudes." Since I wrote one of the first books for a general audience about global warming way back in 1989, and since I've spent the intervening decades working ineffectively to slow that warming, I can say with some confidence that we're losing the fight, badly and quickly – losing it because, most of all, we remain in denial about the peril that human civilization is in.
When we think about global warming at all, the arguments tend to be ideological, theological and economic. But to grasp the seriousness of our predicament, you just need to do a little math. For the past year, an easy and powerful bit of arithmetical analysis first published by financial analysts in the U.K. has been making the rounds of environmental conferences and journals, but it hasn't yet broken through to the larger public. This analysis upends most of the conventional political thinking about climate change. And it allows us to understand our precarious – our almost-but-not-quite-finally hopeless – position with three simple numbers.
The First Number: 2° Celsius
If the movie had ended in Hollywood fashion, the Copenhagen climate conference in 2009 would have marked the culmination of the global fight to slow a changing climate. The world's nations had gathered in the December gloom of the Danish capital for what a leading climate economist, Sir Nicholas Stern of Britain, called the "most important gathering since the Second World War, given what is at stake." As Danish energy minister Connie Hedegaard, who presided over the conference, declared at the time: "This is our chance. If we miss it, it could take years before we get a new and better one. If ever."
In the event, of course, we missed it. Copenhagen failed spectacularly. Neither China nor the United States, which between them are responsible for 40 percent of global carbon emissions, was prepared to offer dramatic concessions, and so the conference drifted aimlessly for two weeks until world leaders jetted in for the final day. Amid considerable chaos, President Obama took the lead in drafting a face-saving "Copenhagen Accord" that fooled very few. Its purely voluntary agreements committed no one to anything, and even if countries signaled their intentions to cut carbon emissions, there was no enforcement mechanism. "Copenhagen is a crime scene tonight," an angry Greenpeace official declared, "with the guilty men and women fleeing to the airport." Headline writers were equally brutal: COPENHAGEN: THE MUNICH OF OUR TIMES? asked one.
The accord did contain one important number, however. In Paragraph 1, it formally recognized "the scientific view that the increase in global temperature should be below two degrees Celsius." And in the very next paragraph, it declared that "we agree that deep cuts in global emissions are required... so as to hold the increase in global temperature below two degrees Celsius." By insisting on two degrees – about 3.6 degrees Fahrenheit – the accord ratified positions taken earlier in 2009 by the G8, and the so-called Major Economies Forum. It was as conventional as conventional wisdom gets. The number first gained prominence, in fact, at a 1995 climate conference chaired by Angela Merkel, then the German minister of the environment and now the center-right chancellor of the nation.
Some context: So far, we've raised the average temperature of the planet just under 0.8 degrees Celsius, and that has caused far more damage than most scientists expected. (A third of summer sea ice in the Arctic is gone, the oceans are 30 percent more acidic, and since warm air holds more water vapor than cold, the atmosphere over the oceans is a shocking five percent wetter, loading the dice for devastating floods.) Given those impacts, in fact, many scientists have come to think that two degrees is far too lenient a target. "Any number much above one degree involves a gamble," writes Kerry Emanuel of MIT, a leading authority on hurricanes, "and the odds become less and less favorable as the temperature goes up." Thomas Lovejoy, once the World Bank's chief biodiversity adviser, puts it like this: "If we're seeing what we're seeing today at 0.8 degrees Celsius, two degrees is simply too much." NASA scientist James Hansen, the planet's most prominent climatologist, is even blunter: "The target that has been talked about in international negotiations for two degrees of warming is actually a prescription for long-term disaster." At the Copenhagen summit, a spokesman for small island nations warned that many would not survive a two-degree rise: "Some countries will flat-out disappear." When delegates from developing nations were warned that two degrees would represent a "suicide pact" for drought-stricken Africa, many of them started chanting, "One degree, one Africa."
Despite such well-founded misgivings, political realism bested scientific data, and the world settled on the two-degree target – indeed, it's fair to say that it's the only thing about climate change the world has settled on. All told, 167 countries responsible for more than 87 percent of the world's carbon emissions have signed on to the Copenhagen Accord, endorsing the two-degree target. Only a few dozen countries have rejected it, including Kuwait, Nicaragua and Venezuela. Even the United Arab Emirates, which makes most of its money exporting oil and gas, signed on. The official position of planet Earth at the moment is that we can't raise the temperature more than two degrees Celsius – it's become the bottomest of bottom lines. Two degrees.

The Second Number: 565 Gigatons
Scientists estimate that humans can pour roughly 565 more gigatons of carbon dioxide into the atmosphere by midcentury and still have some reasonable hope of staying below two degrees. ("Reasonable," in this case, means four chances in five, or somewhat worse odds than playing Russian roulette with a six-shooter.)
This idea of a global "carbon budget" emerged about a decade ago, as scientists began to calculate how much oil, coal and gas could still safely be burned. Since we've increased the Earth's temperature by 0.8 degrees so far, we're currently less than halfway to the target. But, in fact, computer models calculate that even if we stopped increasing CO2 now, the temperature would likely still rise another 0.8 degrees, as previously released carbon continues to overheat the atmosphere. That means we're already three-quarters of the way to the two-degree target.
How good are these numbers? No one is insisting that they're exact, but few dispute that they're generally right. The 565-gigaton figure was derived from one of the most sophisticated computer-simulation models that have been built by climate scientists around the world over the past few decades. And the number is being further confirmed by the latest climate-simulation models currently being finalized in advance of the next report by the Intergovernmental Panel on Climate Change. "Looking at them as they come in, they hardly differ at all," says Tom Wigley, an Australian climatologist at the National Center for Atmospheric Research. "There's maybe 40 models in the data set now, compared with 20 before. But so far the numbers are pretty much the same. We're just fine-tuning things. I don't think much has changed over the last decade." William Collins, a senior climate scientist at the Lawrence Berkeley National Laboratory, agrees. "I think the results of this round of simulations will be quite similar," he says. "We're not getting any free lunch from additional understanding of the climate system."
We're not getting any free lunch from the world's economies, either. With only a single year's lull in 2009 at the height of the financial crisis, we've continued to pour record amounts of carbon into the atmosphere, year after year. In late May, the International Energy Agency published its latest figures – CO2 emissions last year rose to 31.6 gigatons, up 3.2 percent from the year before. America had a warm winter and converted more coal-fired power plants to natural gas, so its emissions fell slightly; China kept booming, so its carbon output (which recently surpassed the U.S.) rose 9.3 percent; the Japanese shut down their fleet of nukes post-Fukushima, so their emissions edged up 2.4 percent. "There have been efforts to use more renewable energy and improve energy efficiency," said Corinne Le Quéré, who runs England's Tyndall Centre for Climate Change Research. "But what this shows is that so far the effects have been marginal." In fact, study after study predicts that carbon emissions will keep growing by roughly three percent a year – and at that rate, we'll blow through our 565-gigaton allowance in 16 years, around the time today's preschoolers will be graduating from high school. "The new data provide further evidence that the door to a two-degree trajectory is about to close," said Fatih Birol, the IEA's chief economist. In fact, he continued, "When I look at this data, the trend is perfectly in line with a temperature increase of about six degrees." That's almost 11 degrees Fahrenheit, which would create a planet straight out of science fiction.
So, new data in hand, everyone at the Rio conference renewed their ritual calls for serious international action to move us back to a two-degree trajectory. The charade will continue in November, when the next Conference of the Parties (COP) of the U.N. Framework Convention on Climate Change convenes in Qatar. This will be COP 18 – COP 1 was held in Berlin in 1995, and since then the process has accomplished essentially nothing. Even scientists, who are notoriously reluctant to speak out, are slowly overcoming their natural preference to simply provide data. "The message has been consistent for close to 30 years now," Collins says with a wry laugh, "and we have the instrumentation and the computer power required to present the evidence in detail. If we choose to continue on our present course of action, it should be done with a full evaluation of the evidence the scientific community has presented." He pauses, suddenly conscious of being on the record. "I should say, a fuller evaluation of the evidence."
So far, though, such calls have had little effect. We're in the same position we've been in for a quarter-century: scientific warning followed by political inaction. Among scientists speaking off the record, disgusted candor is the rule. One senior scientist told me, "You know those new cigarette packs, where governments make them put a picture of someone with a hole in their throats? Gas pumps should have something like that."
The Third Number: 2,795 Gigatons
This number is the scariest of all – one that, for the first time, meshes the political and scientific dimensions of our dilemma. It was highlighted last summer by the Carbon Tracker Initiative, a team of London financial analysts and environmentalists who published a report in an effort to educate investors about the possible risks that climate change poses to their stock portfolios. The number describes the amount of carbon already contained in the proven coal and oil and gas reserves of the fossil-fuel companies, and the countries (think Venezuela or Kuwait) that act like fossil-fuel companies. In short, it's the fossil fuel we're currently planning to burn. And the key point is that this new number – 2,795 – is higher than 565. Five times higher.
The Carbon Tracker Initiative – led by James Leaton, an environmentalist who served as an adviser at the accounting giant PricewaterhouseCoopers – combed through proprietary databases to figure out how much oil, gas and coal the world's major energy companies hold in reserve. The numbers aren't perfect – they don't fully reflect the recent surge in unconventional energy sources like shale gas, and they don't accurately reflect coal reserves, which are subject to less stringent reporting requirements than oil and gas. But for the biggest companies, the figures are quite exact: If you burned everything in the inventories of Russia's Lukoil and America's ExxonMobil, for instance, which lead the list of oil and gas companies, each would release more than 40 gigatons of carbon dioxide into the atmosphere.
Which is exactly why this new number, 2,795 gigatons, is such a big deal. Think of two degrees Celsius as the legal drinking limit – equivalent to the 0.08 blood-alcohol level below which you might get away with driving home. The 565 gigatons is how many drinks you could have and still stay below that limit – the six beers, say, you might consume in an evening. And the 2,795 gigatons? That's the three 12-packs the fossil-fuel industry has on the table, already opened and ready to pour.
We have five times as much oil and coal and gas on the books as climate scientists think is safe to burn. We'd have to keep 80 percent of those reserves locked away underground to avoid that fate. Before we knew those numbers, our fate had been likely. Now, barring some massive intervention, it seems certain.
Yes, this coal and gas and oil is still technically in the soil. But it's already economically aboveground – it's figured into share prices, companies are borrowing money against it, nations are basing their budgets on the presumed returns from their patrimony. It explains why the big fossil-fuel companies have fought so hard to prevent the regulation of carbon dioxide – those reserves are their primary asset, the holding that gives their companies their value. It's why they've worked so hard these past years to figure out how to unlock the oil in Canada's tar sands, or how to drill miles beneath the sea, or how to frack the Appalachians.

If you told Exxon or Lukoil that, in order to avoid wrecking the climate, they couldn't pump out their reserves, the value of their companies would plummet. John Fullerton, a former managing director at JP Morgan who now runs the Capital Institute, calculates that at today's market value, those 2,795 gigatons of carbon emissions are worth about $27 trillion. Which is to say, if you paid attention to the scientists and kept 80 percent of it underground, you'd be writing off $20 trillion in assets. The numbers aren't exact, of course, but that carbon bubble makes the housing bubble look small by comparison. It won't necessarily burst – we might well burn all that carbon, in which case investors will do fine. But if we do, the planet will crater. You can have a healthy fossil-fuel balance sheet, or a relatively healthy planet – but now that we know the numbers, it looks like you can't have both. Do the math: 2,795 is five times 565. That's how the story ends.
So far, as I said at the start, environmental efforts to tackle global warming have failed. The planet's emissions of carbon dioxide continue to soar, especially as developing countries emulate (and supplant) the industries of the West. Even in rich countries, small reductions in emissions offer no sign of the real break with the status quo we'd need to upend the iron logic of these three numbers. Germany is one of the only big countries that has actually tried hard to change its energy mix; on one sunny Saturday in late May, that northern-latitude nation generated nearly half its power from solar panels within its borders. That's a small miracle – and it demonstrates that we have the technology to solve our problems. But we lack the will. So far, Germany's the exception; the rule is ever more carbon.
This record of failure means we know a lot about what strategies don't work. Green groups, for instance, have spent a lot of time trying to change individual lifestyles: the iconic twisty light bulb has been installed by the millions, but so have a new generation of energy-sucking flatscreen TVs. Most of us are fundamentally ambivalent about going green: We like cheap flights to warm places, and we're certainly not going to give them up if everyone else is still taking them. Since all of us are in some way the beneficiaries of cheap fossil fuel, tackling climate change has been like trying to build a movement against yourself – it's as if the gay-rights movement had to be constructed entirely from evangelical preachers, or the abolition movement from slaveholders.
People perceive – correctly – that their individual actions will not make a decisive difference in the atmospheric concentration of CO2; by 2010, a poll found that "while recycling is widespread in America and 73 percent of those polled are paying bills online in order to save paper," only four percent had reduced their utility use and only three percent had purchased hybrid cars. Given a hundred years, you could conceivably change lifestyles enough to matter – but time is precisely what we lack.
A more efficient method, of course, would be to work through the political system, and environmentalists have tried that, too, with the same limited success. They've patiently lobbied leaders, trying to convince them of our peril and assuming that politicians would heed the warnings. Sometimes it has seemed to work. Barack Obama, for instance, campaigned more aggressively about climate change than any president before him – the night he won the nomination, he told supporters that his election would mark the moment "the rise of the oceans began to slow and the planet began to heal." And he has achieved one significant change: a steady increase in the fuel efficiency mandated for automobiles. It's the kind of measure, adopted a quarter-century ago, that would have helped enormously. But in light of the numbers I've just described, it's obviously a very small start indeed.
At this point, effective action would require actually keeping most of the carbon the fossil-fuel industry wants to burn safely in the soil, not just changing slightly the speed at which it's burned. And there the president, apparently haunted by the still-echoing cry of "Drill, baby, drill," has gone out of his way to frack and mine. His secretary of interior, for instance, opened up a huge swath of the Powder River Basin in Wyoming for coal extraction: The total basin contains some 67.5 gigatons worth of carbon (or more than 10 percent of the available atmospheric space). He's doing the same thing with Arctic and offshore drilling; in fact, as he explained on the stump in March, "You have my word that we will keep drilling everywhere we can... That's a commitment that I make." The next day, in a yard full of oil pipe in Cushing, Oklahoma, the president promised to work on wind and solar energy but, at the same time, to speed up fossil-fuel development: "Producing more oil and gas here at home has been, and will continue to be, a critical part of an all-of-the-above energy strategy." That is, he's committed to finding even more stock to add to the 2,795-gigaton inventory of unburned carbon.
Sometimes the irony is almost Borat-scale obvious: In early June, Secretary of State Hillary Clinton traveled on a Norwegian research trawler to see firsthand the growing damage from climate change. "Many of the predictions about warming in the Arctic are being surpassed by the actual data," she said, describing the sight as "sobering." But the discussions she traveled to Scandinavia to have with other foreign ministers were mostly about how to make sure Western nations get their share of the estimated $9 trillion in oil (that's more than 90 billion barrels, or 37 gigatons of carbon) that will become accessible as the Arctic ice melts. Last month, the Obama administration indicated that it would give Shell permission to start drilling in sections of the Arctic.
Almost every government with deposits of hydrocarbons straddles the same divide. Canada, for instance, is a liberal democracy renowned for its internationalism – no wonder, then, that it signed on to the Kyoto treaty, promising to cut its carbon emissions substantially by 2012. But the rising price of oil suddenly made the tar sands of Alberta economically attractive – and since, as NASA climatologist James Hansen pointed out in May, they contain as much as 240 gigatons of carbon (or almost half of the available space if we take the 565 limit seriously), that meant Canada's commitment to Kyoto was nonsense. In December, the Canadian government withdrew from the treaty before it faced fines for failing to meet its commitments.
The same kind of hypocrisy applies across the ideological board: In his speech to the Copenhagen conference, Venezuela's Hugo Chavez quoted Rosa Luxemburg, Jean-Jacques Rousseau and "Christ the Redeemer," insisting that "climate change is undoubtedly the most devastating environmental problem of this century." But the next spring, in the Simon Bolivar Hall of the state-run oil company, he signed an agreement with a consortium of international players to develop the vast Orinoco tar sands as "the most significant engine for a comprehensive development of the entire territory and Venezuelan population." The Orinoco deposits are larger than Alberta's – taken together, they'd fill up the whole available atmospheric space.
So: the paths we have tried to tackle global warming have so far produced only gradual, halting shifts. A rapid, transformative change would require building a movement, and movements require enemies. As John F. Kennedy put it, "The civil rights movement should thank God for Bull Connor. He's helped it as much as Abraham Lincoln." And enemies are what climate change has lacked.
But what all these climate numbers make painfully, usefully clear is that the planet does indeed have an enemy – one far more committed to action than governments or individuals. Given this hard math, we need to view the fossil-fuel industry in a new light. It has become a rogue industry, reckless like no other force on Earth. It is Public Enemy Number One to the survival of our planetary civilization. "Lots of companies do rotten things in the course of their business – pay terrible wages, make people work in sweatshops – and we pressure them to change those practices," says veteran anti-corporate leader Naomi Klein, who is at work on a book about the climate crisis. "But these numbers make clear that with the fossil-fuel industry, wrecking the planet is their business model. It's what they do."
According to the Carbon Tracker report, if Exxon burns its current reserves, it would use up more than seven percent of the available atmospheric space between us and the risk of two degrees. BP is just behind, followed by the Russian firm Gazprom, then Chevron, ConocoPhillips and Shell, each of which would fill between three and four percent. Taken together, just these six firms, of the 200 listed in the Carbon Tracker report, would use up more than a quarter of the remaining two-degree budget. Severstal, the Russian mining giant, leads the list of coal companies, followed by firms like BHP Billiton and Peabody. The numbers are simply staggering – this industry, and this industry alone, holds the power to change the physics and chemistry of our planet, and they're planning to use it.
They're clearly cognizant of global warming – they employ some of the world's best scientists, after all, and they're bidding on all those oil leases made possible by the staggering melt of Arctic ice. And yet they relentlessly search for more hydrocarbons – in early March, Exxon CEO Rex Tillerson told Wall Street analysts that the company plans to spend $37 billion a year through 2016 (about $100 million a day) searching for yet more oil and gas.

There's not a more reckless man on the planet than Tillerson. Late last month, on the same day the Colorado fires reached their height, he told a New York audience that global warming is real, but dismissed it as an "engineering problem" that has "engineering solutions." Such as? "Changes to weather patterns that move crop-production areas around – we'll adapt to that." This in a week when Kentucky farmers were reporting that corn kernels were "aborting" in record heat, threatening a spike in global food prices. "The fear factor that people want to throw out there to say, 'We just have to stop this,' I do not accept," Tillerson said. Of course not – if he did accept it, he'd have to keep his reserves in the ground. Which would cost him money. It's not an engineering problem, in other words – it's a greed problem.
You could argue that this is simply in the nature of these companies – that having found a profitable vein, they're compelled to keep mining it, more like efficient automatons than people with free will. But as the Supreme Court has made clear, they are people of a sort. In fact, thanks to the size of its bankroll, the fossil-fuel industry has far more free will than the rest of us. These companies don't simply exist in a world whose hungers they fulfill – they help create the boundaries of that world.
Left to our own devices, citizens might decide to regulate carbon and stop short of the brink; according to a recent poll, nearly two-thirds of Americans would back an international agreement that cut carbon emissions 90 percent by 2050. But we aren't left to our own devices. The Koch brothers, for instance, have a combined wealth of $50 billion, meaning they trail only Bill Gates on the list of richest Americans. They've made most of their money in hydrocarbons, they know any system to regulate carbon would cut those profits, and they reportedly plan to lavish as much as $200 million on this year's elections. In 2009, for the first time, the U.S. Chamber of Commerce surpassed both the Republican and Democratic National Committees on political spending; the following year, more than 90 percent of the Chamber's cash went to GOP candidates, many of whom deny the existence of global warming. Not long ago, the Chamber even filed a brief with the EPA urging the agency not to regulate carbon – should the world's scientists turn out to be right and the planet heats up, the Chamber advised, "populations can acclimatize to warmer climates via a range of behavioral, physiological and technological adaptations." As radical goes, demanding that we change our physiology seems right up there.
Environmentalists, understandably, have been loath to make the fossil-fuel industry their enemy, respecting its political power and hoping instead to convince these giants that they should turn away from coal, oil and gas and transform themselves more broadly into "energy companies." Sometimes that strategy appeared to be working – emphasis on appeared. Around the turn of the century, for instance, BP made a brief attempt to restyle itself as "Beyond Petroleum," adapting a logo that looked like the sun and sticking solar panels on some of its gas stations. But its investments in alternative energy were never more than a tiny fraction of its budget for hydrocarbon exploration, and after a few years, many of those were wound down as new CEOs insisted on returning to the company's "core business." In December, BP finally closed its solar division. Shell shut down its solar and wind efforts in 2009. The five biggest oil companies have made more than $1 trillion in profits since the millennium – there's simply too much money to be made on oil and gas and coal to go chasing after zephyrs and sunbeams.
Much of that profit stems from a single historical accident: Alone among businesses, the fossil-fuel industry is allowed to dump its main waste, carbon dioxide, for free. Nobody else gets that break – if you own a restaurant, you have to pay someone to cart away your trash, since piling it in the street would breed rats. But the fossil-fuel industry is different, and for sound historical reasons: Until a quarter-century ago, almost no one knew that CO2 was dangerous. But now that we understand that carbon is heating the planet and acidifying the oceans, its price becomes the central issue.
If you put a price on carbon, through a direct tax or other methods, it would enlist markets in the fight against global warming. Once Exxon has to pay for the damage its carbon is doing to the atmosphere, the price of its products would rise. Consumers would get a strong signal to use less fossil fuel – every time they stopped at the pump, they'd be reminded that you don't need a semimilitary vehicle to go to the grocery store. The economic playing field would now be a level one for nonpolluting energy sources. And you could do it all without bankrupting citizens – a so-called "fee-and-dividend" scheme would put a hefty tax on coal and gas and oil, then simply divide up the proceeds, sending everyone in the country a check each month for their share of the added costs of carbon. By switching to cleaner energy sources, most people would actually come out ahead.
There's only one problem: Putting a price on carbon would reduce the profitability of the fossil-fuel industry. After all, the answer to the question "How high should the price of carbon be?" is "High enough to keep those carbon reserves that would take us past two degrees safely in the ground." The higher the price on carbon, the more of those reserves would be worthless. The fight, in the end, is about whether the industry will succeed in its fight to keep its special pollution break alive past the point of climate catastrophe, or whether, in the economists' parlance, we'll make them internalize those externalities.
It's not clear, of course, that the power of the fossil-fuel industry can be broken. The U.K. analysts who wrote the Carbon Tracker report and drew attention to these numbers had a relatively modest goal – they simply wanted to remind investors that climate change poses a very real risk to the stock prices of energy companies. Say something so big finally happens (a giant hurricane swamps Manhattan, a megadrought wipes out Midwest agriculture) that even the political power of the industry is inadequate to restrain legislators, who manage to regulate carbon. Suddenly those Chevron reserves would be a lot less valuable, and the stock would tank. Given that risk, the Carbon Tracker report warned investors to lessen their exposure, hedge it with some big plays in alternative energy.
"The regular process of economic evolution is that businesses are left with stranded assets all the time," says Nick Robins, who runs HSBC's Climate Change Centre. "Think of film cameras, or typewriters. The question is not whether this will happen. It will. Pension systems have been hit by the dot-com and credit crunch. They'll be hit by this." Still, it hasn't been easy to convince investors, who have shared in the oil industry's record profits. "The reason you get bubbles," sighs Leaton, "is that everyone thinks they're the best analyst – that they'll go to the edge of the cliff and then jump back when everyone else goes over."
So pure self-interest probably won't spark a transformative challenge to fossil fuel. But moral outrage just might – and that's the real meaning of this new math. It could, plausibly, give rise to a real movement.
Once, in recent corporate history, anger forced an industry to make basic changes. That was the campaign in the 1980s demanding divestment from companies doing business in South Africa. It rose first on college campuses and then spread to municipal and state governments; 155 campuses eventually divested, and by the end of the decade, more than 80 cities, 25 states and 19 counties had taken some form of binding economic action against companies connected to the apartheid regime. "The end of apartheid stands as one of the crowning accomplishments of the past century," as Archbishop Desmond Tutu put it, "but we would not have succeeded without the help of international pressure," especially from "the divestment movement of the 1980s."

The fossil-fuel industry is obviously a tougher opponent, and even if you could force the hand of particular companies, you'd still have to figure out a strategy for dealing with all the sovereign nations that, in effect, act as fossil-fuel companies. But the link for college students is even more obvious in this case. If their college's endowment portfolio has fossil-fuel stock, then their educations are being subsidized by investments that guarantee they won't have much of a planet on which to make use of their degree. (The same logic applies to the world's largest investors, pension funds, which are also theoretically interested in the future – that's when their members will "enjoy their retirement.") "Given the severity of the climate crisis, a comparable demand that our institutions dump stock from companies that are destroying the planet would not only be appropriate but effective," says Bob Massie, a former anti-apartheid activist who helped found the Investor Network on Climate Risk. "The message is simple: We have had enough. We must sever the ties with those who profit from climate change – now."
Movements rarely have predictable outcomes. But any campaign that weakens the fossil-fuel industry's political standing clearly increases the chances of retiring its special breaks. Consider President Obama's signal achievement in the climate fight, the large increase he won in mileage requirements for cars. Scientists, environmentalists and engineers had advocated such policies for decades, but until Detroit came under severe financial pressure, it was politically powerful enough to fend them off. If people come to understand the cold, mathematical truth – that the fossil-fuel industry is systematically undermining the planet's physical systems – it might weaken it enough to matter politically. Exxon and their ilk might drop their opposition to a fee-and-dividend solution; they might even decide to become true energy companies, this time for real.
Even if such a campaign is possible, however, we may have waited too long to start it. To make a real difference – to keep us under a temperature increase of two degrees – you'd need to change carbon pricing in Washington, and then use that victory to leverage similar shifts around the world. At this point, what happens in the U.S. is most important for how it will influence China and India, where emissions are growing fastest. (In early June, researchers concluded that China has probably under-reported its emissions by up to 20 percent.) The three numbers I've described are daunting – they may define an essentially impossible future. But at least they provide intellectual clarity about the greatest challenge humans have ever faced. We know how much we can burn, and we know who's planning to burn more. Climate change operates on a geological scale and time frame, but it's not an impersonal force of nature; the more carefully you do the math, the more thoroughly you realize that this is, at bottom, a moral issue; we have met the enemy and they is Shell.
Meanwhile the tide of numbers continues. The week after the Rio conference limped to its conclusion, Arctic sea ice hit the lowest level ever recorded for that date. Last month, on a single weekend, Tropical Storm Debby dumped more than 20 inches of rain on Florida – the earliest the season's fourth-named cyclone has ever arrived. At the same time, the largest fire in New Mexico history burned on, and the most destructive fire in Colorado's annals claimed 346 homes in Colorado Springs – breaking a record set the week before in Fort Collins. This month, scientists issued a new study concluding that global warming has dramatically increased the likelihood of severe heat and drought – days after a heat wave across the Plains and Midwest broke records that had stood since the Dust Bowl, threatening this year's harvest. You want a big number? In the course of this month, a quadrillion kernels of corn need to pollinate across the grain belt, something they can't do if temperatures remain off the charts. Just like us, our crops are adapted to the Holocene, the 11,000-year period of climatic stability we're now leaving... in the dust.
This story is from the August 2nd, 2012 issue of Rolling Stone. 

01 March 2013

Topology and Data

Listen to Robert Ghrist explain how topology helps make sense of large data sets:
http://www.ams.org/samplings/mathmoments/mm77-topology-data-podcast

For More Information: “Topology and Data”, Gunnar Carlson, Bulletin of the American Mathematical Society (Vol. 46, No. 2), April 2009.


Much of modern research—from genome sequencing to digital surveys of outer space—generates tremendous amounts of multi-dimensional data. Unfortunately, visualizing dimensions higher than three is not easy, which makes analyzing and understanding the data difficult. Topology, a branch of mathematics concerned with the properties of geometrical structures, helps make sense of large data sets by providing a way of classifying the shapes of these sets. It’s especially useful for locating groups of similar points called “clusters”, which can, for example, distinguish between distinct types of a given disease, each requiring its own treatment.
Topology (specifically algebraic topology) is also important in the operation of wireless sensor networks, which are used in applications as diverse as monitoring automobile traffic and controlling irrigation. Combined with numerical integration, results from algebraic topology provide the complete picture based on strictly local data. The advantage is that such sensor networks, maintained without GPS or other distance measures, are generally much cheaper to operate. So, in the case of irrigation, mathematical discoveries made almost a century before the advent of today’s technology save money while helping us use precious water wisely. In topological terms, just like the Möbius strip: What goes around, comes around.

24 February 2013

On data access and digital dissent


Jeremy Hammond on Aaron Swartz and the Criminalization of Digital Dissent.

A statement released, Feb 20th, by Jeremy Hammonds lawyer. This is Jeremy Hammond in his own words, from solitary confinement.
 Jeremy Hammond on Aaron Swartz and the Criminalization of Digital Dissent
 The tragic death of internet freedom fighter Aaron Swartz reveals the government’s flawed “cyber security strategy” as well as its systematic corruption involving computer crime investigations, intellectual property law, and government/corporate transparency. In a society supposedly based on principles of democracy and due process, Aaron’s efforts to liberate the internet, including free distribution of JSTOR academic essays, access to public court records on PACER, stopping the passage of SOPA/PIPA, and developing the Creative Commons, make him a hero, not a criminal. It is not the “crimes” Aaron may have committed that made him a target of federal prosecution, but his ideas – elaborated in his “Guerrilla Open Access Manifesto” – that the government has found so dangerous. The United States Attorney’s aggressive prosecution, riddled with abuse and misconduct, is what led to the death of this hero. This sad and angering chapter should serve as a wake up call for all of us to acknowledge the danger inherent in our criminal justice system.
Aaron’s case is part of the recent aggressive, politically-motivated expansion of computer crime law where hackers and activists are increasingly criminalized because of alleged “cyber-terrorist” threats. The United States Attorney for the Southern District of New York, Preet Bharara, whose office is prosecuting me and my co-defendants in the Lulzsec indictment, has used alarmist rhetoric such as the threat of an imminent “Pearl Harbor like cyber attack” to justify these prosecutions. At the same time the government routinely trains and deploys their own hackers to launch sophisticated cyber attacks against the infrastructure of foreign countries, such as the Stuxnet and Flame viruses, without public knowledge, oversight, declarations of war, or consent from international authorities. DARPA, US Cyber Command, the NSA, and numerous federally-contracted private corporations openly recruit hackers to develop defensive and offensive capabilities and build Orwellian digital surveillance networks, designed not to enhance national security but to advance U.S. imperialism. They even attend and speak at hacker conferences, such as DEFCON, offer to bribe hackerspaces for their research, and created the insulting “National Civic Hacker Day” – efforts which should be boycotted or confronted every step of the way.
Aaron is a hero because he refused to play along with the government’s agenda, instead he used his brilliance and passion to create a more transparent society. Through the free software movement, open publishing and file sharing, and development of cryptography and anonymity technology, digital activists have revealed the poverty of neo-liberalism and intellectual property. Aaron opposed reducing everything to a commodity to be bought or sold for a profit.
The rise in effectiveness of, and public support for, movements like Anonymous and Wikileaks has led to an expansion of computer crime investigations – most importantly enhancements to 18 U.S.C § 1030, the Computer Fraud and Abuse Act (CFAA). Over the years the CFAA has been amended five times and has gone through a number of important court rulings that have greatly expanded what the act covers concerning “accessing a protected computer without authorization.” It is now difficult to determine exactly what conduct would be considered legal. The definition of a “protected computer” has been incrementally expanded to include any government or corporate computer in or outside the U.S. “Authorization,” not explicitly defined by the CFAA, has also been expanded to be so ambiguous that any use of a website, network, or PC that is outside of the interest, agenda, or contractual obligations of a private or government entity could be criminalized. In Aaron’s case and others the government has defined violating a service’s Acceptable Use Policy (AUP), Terms of Service (TOS), or End-User License Agreement (EULA) as illegal. Every time you sign up for a service like Gmail, Hotmail, or Facebook and click the “I agree” button that follows a long contract that no one ever reads, you could be prosecuted under the CFAA if you violate any of the terms.
The sheer number of everyday computer users who could be considered criminals under these broad and ambiguous definitions enables the politically motivated prosecution of anyone who voices dissent. The CFAA should be found unconstitutional under the void-for-vagueness doctrine of the due process clause. Instead, Congress proposed bills last year which would double the statutory maximum sentences and introduce mandatory minimum sentences, similar to the excessive sentences imposed in drug cases which have been widely opposed by many federal and state judges.
The “Operation Payback” case in San Jose, California is another miscarriage of justice where 16 suspected Anonymous members (including a 16 year old boy) allegedly participated in a denial-of-service action against PayPal in protest of it’s financial blockade of Wikileaks. Denial-of-service does not “exceed authorized access,” as it is virtually indistinguishable from standard web requests. It is more akin to an electronic sit-in protest, overloading the website’s servers making it incapable of serving legitimate traffic, than a criminal act involving stolen private information or destruction of servers. PayPal’s website was only slow or unavailable for a matter of hours, yet these digital activists face prison time of more that 10 years, $250,000 in fines, and felony convictions because the government wants to criminalize this form of internet protest and send a warning to would be Wikileaks supporters.
Another recent case is that of Andrew “Weev” Auernheimer, who last November was convicted under the CFAA. Andrew discovered that AT&T was publishing customer names and email addresses on it’s public-facing website, without password protection, encryption, or firewalls. Instead of acknowledging their own mistake in violating customer privacy, AT&T sought prison time for Andrew. Andrew has defended his actions saying, “We have not only a right as Americans to analyze things that corporations publish and make publicly accessible but perhaps a moral obligation to tell people about it.”
I am currently facing multiple computer hacking conspiracy charges due to my alleged involvement with Anonymous, LulzSec, andAntiSec, groups which have targeted and exposed corruption in government institutions and corporations such as Stratfor, The Arizona Department of Public Safety, and HB Gary Federal. My potential sentence is dramatically increased because the Patriot Act expanded the CFAA’s definition of “loss.” This allowed Stratfor to claim over 5 million dollars in damages, including the exorbitant cost of hiring outside credit protection agencies and “infosec” corporations, purchasing new servers, 1.6 million dollars in “lost potential revenue” for the time their website was down, and even the cost of a 1.3 million dollar settlement for a class action lawsuit filed against them. Coupled with use of “sophisticated means” and “affecting critical infrastructure” sentence enhancements, if convicted at trial I am facing a sentence of 30-years-to-life.
Dirty trial tactics and lengthy sentences are not anomalies but are part of a fundamentally flawed and corrupt two-tiered system of “justice” which seeks to reap profits from the mass incarceration of millions, especially people of color and the impoverished. The use of informants who cooperate in exchange for lighter sentences is not just utilized in the repressive prosecutions of protest movements and manufactured “terrorist” Islamophobic witch-hunts, but also in most drug cases, where defendants face some of the harshest sentences in the world.
For Aaron Swartz, himself facing 13 felony CFAA charges, it is likely that it was this intense pressure from relentless and uncompromising prosecutors, who, while being aware of Aaron’s psychological fragility, continued to demand prison time, that led to his untimely death.
Due to widespread public outrage, there is talk of congressional investigations into the CFAA. But since the same Congress had proposed increased penalties not even one year ago, any efforts at reform are unlikely to be more than symbolic. What is needed is not reform but total transformation; not amendments but abolition. Aaron is a hero to me because he did not wait for those in power to realize his vision and change their game, he sought to change the game himself, and he did so without fear of being labeled a criminal and imprisoned by a backwards system of justice.
We the people demand free and equal access to information and technology. We demand transparency and accountability from governments and big corporations, and privacy for the masses from invasive surveillance networks.
The government will never be forgiven. Aaron Swartz will never be forgotten.

Event page for Courthouse Support Rally https://www.facebook.com/events/567165369962801/
Get involved with the Jeremy Hammond Support Network
FreeHammond.org
On Facebook https://www.facebook.com/supporthammond
Twitter https://twitter.com/Free_Hammond
The address to mail postcards to Jeremy is as follows:
Jeremy Hammond – #18729-424, Metropolitan Correctional Center, 150 Park Row , New York, New York, 10007

23 February 2013

Algos for big data

http://www.industryweek.com/emerging-technologies/algorithms-silent-game-changer-big-data


Algorithms: The Silent Game Changer in Big Data

By Radhika Subramanian, CEO of Emcien.

To keep pace with the Big Data explosion, we must take the human element out of the equation and let the computers take over.

Hype is synonymous with Big Data. But data is nothing new.
What’s new is that manufacturers are finally learning how to draw critical insights from that data for competitive advantage. The key to that, of course, is having the knowledge and understanding to spend wisely on knowledge discovery and getting the data to talk to us.
According to the results of a recent Gartner study, enterprise data is on pace to grow 650% over the next few years, with 80% of that data in unstructured form. This is more than any data scientist -- or person -- can manage.
People’s ability to process data hasn’t changed much over the years, but the technology we employ to help us process it changes constantly. Therein lays the promise of Big Data. Specifically, Big Data analytics that rely on algorithmsare our only hope of keeping up with the data explosion.

Algorithms and Big Data

Every time we have faced technology challenges that are computationally intensive, algorithms have come to our rescue, helped us to overcome the barriers and we have pushed the frontier of innovation. A few good examples are:
  • Operating systems are algorithms that enable computers to interact with humans. Without operating systems there would no PCs, laptops or smart phones.
  • An algorithm that defines how content is stored and transmitted enables the Internet, or the World Wide Web. These were world-changing algorithms and today we have more than 200 Billion web sites at our disposal.
  • Google’s ranked search is an algorithm that lets us quickly search for highly relevant content on the web site. 
And yes, there are many more. What they all share in common in the ability to break through barriers that limit our ability to reach across get to the next frontier of innovation. Big data poses a ton of such barriers -- and smart algorithms will help us to overcome these barrier.
The biggest problem with a traditional approach to data is that it’s typically an extension of old thinking and hence time-consuming because it may fail to consider the barriers facing us today -- the volume of data and the cost and time to sift it.
It’s time for an entirely new approach that addresses these growing needs.
This new approach demands a paradigm shift that focuses on the following:
• A fundamental change in the role played by analysts from data-miners to insight-evaluators.
• Fast and efficient methods that automatically convert data to insight for analysts to evaluate and operationalize.
• Continual improvement of these automation methods to keep up with the speed of data and critical need for timely insights.
The next wave of challenges is upon us. How can we scale these efforts to automate getting the gold from the data?
The data-scientist approach is labor intensive, time and cost prohibitive -- and most importantly cannot scale to the speed of data. The new approaches need to automate the process just like processes before them such as automated mining of gold and other precious metals from dirt. The analogy is so very appropriate.
Manufacturing companies are finally looking to harness the power of Big Databecause it drives enormous opportunity for business improvement; however, this is still just the first inning of the game, and I would caution against investing too heavily in approaches that rely in hardware and staffing increases.
If a method you’re considering is intrusive, watch out! It’s not sustainable.
Ask yourself how much training the new tool is going to require, and whether or not it demands hiring a lot of experts to manage. Do you really need a team of data scientists to surface insight from data? How smoothly will it fit into your current business to help drive intelligent decisions with insight from data. The best approaches are ones that are sustainable.
And let me leave you with one more question to ponder as you go about your busy day -- Why is it that we have so much data, yet less than 1% of data is analyzed? It is because we have been so consumed, due to our hoarding mentality, with saving all this data while thinking not enough of how to get to the gold in them thar hills!

Emcien CEO Radhika Subramanian is a seasoned entrepreneur with decades of experience helping organizations utilize the insight buried within their data. Tweet to @RadhikaAtEmcien & join her Big Data Apps LinkedIn group.


14 February 2013

Data Scientist: The Sexiest Job of the 21st Century

1 Jan 2015: Here is a link to a more recent article on data science: http://insidebigdata.com/2014/03/14/death-data-science-greatly-exagerated/

---

http://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/ar/pr

Data Scientist: The Sexiest Job of the 21st Century

by Thomas H. Davenport and D.J. Patil
When Jonathan Goldman arrived for work in June 2006 at LinkedIn, the business networking site, the place still felt like a start-up. The company had just under 8 million accounts, and the number was growing quickly as existing members invited their friends and colleagues to join. But users weren’t seeking out connections with the people who were already on the site at the rate executives had expected. Something was apparently missing in the social experience. As one LinkedIn manager put it, “It was like arriving at a conference reception and realizing you don’t know anyone. So you just stand in the corner sipping your drink—and you probably leave early.”
Goldman, a PhD in physics from Stanford, was intrigued by the linking he did see going on and by the richness of the user profiles. It all made for messy data and unwieldy analysis, but as he began exploring people’s connections, he started to see possibilities. He began forming theories, testing hunches, and finding patterns that allowed him to predict whose networks a given profile would land in. He could imagine that new features capitalizing on the heuristics he was developing might provide value to users. But LinkedIn’s engineering team, caught up in the challenges of scaling up the site, seemed uninterested. Some colleagues were openly dismissive of Goldman’s ideas. Why would users need LinkedIn to figure out their networks for them? The site already had an address book importer that could pull in all a member’s connections.
Luckily, Reid Hoffman, LinkedIn’s cofounder and CEO at the time (now its executive chairman), had faith in the power of analytics because of his experiences at PayPal, and he had granted Goldman a high degree of autonomy. For one thing, he had given Goldman a way to circumvent the traditional product release cycle by publishing small modules in the form of ads on the site’s most popular pages.
Through one such module, Goldman started to test what would happen if you presented users with names of people they hadn’t yet connected with but seemed likely to know—for example, people who had shared their tenures at schools and workplaces. He did this by ginning up a custom ad that displayed the three best new matches for each user based on the background entered in his or her LinkedIn profile. Within days it was obvious that something remarkable was taking place. The click-through rate on those ads was the highest ever seen. Goldman continued to refine how the suggestions were generated, incorporating networking ideas such as “triangle closing”—the notion that if you know Larry and Sue, there’s a good chance that Larry and Sue know each other. Goldman and his team also got the action required to respond to a suggestion down to one click.
It didn’t take long for LinkedIn’s top managers to recognize a good idea and make it a standard feature. That’s when things really took off. “People You May Know” ads achieved a click-through rate 30% higher than the rate obtained by other prompts to visit more pages on the site. They generated millions of new page views. Thanks to this one feature, LinkedIn’s growth trajectory shifted significantly upward.
A New Breed
Goldman is a good example of a new key player in organizations: the “data scientist.” It’s a high-ranking professional with the training and curiosity to make discoveries in the world of big data. The title has been around for only a few years. (It was coined in 2008 by one of us, D.J. Patil, and Jeff Hammerbacher, then the respective leads of data and analytics efforts at LinkedIn and Facebook.) But thousands of data scientists are already working at both start-ups and well-established companies. Their sudden appearance on the business scene reflects the fact that companies are now wrestling with information that comes in varieties and volumes never encountered before. If your organization stores multiple petabytes of data, if the information most critical to your business resides in forms other than rows and columns of numbers, or if answering your biggest question would involve a “mashup” of several analytical efforts, you’ve got a big data opportunity.
Much of the current enthusiasm for big data focuses on technologies that make taming it possible, including Hadoop (the most widely used framework for distributed file system processing) and related open-source tools, cloud computing, and data visualization. While those are important breakthroughs, at least as important are the people with the skill set (and the mind-set) to put them to good use. On this front, demand has raced ahead of supply. Indeed, the shortage of data scientists is becoming a serious constraint in some sectors. Greylock Partners, an early-stage venture firm that has backed companies such as Facebook, LinkedIn, Palo Alto Networks, and Workday, is worried enough about the tight labor pool that it has built its own specialized recruiting team to channel talent to businesses in its portfolio. “Once they have data,” says Dan Portillo, who leads that team, “they really need people who can manage it and find insights in it.”
Who Are These People?
If capitalizing on big data depends on hiring scarce data scientists, then the challenge for managers is to learn how to identify that talent, attract it to an enterprise, and make it productive. None of those tasks is as straightforward as it is with other, established organizational roles. Start with the fact that there are no university programs offering degrees in data science. There is also little consensus on where the role fits in an organization, how data scientists can add the most value, and how their performance should be measured.
The first step in filling the need for data scientists, therefore, is to understand what they do in businesses. Then ask, What skills do they need? And what fields are those skills most readily found in?
More than anything, what data scientists do is make discoveries while swimming in data. It’s their preferred method of navigating the world around them. At ease in the digital realm, they are able to bring structure to large quantities of formless data and make analysis possible. They identify rich data sources, join them with other, potentially incomplete data sources, and clean the resulting set. In a competitive landscape where challenges keep changing and data never stop flowing, data scientists help decision makers shift from ad hoc analysis to an ongoing conversation with data.
Data scientists realize that they face technical limitations, but they don’t allow that to bog down their search for novel solutions. As they make discoveries, they communicate what they’ve learned and suggest its implications for new business directions. Often they are creative in displaying information visually and making the patterns they find clear and compelling. They advise executives and product managers on the implications of the data for products, processes, and decisions.
Given the nascent state of their trade, it often falls to data scientists to fashion their own tools and even conduct academic-style research. Yahoo, one of the firms that employed a group of data scientists early on, was instrumental in developing Hadoop. Facebook’s data team created the language Hive for programming Hadoop projects. Many other data scientists, especially at data-driven companies such as Google, Amazon, Microsoft, Walmart, eBay, LinkedIn, and Twitter, have added to and refined the tool kit.
What kind of person does all this? What abilities make a data scientist successful? Think of him or her as a hybrid of data hacker, analyst, communicator, and trusted adviser. The combination is extremely powerful—and rare.
Data scientists’ most basic, universal skill is the ability to write code. This may be less true in five years’ time, when many more people will have the title “data scientist” on their business cards. More enduring will be the need for data scientists to communicate in language that all their stakeholders understand—and to demonstrate the special skills involved in storytelling with data, whether verbally, visually, or—ideally—both.
But we would say the dominant trait among data scientists is an intense curiosity—a desire to go beneath the surface of a problem, find the questions at its heart, and distill them into a very clear set of hypotheses that can be tested. This often entails the associative thinking that characterizes the most creative scientists in any field. For example, we know of a data scientist studying a fraud problem who realized that it was analogous to a type of DNA sequencing problem. By bringing together those disparate worlds, he and his team were able to craft a solution that dramatically reduced fraud losses.
Perhaps it’s becoming clear why the word “scientist” fits this emerging role. Experimental physicists, for example, also have to design equipment, gather data, conduct multiple experiments, and communicate their results. Thus, companies looking for people who can work with complex data have had good luck recruiting among those with educational and work backgrounds in the physical or social sciences. Some of the best and brightest data scientists are PhDs in esoteric fields like ecology and systems biology. George Roumeliotis, the head of a data science team at Intuit in Silicon Valley, holds a doctorate in astrophysics. A little less surprisingly, many of the data scientists working in business today were formally trained in computer science, math, or economics. They can emerge from any field that has a strong data and computational focus.
It’s important to keep that image of the scientist in mind—because the word “data” might easily send a search for talent down the wrong path. As Portillo told us, “The traditional backgrounds of people you saw 10 to 15 years ago just don’t cut it these days.” A quantitative analyst can be great at analyzing data but not at subduing a mass of unstructured data and getting it into a form in which it can be analyzed. A data management expert might be great at generating and organizing data in structured form but not at turning unstructured data into structured data—and also not at actually analyzing the data. And while people without strong social skills might thrive in traditional data professions, data scientists must have such skills to be effective.
Roumeliotis was clear with us that he doesn’t hire on the basis of statistical or analytical capabilities. He begins his search for data scientists by asking candidates if they can develop prototypes in a mainstream programming language such as Java. Roumeliotis seeks both a skill set—a solid foundation in math, statistics, probability, and computer science—and certain habits of mind. He wants people with a feel for business issues and empathy for customers. Then, he says, he builds on all that with on-the-job training and an occasional course in a particular technology.
Several universities are planning to launch data science programs, and existing programs in analytics, such as the Master of Science in Analytics program at North Carolina State, are busy adding big data exercises and coursework. Some companies are also trying to develop their own data scientists. After acquiring the big data firm Greenplum, EMC decided that the availability of data scientists would be a gating factor in its own—and customers’—exploitation of big data. So its Education Services division launched a data science and big data analytics training and certification program. EMC makes the program available to both employees and customers, and some of its graduates are already working on internal big data initiatives.
As educational offerings proliferate, the pipeline of talent should expand. Vendors of big data technologies are also working to make them easier to use. In the meantime one data scientist has come up with a creative approach to closing the gap. The Insight Data Science Fellows Program, a postdoctoral fellowship designed by Jake Klamka (a high-energy physicist by training), takes scientists from academia and in six weeks prepares them to succeed as data scientists. The program combines mentoring by data experts from local companies (such as Facebook, Twitter, Google, and LinkedIn) with exposure to actual big data challenges. Originally aiming for 10 fellows, Klamka wound up accepting 30, from an applicant pool numbering more than 200. More organizations are now lining up to participate. “The demand from companies has been phenomenal,” Klamka told us. “They just can’t get this kind of high-quality talent.”
Why Would a Data Scientist Want to Work Here?
Even as the ranks of data scientists swell, competition for top talent will remain fierce. Expect candidates to size up employment opportunities on the basis of how interesting the big data challenges are. As one of them commented, “If we wanted to work with structured data, we’d be on Wall Street.” Given that today’s most qualified prospects come from nonbusiness backgrounds, hiring managers may need to figure out how to paint an exciting picture of the potential for breakthroughs that their problems offer.
Pay will of course be a factor. A good data scientist will have many doors open to him or her, and salaries will be bid upward. Several data scientists working at start-ups commented that they’d demanded and got large stock option packages. Even for someone accepting a position for other reasons, compensation signals a level of respect and the value the role is expected to add to the business. But our informal survey of the priorities of data scientists revealed something more fundamentally important. They want to be “on the bridge.” The reference is to the 1960s television show Star Trek, in which the starship captain James Kirk relies heavily on data supplied by Mr. Spock. Data scientists want to be in the thick of a developing situation, with real-time awareness of the evolving set of choices it presents.
Considering the difficulty of finding and keeping data scientists, one would think that a good strategy would involve hiring them as consultants. Most consulting firms have yet to assemble many of them. Even the largest firms, such as Accenture, Deloitte, and IBM Global Services, are in the early stages of leading big data projects for their clients. The skills of the data scientists they do have on staff are mainly being applied to more-conventional quantitative analysis problems. Offshore analytics services firms, such as Mu Sigma, might be the ones to make the first major inroads with data scientists.
But the data scientists we’ve spoken with say they want to build things, not just give advice to a decision maker. One described being a consultant as “the dead zone—all you get to do is tell someone else what the analyses say they should do.” By creating solutions that work, they can have more impact and leave their marks as pioneers of their profession.
Care and Feeding
Data scientists don’t do well on a short leash. They should have the freedom to experiment and explore possibilities. That said, they need close relationships with the rest of the business. The most important ties for them to forge are with executives in charge of products and services rather than with people overseeing business functions. As the story of Jonathan Goldman illustrates, their greatest opportunity to add value is not in creating reports or presentations for senior executives but in innovating with customer-facing products and processes.
LinkedIn isn’t the only company to use data scientists to generate ideas for products, features, and value-adding services. At Intuit data scientists are asked to develop insights for small-business customers and consumers and report to a new senior vice president of big data, social design, and marketing. GE is already using data science to optimize the service contracts and maintenance intervals for industrial products. Google, of course, uses data scientists to refine its core search and ad-serving algorithms. Zynga uses data scientists to optimize the game experience for both long-term engagement and revenue. Netflix created the well-known Netflix Prize, given to the data science team that developed the best way to improve the company’s movie recommendation system. The test-preparation firm Kaplan uses its data scientists to uncover effective learning strategies.
There is, however, a potential downside to having people with sophisticated skills in a fast-evolving field spend their time among general management colleagues. They’ll have less interaction with similar specialists, which they need to keep their skills sharp and their tool kit state-of-the-art. Data scientists have to connect with communities of practice, either within large firms or externally. New conferences and informal associations are springing up to support collaboration and technology sharing, and companies should encourage scientists to become involved in them with the understanding that “more water in the harbor floats all boats.”
Data scientists tend to be more motivated, too, when more is expected of them. The challenges of accessing and structuring big data sometimes leave little time or energy for sophisticated analytics involving prediction or optimization. Yet if executives make it clear that simple reports are not enough, data scientists will devote more effort to advanced analytics. Big data shouldn’t equal “small math.”
The Hot Job of the Decade
Hal Varian, the chief economist at Google, is known to have said, “The sexy job in the next 10 years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s?”
If “sexy” means having rare qualities that are much in demand, data scientists are already there. They are difficult and expensive to hire and, given the very competitive market for their services, difficult to retain. There simply aren’t a lot of people with their combination of scientific background and computational and analytical skills.
Data scientists today are akin to Wall Street “quants” of the 1980s and 1990s. In those days people with backgrounds in physics and math streamed to investment banks and hedge funds, where they could devise entirely new algorithms and data strategies. Then a variety of universities developed master’s programs in financial engineering, which churned out a second generation of talent that was more accessible to mainstream firms. The pattern was repeated later in the 1990s with search engineers, whose rarefied skills soon came to be taught in computer science programs.
One question raised by this is whether some firms would be wise to wait until that second generation of data scientists emerges, and the candidates are more numerous, less expensive, and easier to vet and assimilate in a business setting. Why not leave the trouble of hunting down and domesticating exotic talent to the big data start-ups and to firms like GE and Walmart, whose aggressive strategies require them to be at the forefront?
The problem with that reasoning is that the advance of big data shows no signs of slowing. If companies sit out this trend’s early days for lack of talent, they risk falling behind as competitors and channel partners gain nearly unassailable advantages. Think of big data as an epic wave gathering now, starting to crest. If you want to catch it, you need people who can surf.
Thomas H. Davenport is a visiting professor at Harvard Business School, a senior adviser to Deloitte Analytics, and a coauthor of Judgment Calls (Harvard Business Review Press, 2012). D.J. Patil is the data scientist in residence at Greylock Partners, was formerly the head of data products at LinkedIn, and is the author of Data Jujitsu: The Art of Turning Data into Product (O’Reilly Media, 2012).

22 January 2013

School code breaking

http://www.bbc.co.uk/news/technology-21121058


Schools' codebreaking competition winners announced

Enigma machine 
Bletchley Park codebreakers during World War II used Enigma machines to crack German messages


A team from City of London School has been named the winner of a national codebreaking competition.
In total 6,268 pupils from 725 British schools took part in the National Cypher Challenge at the end of 2012.
Previous years have seen 200 teams take part but this year 1,600 teams signed up to decode a series of cryptic codes released online.
The event was organised by Southampton University with support from GCHQ and commercial partners.
The competition was only for UK schools but teams from Tokyo, Bangkok, Florida and Honolulu also applied to take part.
It ran over a period of two months, with codes of increasing difficulty being issued periodically on the internet for school teams to crack, explained Prof Graham Niblo, organiser of the contest and head of mathematics at Southampton University.
While 1,600 teams signed up for the challenge only 30 managed to complete every level, he said.
Cypher secrets "We started with a Caesar cypher, which is the most basic cypher that everybody learns at school. We ended with a Trifid cypher - the most difficult one this year," he told the BBC.
A Caesar cypher is decoded by substituting the letters in a code with others further down the alphabet. For example, an A becomes a C, a B becomes a D and so on.

A Trifid cypher is a complex combination of both an advanced version of the Caesar cypher and anagrams, where words are jumbled.
"It is the longest it has ever taken for the winners to decode any of our cyphers. This year they had to think quite hard about it," said Prof Niblo.
He added that it was very difficult for contestants to cheat.
"We did catch a couple of people trying to get help online. We tracked them down and warned them off," he said.
"The last cypher was not a standard cypher, and we modified it so they couldn't use any standard decryption software."
The winning team - Samson Danziger, Daniel Hu, Anthony Landau and Charlie Hu - cracked the Trifid code in 44 hours and 20 minutes. They will receive a £1,000 prize from GCHQ.
In second place was Andrew Carlotti from Sir Roger Manwood's School in Kent and a team from King Edward VII School in Sheffield came third.
The top 50 runners-up will also receive Raspberry Pi computers.
Competition history The National Cypher Challenge is now in its 11th year.
Foreign Secretary William Hague MP launched the competition at codebreaking museum Bletchley Park in October 2012, as part of three GCHQ initiatives aimed at attracting young people into maths and computer science.
"Teachers tell me the children find it very gripping and engaging," said Prof Niblo.
"A number of them worked on it at home or went along to maths clubs they wouldn't usually attend."