What if San Francisco city government could spot a problem before it was a big problem? Without doubt, that would make the city government smarter, faster and maybe even less expensive.

But there hasn’t been a government crystal ball that could spot problems in advance.

Until now.

Gov 2.0 and the Twitter Predictor

Over the past year, a number of academics have been taking a look at Twitter and other social networks and asking if this fire hose of data could be digested to predict future behavior, such as crime patterns, or to predict future events, like outbreaks of communicable diseases.

The answer from experts – a cautious “yes.”

Already, Twitter is being used to successfully predict movie grosses and even future stock trends. Now some futurists are even suggesting that using the Web 2.0 data stream to predict future criminal behavior can’t be far behind.

For many of us this raises the specter of Big Brother. But if an analysis of the social media data stream could predict the outbreak of gang violence, or a deadly flu, wouldn’t you want government to have the ability to access that data and act on it to save lives?

That is the debate we are about to have.

From CompStat to Twitter: Gov. 2.0 Leaps Forward

Already, some cities are experimenting with data driven algorithms to predict crime trends. This is a technology built on the 1980’s CompStat model, which used early mapping software to spot crime patterns and then deployed police and other resources to reduce those crime hotspots.

While tremendously powerful, and widely credited with helping cities where CompStat was employed to dramatically lower crime, the data was essentially looking backward. It was a pretty good guess that crime would occur where crimes had already occurred. But the CompStat model was built to spot trends – not to predict them.

In the last year the Los Angeles Police Department won a $3 million federal grant to use internal crime data to see if future crime trends could be predicted. This “predictive policing” model looks for “micro trends” in police reports and other internal data and seeks to spot future trends.

This is basically CompStat on steroids – taking the existing model and applying modern computing power and a predictive algorithm.

A Twitter Crystal Ball

In the last two years social scientists and marketers have been looking at the nearly daily Everest of data being generated by social media and asking if it can be used to predict trends in real time.

Google was an early pioneer of using search data for social purposes. Their Google Flu Tracker spotted word searches that predicted flu outbreaks and mapped that data for use by concerned residents and government health agencies.

Now a new startup, called Sickweather is doing the same thing ¬ – but using the Twitter (and Facebook) stream to spot where colds, flues and other maladies large and small are clustered. Presumably residents, or health agencies, could monitor a product like Sickweather for outbreaks. For example, if they saw that eating sprouts was creating health problems – agencies could investigate if E. Coli was lurking in the veggies and regular residents could switch to romaine until the “Sickweather” had passed.

This is just one of the many uses of the real time search data being provided by Twitter. While Google used its propriety search database to create Google Flu Tracker, the Twitter stream is searchable by any company or agency looking to spot trends.

An Automatic 311

While the city’s pioneer 311 system has helped make it easier for residents to lodge complaints and to gather information, it is still far, far from its original promise. The idea of 311 was that it would be a management tool – allowing policy makers and administrators to gather data, spot trends and act on them.

That, largely, has not happened. But what if we could have Sickweather for all kinds of trends?

LandLordWeather.com could predict if tenants were having problems with landlords. GiantPothole.com could see where the worst potholes were clustered. You get the idea.

Instead of waiting for problems to happen – or waiting for residents to complain – an effective use of Government 2.0 would be monitor the Twitterverse in real time to see what residents want or need from San Francisco city government.

An Answer in Algorithms

Here at Reset’s virtual HQ, Pandora is playing right now. It has accurately predicted that Warren Zevon was what we wanted to hear next. A small little service provided by a powerful algorithm crunching data.

As Gov 2.0 advances, the next step is to see if we can get troubled city governments like San Francisco to change their tune by using predictive tools and the mountains of publicly available data.