Microsoft Azure helps researchers predict traffic jams

More than half of the world’s population now lives in cities and suburbs, and as just about any of these billions of people can tell you, urban traffic can be a nightmare. Cars stack up bumper-to-bumper, clogging our highways, jangling our nerves, taxing our patience, polluting our air, and taking a toll on our productivity. In short, traffic jams impair on our emotional, physical, and economic wellbeing.

A study by the Brazilian National Association of Public Transport showed that the country’s traffic exacted an economic toll of about US$7.2 million in 1998. Unfortunately, it’s only getting worse; there are now about three times as many vehicles in Brazil, making traffic exponentially worse, according to Fernando de Oliveira Pessoa, a traffic expert in Belo Horizonte, Brazil’s sixth-largest city.

Microsoft Research has joined forces with the Federal University of Minas Gerais, home to one of Brazil’s foremost computer science programs, to tackle the seemingly intractable problem of traffic jams. The immediate objective of this research is to predict traffic conditions over the next 15 minutes to an hour, so that drivers can be forewarned of likely traffic snarls.

The aptly named Traffic Prediction Project plans to combine all available traffic data—including both historic and current information gleaned from transportation departments, Bing traffic maps, road cameras and sensors, and the social networks of the drivers themselves—to create a solution that gets motorists from point A to point B with minimal stop-and-go. The use of historic data and information from social networks are both unique aspects of the project.

By using algorithms to process all these data, the project team intends to predict traffic jams accurately so that drivers can make smart, real-time choices, like taking an alternative route, using public transit, or maybe even just postponing a trip. The predictions should also be invaluable to traffic planners, especially when they are working to accommodate traffic from special events and when planning for future transportation needs.

Achieving reliable predictions will involve processing terabytes of data, which is why the researchers are using Microsoft Azure as the platform for the service. The exceptional scalability, immense storage capacity, and prodigious computational power of Microsoft Azure makes it the perfect resource for this data-intensive project. And because Microsoft Azure is cloud-based, running the Traffic Prediction service on Azure makes it accessible to all users, in real time, all of the time.

To date, the researchers have tested their prediction model in some of the world’s most traffic-challenged cities: New York, Los Angeles, London, and Chicago. The model achieved a prediction accuracy of 80 percent, and that was based on using only traffic-flow data. The researchers expect the accuracy to increase to 90 percent when traffic incidents and data from social networks are folded in.

So the next time your highway resembles a long, thin parking lot, you might calm yourself by contemplating how Microsoft Azure and the Traffic Prediction Project might help you avoid such tie-ups in the future.

—Juliana Salles, Senior Program Manager, Microsoft Research

Preventing flood disasters with Cortana Intelligence Suite

By Kristin Tolle as written on blogs.msdn.microsoft.com

On October 31, 2013, the city of Austin, Texas, faced a destructive flood. At the time, I was visiting David Maidment, Chaired Professor of the Civil Engineering Center for Research in Water Resources on site at the University of Texas at Austin. The day before the flood, we had been discussing research and analytics around the long-standing drought conditions across western Texas. Overnight, a flash flood wreaked havoc on the Austin area, largely due to the failure of a stream gauge on Onion Creek, which prevented local emergency response officials from being properly informed about the situation.

On the morning of October 30, the stream gauge monitoring Onion Creek’s was operational and reporting that the stream level was rising to dangerous levels. First responders were monitoring the gauge so that they would be prepared for sending out support crews. However, around 5:00 a.m., the stream level reported by the gauge dropped to zero—which is not uncommon in the southern United States, where washes and stream levels can quickly drop to normal levels once the initial precipitation pattern passes. With the disaster appearing to have been averted, emergency responder turned their attention elsewhere. In actuality, the gauge had failed, the stream overran its banks, and more than 500 homes flooded and five people died.

Since the Onion Creek event, every year and often several times each year, Texas and nearby Oklahoma have experienced several floods, some of which have been more deadly than the 2013 event. In May 2015 a flood in this region claimed 48 lives, including two first responders, Deputy Jessica Hollis of the Austin Police Department and Captain Jason Farley of the Claremore, Oklahoma, Fire Department.

Researchers from the University of Texas at Austin (UT Austin) are collaborating with other researchers, federal agencies, commercial partners, and first responders to create the National Flood Interoperability Experiment (NFIE). The goals of the NFIE include standardizing data, demonstrating a scalable solution, and helping to close the gap between national flood forecasting and local emergency response. The objective is to create a system that interoperates between different publically available data sources to model floods, based on predictions.

Systems for each of the 13 water regions in the United States were developed, two of them at Microsoft Research by my visiting researcher, Marcello Somos (New England region), and intern Fernando Salas (Gulf region), both from the UT Austin. After Marcello and Fernando returned to Austin, they collaborated with other institutions to create a national flood map for the entire nation. This interoperated data product was used by NOAA to run a summer institute at the National Water Center in Tuscaloosa, Alabama, with 38 top hydrology and meteorology graduate students from around the world.

My colleague Prashant Dhingra and I presented Microsoft Azure and the recently announced big data advanced analytics and intelligence platform, Cortana Intelligence Suite, to the students at the annual National Water Center Summer Institute. Several enterprising attendees created interesting analytics projects. Tim Petty, a PhD candidate at the University of Alaska, Fairbanks, wanted to address “the Onion Creek Problem,” and what we can do to estimate flood levels when stream gauges fail. And so project SHEM began.

Streamflow hydrology estimate using machine learning (SHEM) is a Cortana Intelligence Suite experiment that creates a predictive model that can act as a proxy streamflow data when a stream gauge fails. And due to the machine learning capabilities, it can even make estimates of stream levels where there is no actual stream gauge present.

SHEM differs from most existing models as it does not rely on distances between stream gauges and their location attributes, but is based solely on machine learning to process from historical patterns of discharge and interpret large volumes of complex hydrology data. This “training” prepares SHEM to predict streamflow information for a given location and time as it is impacted by multivariate attributes (for example, type of stream, type of reservoir, amount of precipitation, and surface and subsurface flow conditions).

Using Cortana Intelligence Suite (CIS), our joint research team was able to ingest, clean, refine, and format the historical US Geological Survey stream gauge data. We leveraged the Boosted Decision Tree Regression module which is one of many built-in machine learning algorithms. We also used built-in modules for data cleaning and transformation as well as modules for model scoring and evaluation. Wherever custom functionality is needed, you can add R or python modules directly to the workflow. And this is the advantage of Azure Machine Learning—that you can test multiple built-in or hand-coded algorithms and workflows in order to build an optimal solution rerunning and testing with reproducible results.

As with NFIE a year ago, SHEM is in the early stages of development and expanding it to more and more states is ongoing work. But the results bode well. All indications are that Cortana Intelligence Suite can use NFIE data and analysis products to effectively provide a reasonable estimate when a gauge is not present. Another byproduct of this experiment is that we can evaluate where there is the greatest variance in accuracy, which can, in turn, give us a good idea where it might be best to install new stream gauges.

And that should help all of us sleep a lot better—even in Austin.