The BT Traffic and weather livecentre aims to gather data from public authorities, social media and editorial systems to provide insights and real time information in a mobile friendly web application. A custom built framework allows us to easily connect to new public or private data sources, fetch data, parse and transform it to our needs and apply algorithms to prioritize which data to show and how. Data widgets appear automatically when there i.e. is a weather alert in place, or displays a map when there is a power outage somewhere in our region.
Using colors, labels and symbols we try to attract attention to important data. The livecentre is alive even when there is no editorial content being manually added, using rules and algorithms to prioritize and label incoming public data.
The livecentre app has also been used for other data driven live coverage sites, like the US election live coverage, and is constantly being updated to retrieve and display data from new public sources.
Our region is heavily affected by bad weather and traffic often suffers from that. We wanted to give our readers a one-stop solution where we automatically prioritize and show relevant content related to traffic and weather data. There are multiple sites giving some of this information, but none that gathers it in one place. And none that automatically prioritizes what to show, how and when based on real time events.
We wanted to create both a system for presenting data in a user friendly, simple and mobile friendly way, and a flexible system to fetch data from multiple public APIs and data sources.
By adding such a system we could provide a faster and more updated live experience for our readers and at the same time enabling the journalists to focus more on the big stories and less on short updates from the traffic and weather.
The system fetches data from multiple public sources and private sources. We have set up a flexible harvesting system that cater for most common types of APIs and data formats. The system has configurable scheduling for fetching data, and applies rules and algorithms for data before it is stored for presentation.
Data sources includes traffic messages, road conditions, weather data, webcam data and travel time data via geographical APIs of the Public Roads Administration, weather observation data, weather alerts, and extreme weather alerts from the Meteorological Institute, air quality measurements from Norwegian Institute for Air Research, power outage data from local power grid operator, feeds from internal live blog tools, feeds from internal webcam systems, Twitter and Google maps.
Two data journalists/developers working with the project as a side/part time project for around 6 months, building the harvesting tool and the livecentre presentation app. This was done parallel to other data journalism work in our small team. One project lead part time (our team leader)
The project has given the citizens of our region a unique site to get updated on weather and traffic information that impacts on their daily life. The site has attracted a lot of web traffic and is daily one of our top three visited pages. User engagement is high, and the project has received attention from managers of public data for its use of open data.
Internally the project has had a big impact on our newsroom. It has enabled us to have an always up-to-date live coverage using automated data processing. This has made it possible to redirect journalist resources to interacting with users, gathering stories and images not available as data. The system also has an alert module that can alert the newsroom on weather and traffic related events based on the real time data feed/parsing.
Getting access to public data in machine readable formats is always a challenge. Our main obstacle was getting traffic messages in a format that was useable to us. First we tried to parse the complex EU standard DATEX format for traffic information, but after debating with the roads administration we got access to other and better data. Another obstacle was inconsistent data and categories, making it difficult to create rules and triggers. We spent a lot of time inspecting and analyzing data over time to create the best possible rules/algorithms for displaying relevant info and hide “noise”.
We also had a lot of issues with creating a fast and responsive site optimized for mobile phones and at the same time presenting a lot of data in a way that made it relevant for the user. We had to invest a lot of time into learning new web frameworks and methods for presenting data. Getting the end site to work great on slow mobile connections took time, but was an important issue.