Inside IODA’s Pilot Program for Monitoring Global Power Outage Data
Digital media consultant Tara Kelly sits down with Dr Zachary Bischof to learn more about how IODA is building a power outage dataset for Internet measurement researchers, journalists, advocates and policymakers.
I hear the IODA team has been working on building a global dataset of power outages. Tell us about the pilot project.
IODA’s data helps researchers, journalists, Internet freedom advocates and policymakers understand when and where Internet connectivity is disrupted, though identifying the exact cause is often complex. By integrating power outage data with connectivity information, the IODA team aims to speed up investigations and clarify whether power outages are responsible for disruptions. The new pilot involves identifying sources of publicly available power outage data. The team writes scripts to scrape that data from the web, making it publicly available. Making this data available through the IODA dashboard or as downloadable files will enhance clarity and efficiency for users analyzing connectivity issues.
Can you share what motivated this work?
The pilot project was inspired by research that connects weather events to disruptions in power networks. The IODA team collaborated with Georgia Tech’s GROWER Lab, which stands for Grid Resilience Outage Weather Emergency Response. The GROWER Lab provides power outage data that is focused on the United States and is relatively easier to collect and analyze compared to data from many other countries. Given IODA’s global focus, the team initiated this pilot to fill the gap and develop power outage datasets that extend beyond the United States.
This sounds like really important but difficult work. How do you go about collecting this data?
We selected a dozen countries and scraped power outage data from their power companies or government websites, despite varying formats and publication schedules. First, we developed scraping scripts, and then we collaborated with Georgia Tech engineering students to design the infrastructure for managing, storing, and publishing this unstructured data. This infrastructure is designed to handle data variability and handles fault management.
What have been your biggest challenges with this pilot project?
The biggest challenge is managing messy, inconsistent data from various sources that differ in format, detail, and publication timing, complicated further by power industry jargon, local language and knowledge differences. Data relating to geography/GIS is another challenge. For instance, it can be quite tricky to integrate different datasets that provide data at various geographic granularities and/or specificities by country and region. Additionally, ensuring that power data provider websites remain accessible during outages is crucial for real-time data, though historical data can still be provided if they go offline. These factors make reliable data collection and analysis difficult but essential for accurate outage monitoring.
Can you tell me more about the team working on this?
IODA is a project that is led by Georgia Tech’s Internet Intelligence Lab run by Internet measurement experts who are professors or researchers. So far, four Online Masters In Computer Science students have worked with us on the project for student class credit.
How do you plan to make this data accessible?
Several options exist for making this data accessible. We could provide the following: a) make the raw power data format available, which are completely unedited; b) provide a unified format that preserves much of the details of the raw files c) provide the simplified version which is just time ranges and locations of potential power outages.
We could also embed a simplified version of this data into the IODA dashboard and annotate the graph to show where and when power outages have occurred, alongside disruptions to Internet connectivity. We plan to make both publicly accessible to the research community.
How many regions or countries are covered in this first pilot?
A dozen countries are part of this first pilot. At the start of the project, we began investigating the power data of countries that experience more Internet shutdowns. However, we also decided to choose countries that are more stable and experience fewer Internet disruptions because their data was more accessible. It wasn’t possible to include some countries on our original wish list due to a lack of publicly available power outage data.
What is the plan moving forward?
Our near-term goal is to operationalize the pilot project, while the long-term plan is to publish the data and integrate it into the IODA platform. We aim to centralize all scripts for data collection and set up alerts for any failures, building infrastructure to handle issues related to data fetching, parsing, and publishing. Eventually, we want to expand to more countries and make the project community-led, with local contributors helping with scripting, translation, and geolocation.