California Data Collaborative Fights Most Severe Drought in History with Enigma


New Data Infrastructure Allows CaDC to Lay Foundation to Improve Water Efficiency, Reliability and Conservation

In the face of a historic drought and future uncertainty about water scarcity, the California Data Collaborative (CaDC) is pioneering a new data infrastructure to centralize water use data across California’s more than 1,000 water providers. With the support of Enigma, the organization helps California’s water district managers leverage data as an enterprise asset in order to improve water reliability, efficiency and conservation.

Organization: California Data Collaborative
Industry: Coalition of California Water Utilities
Key Challenge: Fragmented water usage data across hundreds of utilities
Key Strategy: Create a bottom-up coalition, leveraging innovative technology, to deliver on the Governor’s call for water efficiency
Scale: Integrate data from water utilities serving 21 Million Californians
Solution: Enigma ParseKit
Value: Automate water data integration to routinize mission critical analytics, in real time

Founded in January 2016, The California Data Collaborative is a water manager-led coalition that brings together water agencies throughout California to share data and help ensure water reliability. By developing tools and strategies to collect and analyze data, CaDC shares insights to help water managers improve efficiencies and promote long-term sustainable solutions across California’s natural resources.

Historically, California’s water usage data has been fragmented and difficult to extract from legacy databases. Since every utility uses its own methods to classify customers and data varies based on the characteristics of each unique district, California lacked consistent benchmarks. As a result, esoteric averages across different populations made it difficult to tell an accurate story of how much water was being used across the state.

For example, the Governor’s orders in 2014 called for decreased water use by 25% in urban areas. However, this one-size-fits-all mandate ignored the nuance that some districts had been doing the right thing and already had become incredibly water efficient over the past decade; being asked to conserve 25% more was unrealistic.

CaDC needed a better way to capture accurate data in order to get a realistic understanding of statewide water efficiency. “This wasn’t a top down mandate, but a bottom-up approach to meet the needs of all the different districts,” said Patrick Atwater, CaDC Project Manager.

“Some civic data initiatives are more about how shiny the dashboard is, but we remained laser focused on how data can deliver a basic service better.”

CaDC set out to establish a more thoughtful approach to benchmark water efficiency through improved data management. By integrating and analyzing data across different districts, CaDC could help water managers understand and manage statewide conservation policy; refine and measure demand management strategies; and drive revenue stability.

“The districts we work with lead the way in being smart about water efficiency,” said Atwater. “CaDC analytics provide a path for utilities to manage demand as rigorously as new supply projects, and represents the next frontier of excellence in water management.”

As part of the improved data management program for the State of California, CaDC relies on Enigma’s ParseKit, a data operations platform built to integrate and deliver data at scale. ParseKit allows CaDC’s data pipelines to be fast, reliable, and collaborative. Instead of fragmented across 411 different databases, water data from the major urban suppliers – who provide services for nearly 21 million Californians – is now consolidated on a centralized platform.

CaDC leverages that data to show the implications of the Governor’s new formula to benchmark statewide water efficiency. With operational data at their fingertips and a single benchmark for statewide water efficiency, water managers have more clarity into how water use varies by demographic, climate, neighborhood, income, etc., and can perform meaningful analytics to drive better decisions. Whether a water manager trying to understand impact of rates on revenue, or an everyday citizen reading about water conservation in the LA Times, everyone will be working from the same set of numbers.

While CaDC is largely relying on a manual approach for ingesting data, as they look to grow and bring more districts on board, the organization will need to automate the integration of meter data. “It’s a question of scale. Enigma gives us power to say we can do this for any district in California,” said Atwater.

“We’ve already seen the power of deploying ParseKit parser in automating data ingestion workflows from weeks of staff time down to mere minutes of cloud runtime. I have 100% confidence in Enigma’s platform and the repeatability of their results ingesting and integrating the State of California’s water data.”

Our ability to answer basic questions – such as typical water use for a multi-family customer in a certain service era and how it’s trending over time – is orders of magnitude faster. We can find answers in hours instead of weeks. By answering basic questions faster, staff can ask deeper questions and engage at a more rigorous level, thinking more strategically about how to achieve water efficiency and revenue stability.

In one example, augmented data for recycled water demand forecasting helped avoid $5M in capital costs. In addition, centralizing and standardizing water use data enables CaDC to streamline how data is shared with academic research projects that water managers elect to participate in. CaDC now offers an integrated approach to world-leading academic research on conservation marketing, the price elasticity of water demand and other key topics through CaDC’s global network of research partnerships. This integrated research agenda helps water managers learn what’s actually working to achieve water efficiency and better plan for future droughts.

“The biggest benefit so far has been the planning,” said Atwater. “With climate change and population growth, California faces substantial future water supply uncertainty. Yet by working smartly and collaboratively, we can adapt to whatever the future holds.”