The State of Arizona Manages Water Resources More Sustainably With Pentaho Data Catalog

Challenge

The State of Arizona’s Department of Water Resources protects, conserves, and improves the state’s water supply for over seven million inhabitants. The Department must sustainably manage Arizona’s water resources amid extreme temperatures and arid desert landscapes. Its work directly impacts Arizona’s present and future economy and environment.

To effectively manage Arizona’s water resources, the Department collects, stores, and conducts analysis on reported water uses of nearly 6 trillion gallons per year and tracks ownership and types of water use for over 300,000 wells and surface water applications. It also records geolocation coordinates, depth to water, and investigates water uses for the adjudication of water rights.

Without a metadata repository, new staff had difficulty finding important data sets. The organization also needed automated data quality, since, with limited resources, hiring more staff wasn’t feasible. “We struggled to find people and affordable tools that could completely fulfil our requirements. We needed a business glossary, but most catalog software didn’t sell that separately, so we created our own in SQL. However, we later realized we could not implement more mature data management best practices without a data catalog tool.”

To simplify data management and help ensure the sustainable management of its water resources, the Department needed an intelligent data discovery and automation tool. Throughout the project, it was essential that the Department could search legacy records, ensure citizen records are secure and minimize downtime.

Solution

The State of Arizona’s Department of Water Resources selected Hitachi Vantara’s Pentaho Data Catalog in May 2020. The automated platform offered all the functionality the Department needed to manage Arizona’s extensive data sources with fewer resources. The solution also included:

• Customized data discovery objectives
• Cataloging process using metadata profiling
• A fully automated migration plan
• Cyber security expertise

Hitachi Vantara prepared the platform to meet the Department’s security requirements and permission framework in advance to prevent delays to data migration. Using machine learning, the Hitachi team trained the Data Catalog, part of the Pentaho Platform, to recognize different structured data types. This data includes geo-coordinate data, personal information strings, and sensitive court data related to on-going adjudication processes. The machine learning algorithms sampled legacy records at scale and generated accompanying metadata to establish key lineage relationships.

The entire data discovery, metadata cataloging and platform migration were completed in just two weeks, with minimal downtime. Via Pentaho Data Catalog visualization dashboards, the Department can now understand, integrate and analyze the unique and critical data sets it has to meet the needs of Arizona’s water users, planners and decision makers. “One of the reasons we’re excited about Universal Data Intelligence is that you just need to type in ‘well’, and an accurate and comprehensive report of over 300 instances of that data element in our transactional databases, data warehouse , document management system and spatial data is ready for export, which is saving us so much time,” comments Lisa Williams, Manager, Office of Enterprise Data Management at State of Arizona’s Department of Water Resources.

Outcome

More accurate data enables sustainable water demand and supply planning

With Hitachi Vantara’s intelligent automation solution, the State of Arizona’s Department of Water now has better visibility of its vital data and total trust in its quality. “In the past, we have had differences in definitions and understandings of what key terms meant,” comments Lisa Williams, Manager, Office of Enterprise Data Management at State of Arizona’s Department of Water Resources.  “We are now modernizing all our licensing and applications processes. Having a centralized metadata repository enables our staff and the consultants to quickly understand existing data used in on-line and automated processes during the project discovery process. And when data is migrated to the cloud, we’ll know the complete lineage of the data.”

The unified catalog of metadata enables staff across the whole Department to create consistent frameworks for ongoing analysis. For example, groundwater hydrologists, water resource managers and application developers can compare definitions of water usage and standardize how the Department understands, compiles, and analyzes usage in the real world.

With a more accurate view of its data and the status of its water resources, the state can protect, conserve, and enhance Arizona’s water supplies with in a bold, thoughtful, and innovative manner. It can also collaborate with other stakeholders with confidence in its analytics and modeling. As a result, the Department is better equipped to manage water resource challenges now and in the future.