A Journey from the Land of 10,000 Lakes - Data Lakes at the Minnesota Department of Health

Sep 11, 2026

Uploaded by Margaret

For years, the Minnesota Department of Health faced slow, manual workflows to share and combine diverse public health data across programs. To enable faster access and more flexible analytics, MDH launched an enterprise data strategy in 2018 and adopted data lakes with Amazon Web Services. The approach supported boundary-water style connections between datasets, enabled user-driven mappings and views, and helped uncover where immunization coverage lagged COVID case rates. Governance guidance evolved to address ownership, access, content, infrastructure, and long-term support.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages4

Reading Time2 min

AccessFree to download

Lessons learned and strategies for data lakes at the Minnesota Department of Health

Building and pivoting the data lake

Connecting lakes and interacting with data

Governing data lakes

A JOURNEY FROM THE LAND OF 10,000 LAKES Lessons learned and strategies for data lakes at the Minnesota Department of Health For years, sharing diverse kinds of public health data across the Minnesota Department of Health (MDH) involved time-consuming, manual processes. Program staff extracted data from one system and then put them on a network drive, or created a separate database for combining data. Access across different program data was limited, and combining data was cumbersome. The agency knew there was a better way of sharing and using data, and in 2018 launched the Office of Data Strategy and Interoperability to advance enterprise data strategies and data exchange across programs of the health department and with external partners. The data strategy team began considering the use of data lakes—centralized systems which can store and process diverse kinds of structured and unstructured data. They reviewed a 2019 Gartner research study that analyzed service providers for data lakes, and chose to work with Amazon Web Services (AWS) for its leadership in the industry. In early 2020, the agency sent a technical team to Seattle to work with AWS on creating a data lake for combining trauma data with death records. This pilot proved helpful when the team had to quickly pivot, building a new lake to deal with the volume of COVID data coming in. The data lakes allowed MDH to quickly build connections between immunization and COVID case data to find pockets of the population where cases were high and immunization coverage was low. They used the data lakes to create maps of vaccination sites and then overlay with vaccination rates, social vulnerability index (SVI) data and distance to sites—identifying where they needed to focus efforts for easy access to vaccination. Aasa Dahlberg Schmit was the director of the Office of Data Strategy and Interoperability at MDH from 2019 to March 2023. She says,“I don’t even think we could have done what we did for COVID without the data lake. I can’t even wrap my head around how we would’ve managed the pandemic if we hadn’t had a data lake, just with all the volume of data, how we needed to combine the data and all the people that needed access to it.” Aasa Dahlberg Schmit [phii.org/dmi-stories](phii.org/dmi-stories) Immunizations Disease Surveillance All Payer Claims Vital Records Negative COVID Test Results Syndromic Data lakes are designed to allow connections between them, also known in the MDH as “boundary waters.”In Minnesota, these separate streams of data are connected with the exception of one. Connecting lakes and interacting with the data Before the data lakes, data across different programs in MDH were siloed, often because policies govern different datasets. Schmit says,“People tend to keep their data to themselves, not because they don’t want to share, but because they think they can’t share.” During the pandemic, the team had limited time to have conversations about protecting different datasets when loaded into a central repository like a data lake. To ensure the right people got access to the data, the agency created a separate data lake for each dataset, such as claims, immunization and syndromic surveillance. Schmit says the separation may indicate the lakes are siloed, but unlike in the past, data lakes are inherently designed to allow connections between them, also known in the department as “boundary waters,” a term used for the border waters between northern Minnesota and Canada. The data lakes contain raw data in different formats, and were a new way for agency staff to interact with the data. Initially, data scientists said it was easy to put data in the lake, but not easy to access it. So, the department added a third-party tool which made it easier for data scientists to create views of the data. With the new user views, staff can now make data mappings themselves. They can also do some of the access control. Schmit says the

FAQs

Why did the Minnesota Department of Health adopt data lakes?

MDH needed a better way to share and use diverse public health data across programs than time-consuming manual processes and siloed datasets.

How did the data lake approach help during the COVID-19 pandemic?

It enabled rapid combination of COVID data with immunization information to identify population pockets with high case rates and low immunization coverage, supporting targeted vaccination efforts.

What governance challenges did MDH face when implementing data lakes?

Initially, MDH lacked time to put formal governance in place, so an ad hoc access and retrieval process was used. Later, governance questions were developed around ownership, access, content, infrastructure, and support.

Sep 11, 2026

Uploaded by Margaret