Over the last ten years the way we manage developer operations has totally transformed. Now it’s time for the same shift to happen in data operations. Today we are launching ParseKit, a new way for data teams to collaborate and manage the procurement of data at any scale. Born out of Enigma’s own work aggregating billions of rows of data across thousands of government sources, ParseKit is a purpose-built infrastructure for getting data where it needs to be through modular pipelines that focus on reliability and collaboration.
Organizations face massive problems managing the increasingly complex operations involved in harnessing both their internal data and crucial sources of public data. Getting data where it needs to be is a non-trivial task and as data pipelines grow linearly, the maintenance and execution burdens faced by organizations grow exponentially.
ParseKit introduces a new paradigm for managing data operations that rests on three key capabilities:
- Onboarding data faster: ParseKit allows teams to minimize repetitive tasks and the maintenance of custom code by building pipelines with our vast library of reusable “steps”. Built-in support for any format, database or protocol allows teams to integrate data from and to any source quickly and scalably.
- Increasing pipeline reliability: Data pipelines require more than just the elastic management of compute resources. Dive into performance metrics and logs collected from each execution cycle to understand what went wrong and where improvements can be made.
- Fostering collaboration and shared awareness: Managing thousands of pipelines requires a high level of visibility across different teams. ParseKit provide teams with a shared real-time view into the scheduling, health and status of every pipeline. View the results of every run and monitor pipeline performance.
ParseKit has already received some fantastic traction helping the City of New Orleans deliver city services more strategically, the California Data Collaborative fight the problem of the state’s scarce water resources, and numerous Fortune 500 companies transform how they bring data to bear on their biggest operational challenges.