Helping a Government Agency Align for AI-Powered Impact
Artificial Intelligence
RevGen helped this government agency break down silos and accelerate AI opportunities through strategic collaboration
Building a Data Pipeline for a Seamless ERP Migration
RevGen designed, built, and executed a Snowflake data pipeline to shift millions of client data records from legacy systems into a new cloud ERP platform.
Project Overview
Our client, a leading manufacturer and wholesale distributor, knew they needed to modernize their ERP platform and processes. They understood that the migration of their legacy data was a massive challenge and the accurate transformation and migration of that data into their new cloud ERP platform would be critical to their success. RevGen built a solution focused on business input, scalability, and automated workflows to enable a smooth go live on day one.
Client Challenge
During this project, RevGen had to ensure the client remained confident in their ERP investment, had all their data, and that nothing crucial to the running of the business broke during the migration.
Potential disruptions to day-to-day operations
Our client needed to convert data from across all business functions into new formats while continuing normal business operations until the official switch.
Degraded data quality
Data quality had degraded in their legacy system, resulting in data which could not be migrated to the new ERP. This included orders which had been open for years, inaccurate and conflicting customer and vendor information, and products which were used once and never again. Through the migration process, the client needed to also improve their data quality by excluding old and inaccurate information from the migration.
Building business confidence in the migration
To ensure the business was confident in their data on day one and enable operations, our client needed 100% traceability on all legacy data assets and whether they were transformed, migrated, or removed.
Approach
Key was ensuring we could use the client’s existing tech stack, which included Matillion and Snowflake, to lift data from the legacy systems into the new ERP. Within Snowflake, a layered approach was taken to source, transform, and flag data records for business review, exclusion, or migration. After migration, field level validation audit was performed to ensure the data in the new ERP matched the legacy values.
[Read More: Delivering Analytic Ready Data through a Snowflake Medallion Architecture]
Solution
Through the use of Snowflake and Matillion, we were able to build a custom pipeline to migrate and transform the legacy data.
Integrated master data into Snowflake
We linked master data held in Snowflake tables to Excel spreadsheets, which enabled business process owners to clean and transform data elements for migration without updating data in their legacy ERP. This kept day-to-day operations running smoothly while correcting potential issues in this migration.
Anticipated and mitigated known failure points
We developed reports and processes to flag missing dependencies in the data as well as schema errors which would cause failures during the migration. These were distributed back to the business through resolution workflows and business validation checkpoints.
Structured the data pipeline with emphasis on audit and oversight
A layered approach within Snowflake provided a fully transparent and accountable data pipeline as it moved from the source systems, through our Snowflake layers, and into the final API templates.
Results
Through the successful completion of the data pipeline, we resolved client concerns about the data and the overall migration, and even pulled the timeline up for migration completion.
Accelerated client readiness
Our newly automated end-to-end Snowflake orchestration to source, convert, and output 57 datasets containing 900 fields and 3.4 million rows was able to execute in under 10 minutes. This speed ensured the data was always ready for the next step during the week–long go–live process.
Complete confidence in data pipeline
The transparent Snowflake solution with its traceable data lineage and audit reporting boosted confidence in the migrated data and allayed lingering concerns about the ability to use the new ERP, which had been a massive investment. Without having to worry about the data migration, the business was able to focus on enabling their people and processes to work with the new ERP.
Enhanced data quality across all operational functions
Improved data quality through the migration framework has already helped the client extract more meaningful insights from their data, while enabling them to consider more advanced analytical and AI use cases.
Success Stories
Enabling a Major Telecommunications Provider to Prioritize and Operationalize AI
Artificial Intelligence
RevGen helped our client turn blue-sky ideas into a realistic, sequenced AI roadmap
Delivering Analytic Ready Data Through a Snowflake Medallion Architecture
Technology ServicesAnalytics & Insights
RevGen built a Snowflake Medallion architecture for our client to help consolidate historical on-premise data sources with new cloud platform data to deliver a comprehensive reporting and analytic environment.
Subscribe to our Newsletter
Get the latest updates and Insights from RevGen delivered straight to your inbox.