Stepping up to Responsibility for Data Migration

[aka mitigating ERP project risk with proper data planning]

John Morris posts a description of a pretty typical data migration: i.e. one in which data quality issues rear their head during user acceptance testing, finger-pointing and accusations fly, and the project timeline falls apart.  Steve Tuck points to similar research for CRM implementations.  I know a couple universities (from my DBA days) with similar stories implementing their Student Information Systems.

Let's face it, most of us who've done a couple migrations have lived this story. 

I've often heard that the two largest risks in an SAP implementation are politics and data issues.  In fact, many system integrators purposely exclude data extraction and preparation from their scope of responsibility.  The stance is "you provide data conforming to this specification and we'll load it into the new system."  This approach reduces the integrator's risk but doesn't much help the customer or the overall timeline.

While I understand the financial drivers in play here, I disagree with the approach.  We can mitigate the risk if we do our jobs right -- which means incorporating a data quality foundation into the project from day one.  Here's a few examples of what should be included:

1. Profile the data early, preferably at project inception.  Use the anomalies and surprises found -- and believe me you will find them -- to improve your understanding of the migration effort.  A side benefit of early profiling is it gets the extraction effort started.  How nice to identify sooner rather than later where the challenges are.
2. Engage the business early to discuss and address data anomalies.  Sometimes data needs to be cleaned in the source, sometimes you've discovered undocumented business rules, sometimes you'll build new ETL rules...you get the idea.  Nearly always you will include business-side subject matter experts to assess the findings.
3. As the business starts configuring the new system, monitor the rules and decisions against the actual data available.  (Aren't you glad you profiled?) 
4. Start testing extract/cleans/load processes early and often, well in advance of user acceptance testing.  Yes, you risk having to tweak the ETL code if the business rules change, but that's a manageable risk.  The opportunity to identify data load issues early is well-worth it.
5. Use that same test system for user training if at all possible.  This gives yet another opportunity to validate the data against business rules.
6. Naturally, if you're consolidating multiple legacy systems, your data profiling includes a cross-system assessment, right?

Addressing data quality early in the project generally reduces risk and overall project spend.  The common "code, load, and explode" implementation can be avoided when business and IT work together with a good methodology.  Whether you're an integrator or an internal resource, stepping up to responsibility for the data needn't be a dangerous move.