If you’ve ever been a part of a Salesforce implementation go-live that had its window extended by unforeseen issues with the data migration, you know how critical it is to plan your data migration and test it frequently. Have you ever wondered if it’s possible to sail through go-live with no incorrectly formatted email addresses, “Unable to lock row” errors, or invalid picklist values? If you yearn to deliver on time and enjoy more of your weekend, you’ve come to the right post! Here are seven key components of a successful data migration.
Like all development, data migrations suffer when a new developer or architect is onboarded and the technical documentation is insufficient to get them up to speed. Do yourself a favor: Don’t put yourself in a position where information about the data migration exists only in your head, and you don’t have time to explain it. Develop a data mapping workbook that comprehensively describes how each object is migrated. Create the workbook as a spreadsheet with a tab for each object, and a row for each field to be migrated. In each row, clearly describe the source field, target field, and transformation logic (written as pseudocode).
In agile projects, changes to objects and fields may occur frequently throughout the project. Implement a policy that requires the data migration team to review and approve any pull requests that create, rename, delete, or modify objects or fields. To make this process foolproof and easy to enforce, implement a policy-as-codeapproach, adding a required reviewer group any time modifications are made to the schema. This enables the data migration team to keep the data mapping workbook up-to-date and minimizes the potential for human error inherent in relying on developers to self-report changes that would impact the workbook. For more on how to automate this approach (and why it’s a good idea), seePolicy-as-Code for Leaner Governance.
Any process that relies on a human to execute it is prone to human error, which, by its nature, is unpredictable and cannot be mitigated through testing. To ensure that your process is repeatable, reliable, and testable, automate as much of your data migration as possible. At a minimum, automate your transformations and data quality reports.
In project environments where two or more individuals are working on the data migration, it becomes more critical to establish policies for quality standards, code reviews, and governance just as you would for configuration and code changes. To ensure that you can implement these policies, use version control and continuous integration for release management. This will enable you to add required reviewers to pull requests, test that the data migration build is not broken, and run the data migration when pull requests are merged.
In large enterprise environments, the goal is to automate absolutely every aspect of the data migration. For steps that require manually intervention, use a browser automation tool like Selenium to automate them.
In many data migrations, data from multiple legacy systems must be migrated to Salesforce. In some cases, you may need to identify relationships between tables in different systems, or identify records that are duplicated across systems. Matching and restructuring data to fit into a new data model takes a heavy toll on performance, especially when data volumes are high. Relational databases come with robust, well-documented, and highly-performant programming languages that are optimized for data transformation.
To make use of these data transformation languages, your data needs to be co-located in a single database. Create staging tables that reflect the schemata of the source systems. Before each data migration (there will be many test migrations), clear your staging tables and load the data from the source systems into the staging tables. From there you can use a data transformation language to prepare your data to be loaded into Salesforce.
