How data verification dictates the success of your data migration project

Data migration, whether it involves transporting electronic data and associated metadata from an application/system to a new or upgraded one, migrating data from one database to another or consolidating data from multiple sources, is a complex and challenging process. While it may seem as simple as moving data from a source to a target, data migration in a GxP environment is a critical and time-consuming exercise that requires meticulous planning, management, and execution. Organizations must be aware of potential pitfalls to avoid costly mistakes. According to the PIC/S Guidance, the migration of data from one system to another should be performed in a controlled manner, in accordance with documented protocols, but it should also include appropriate verification of the complete migration of data (1).

In fact, ensuring the accuracy, completeness and consistency of the data being migrated makes one of the biggest challenges in data migration. In this article, we will focus on how data verification can impact the success of a data migration project and how it helps to overcome some challenges encountered during the execution of a project.

What is Data Verification and why is it so important?

Data verification is a process of checking the accuracy, completeness, and consistency of data during data migration. It involves comparing the source data with the target data to ensure that the migration was successful.

Data verification ensures the quality of the migrated data. It helps identify any issues both during and after the migration process, as well as reduce the risk of data errors after the migration that could prove costly for the company. These costs may include delays in batch releases, costs of additional data verification and potential rework.  It can even lead to product recalls impacting company’s reputation, and most importantly, puts product quality and patient safety at risk.

Data integrity challenges during data migration and how to implement data verification in your project

1. Source Data Quality

One of the biggest challenges of data migration is ensuring the quality of the data being migrated. Before starting the data migration process, it is essential to assess the quality of the source data. This assessment can help identify any data gaps that need to be addressed before the migration begins. It involves performing data profiling to ensure alignment with the target system´s requirements and decide whether data transformation, cleansing and/or enrichment are necessary.

2. Mapping of Data

Mapping data from one system to another can be a complex task. Data may have different structures, formats, and definitions in different systems, and mapping them correctly can be challenging. This mapping is critical for ensuring data quality and avoiding data loss throughout the migration process. This challenge can be overcome by engaging data migration experts who have experience in mapping data from one system to another and are knowledgeable of the data structure in both legacy and target systems.

3. Effective implementation of Data Verification

To effectively implement data verification, it is essential to implement data verification rules, monitor data quality during migration and verify data after migration:

• Implement Data Verification Rules

Implementing data verification rules can help ensure data quality throughout the data migration process. Identifying data that does not meet required standards prevents the migration of incomplete, inconsistent, inaccurate, or non-compliant data.

• Monitor Data Quality During Migration

To ensure seamless data transfer without errors, it is critical to monitor data quality during the migration process. This can be done by applying the data verification rules and setting up data quality checks in the different steps of the data migration process. Any issues or errors should be addressed immediately to prevent the propagation of data quality related problems to the target system.

• Data Verification After Migration

After the data has been migrated into the target system, it is important to verify that no data loss occurred throughout the process and that data integrity is maintained. This is done by comparing data in the source and target systems, following the ALCOA+ principles. These principles ensure that data is attributable, legible, contemporaneous, original, and accurate. By adhering to these principles during data verification, pharmaceutical/biotech companies can maintain the highest data quality standards, safeguarding data integrity throughout the migration process.

To achieve successful data verification:

Sampling rules: these rules provide statistical confidence when selecting representative data subsets, help reduce the workload and minimizing human errors. More detail about sampling rules and its importance for data verification can be found here (2).

Automated tools: these tools can be used to verify 100% of data and ensure accuracy when comparing source and target data (3). The use of tools is very helpful to not only to flag discrepancies, reduce the workload but also minimizing human errors.

The Importance of a Validation Expert in your Data Verification Process

One way to overcome data verification challenges and implement the right data verification approach is to incorporate an experienced validation manager into your project. A validation manager brings expertise in the data verification process and best practices, helps identifying and mitigating risks, ensures that the project meets regulatory and business requirements and ultimately, monitors Good Documentation Practices throughout the data migration project.

With all their expertise, a validation manager usually develops a data migration plan in collaboration with the data migration experts. This data migration plan includes the data migration strategy, the definition of data verification rules and risk assessments and mitigation plans.

Conclusion

Data migration is a complex process that requires careful planning, management, and execution. Ensuring data quality throughout the data migration process is a significant challenge in data migration and is critical to the success of the project. But, with the right data verification approach, companies can overcome this challenge and trust their migrated data in the new system.

Do you need help with defining your data verification strategy and execution? GxP-CC can support you in choosing the adequate risk-based data verification approach during data migration from a legacy system to a new target system with all relevant data integrity requirements. Contact us today to get started!

References

(1) Pharmaceutical Inspection Convention, Pharmaceutical Inspection Co-Operation Scheme, July 2021, PIC/S Guidance, Editor: PIC/S Secratariat; https://picscheme.org/docview/4234

(2) https://www.gxp-cc.com/insights/blog/statistic-based-verification-on-computer-system-validation/

(3) ISPE GAMP 5 Guide: A Risk-Based Approach to Compliant GxP Computerized System – 2nd Edition, 2022