Cross platform Data Migration

In recent years it’s been observed that, organizations are looking to optimize the storage cost. While doing so they are also looking for the consolidate the data and store it on to new edge technology solutions. Again, here too organizations are looking to opt for the products which do not need licensing. For moving from one product to other, organization need to migrate the data from the legacy systems to the newer systems.

Background: There is more ask in migrating the MF – VSAM file data to be migrated to either to NOSQL or the RDBMS databases. At the same time there is a huge demand in migrating the data from one RDBMS to another and in most cases, it is migration to the opensource databases.

The data stored in those datastores could be one of the followings

• PII data
• Image / large object migration
• Datetime
• Float
• Timestamp
• Float / Decimal
• Exponential
• Null Values
• ANSI Padding (Trailing spaces in Character datatypes)
• Bit Values
• =NULL, <> NULL programming behaviour

Existing datatype usage and what data is getting stored in current system need to be analyzed to map the right datatypes supported on the target databases. Also, some of the critical default functions need to be verified if they are performing similar fashion. If not alternative should identified. Time is crucial in some of the banking applications. And hence the storage differences must be identified and must be validated against the business need. Some databases truncate the additional characters that are supplied, bigger than the character length defined at the column level, some failed to do and hence the process will fail.  

Whenever the rounding of the float data done it might not give the consistent approach. The float data need to be converted to decimal and carryout to see you are getting the consistent results.

 Data migration involves transferring data between different storage systems, formats or even the applications. This is true when it comes to the VSAM data conversion to RDBMS structure. Need to develop the process to extract the data from VSAM to match with the requirement of RDBMS. This process is crucial when upgrading the systems (Storages) or consolidating the databases or moving to a newer platform. Here is a general approach for the data migration:

Steps for Data Migration

1. Assessment and Planning
   1. Evaluate the Schema

Evaluate the existing schema, understand the data structure, data types, volume, quality and dependencies in the current systems, use of obsolete functions, Object type which could be proprietary unique solution, commercial database drivers etc.

1. 2. Define Scope and Goal

Determine which data will be migrated, which is static, transactional and historical data. In historical data not all the data will be required to migrate to the new systems. Determine the cut-off boundary. This will help to define the downtime. There might be more copies of the same data getting replicated / stored in other server and you might need to get rid of this.

2. Pre-migration activities
   1. Backup Data

Create Backups of the existing data to prevent data loss. Review the current back-up process. DR set-up in current environment and its requirement in the target systems.

1. 2. Install and Configure tools

Identify the tools required for data migration. Choose the one which can support source and target system.  Ensure all the set-up is done properly and necessary drivers are installed and connections to the source and target are established.

3. Data Extraction
   1. Extract Data

Use the data extraction tool or customized scripts to extract the data from the source system.

• Format Data

Ensure that the extracted data is in the required format that the target system can understand and retain the same meaning on both the sides i.e source and target. (E.g. Data in Date column at source if consist of mm/dd/yyyy, should be retain in the same format on the target system)

4. Data Transformation

1. • Clean Data

Remove duplicates, correct errors and standardize formats. Most of the time the data is being migrated from older systems and there are chances that the data is redundant or exists in multiple places (two or more places same data exists some time on different servers). This is the time to fix. Also, some applications are retired but still the data remains on the systems, this is the time to clean this too. Even some time encryption of data or PII data which might not be required in the target system, or it’s not been used it is better to get rid of this data too.

1. • Transform Data

Apply necessary transformations to align with the schema and requirements of the target systems. (E.g. Source systems store the character data without trailing spaces, but target system may not support, in this case the data transformation might require as the search conditions might fail in the target system and insert / update statements fail.)

5. Data Loading

2. • Load Data

Use data loading tool or scripts to transfer or load the data to the destination system.

3. • Check Data Consistency

Ensure that the data has been loaded correctly and completely with consistency.

6. Post Migration Activities
7. Data Validation and Testing

4. • Data Validation

Compare the data between Source and destination. Ensure accuracy and completeness of the data.

5. • Functional Testing

Verify that the applications interacting with the data work correctly in the new environment. Compare the outcome on source and target system may be reports or extracted files or even the data processed need to be compared to ensure both systems are working exactly same way.  

8. Cutover Planning

6. • Schedule Cutover

Plan the cutover during the maintenance window to minimize the impact to the end user or the backend processes. Plan to migrate the static data way ahead of the cutover time so as the cutover window can be minimized.

7. • Final Data Sync and Switch over

Perform the final data sync to capture any changes made during the data migration period. If required CDC solutions can be used.  Once the data gets sync up, redirect the users and applications to the new system.

9. Post-Migration Activities

Conduct a thorough review to ensure everything is functioning as expected. Need to monitor the new system for its performance.

10. Conclusion:

Managing the permitted downtime, one must design the strategy to extract and load the data. This might include the slice and dice mechanism. Static data and partitioned ( System may not require this data for day to day ) must be loaded before the actual day of roll-out. On the day of roll-out remaining data to be loaded. Volume sizing to be done prior to the day of roll-out.  If CDC is required all the necessary setup and testing me done prior to roll-out day.