Core Migration Challenges in Database Migration

Core Migration Challenges in Database Migration

Introduction

Cloud Migration consists of moving organizational assets such as applications, databases, IT resources and services. This could be either lift and shift or Re Platform or RE Factor or Rewrite. Clear Migration Strategy provides migration processes, use of the right automation tools and implementation criterial. Man cloud providers facilitate the migration process as much as possible, but not 100% automated way. In this article, Core migration challenges are highlighted while the applications are moved from On Prem to AWS. Many of us think that migration is just a lift and shift provided by cloud provider tools and is not very complex migration process. It is not the case, but it involves a lot of planning and strategizing before the migration process is started.

Assessment challenges

Assessing the entire application and database framework for the scope, size, complexity, technical feasibility and right set of tools for automation will be very important before starting migration journey.

Scoping

Scoping is the process of identifying databases to be upgraded or migrated and application landscape to be modernized which belongs to a business area or group of business units that will fetch considerable benefit to the organization. As a result of this scoping exercise, the application inventory is to be identified for an assessment and vetted by customer. The main challenge is to identify the right tool for getting the inventory in quick span of time and estimating the migration effort for them.

Complexity

Application Complexity is slightly different than migration complexity which is very important to be considered for assessment. This complexity varies from different combinations of source and target technologies. The right set of tools can be used to ascertain the complexity of the migration process.

Technical Feasibility

Understanding the existing application landscape is also important so that any roadblocks will be faced in target due to lower version, unsupported software, deprecated functions, 3^rd party software support etc. Every technical component needs to be understood in terms of version, support, compatibility. In case of product migration, it needs to be checked with the vendor.

Tools and Enablers

Identifying the right set of tools and enablers in different phases of migration is key for the migration success. Once all the tools are identified, finalizing the right tool for inventory collection, analysis, migration, testing and implementation will play a vital role. This can be done by doing proof of concept.

It is essential to get the inventory of applications, databases, and their infrastructure. Database inventory can be collected using AWS DMS Fleet advisor. This will help to collect both total and impacted database objects such as DBs, instances, tables, views, stored procedures, packages etc. As far as the application remediation is concerned, there is no tool from AWS to collect the application inventory and technology stack, the proposed changes for migration.

Code Scanning tool can be built for the patterns that need to be changed during application remediation. The tool will help to identify the list of all application components and impacted components for the application remediation. This is one of the critical challenges for scoping the migration components and estimating them.

Data Collection

During this phase, the information about all the applications, associated databases and their interfaces will be collected in the predefined template. This can be collected in multiple ways.

1.Questionnaire

2. Workshop

3. Access to the system, if not restricted

4. Existing documentation

A minimal set of important questions, including inventory will be asked from the customer in the form of a questionnaire. The responses to all the questions from different application owners will be collected as a single document. If access to the database and applications are given, the data can also be collected through automated way. In most cases, it is difficult to get access from customers as it is secure.

The workshop can be conducted with all the stakeholders who can answer the questions, preferably application owners, DBAs, project managers etc. This could be an interactive session and recording for further reference. In most enterprises, the documentation can help in getting most of the information that is needed for an assessment.

Data Summary

The objective of this phase is to summarize the information collected from various sources. The information is verified for correctness and completeness. The information is further analyzed to get required output for the assessment. As the result of this phase, complete application, database and interface inventory are extracted. Also, AS-IS architecture diagram, existing data model, information of each application and databases, and technology landscape.

Once the data is gathered, the following analysis can be done one by one.

1. Complexity Analysis
2. Feasibility Stud
3. Database Affinity analysis
4. Application Affinity analysis
5. Database vs Application affinity analysis
6. Data Analysis
7. Move Groups/Wave planning
8. Testing strategy
9. Implementation requirements study(includes Parallel run strategy, cut over strategy, etc)

After doing the analysis, the migration roadmap can be prepared based on the analysis report. Roadmap should clearly specify the scope of the migration, complexity of each object to be migrated, highlighting technical roadblocks, risks and mitigation, right set of tools to be used in each phase of the migration, Move Groups based on the affinity analysis, Test automation to ensure the correctness, implementing the target system with right strategy in place.

Database Migration Challenges

Stored Procedure Migration is one of the main challenges while doing database migration to cloud. AWS DMS schema conversion can be used to convert the schema, store procedures, triggers and views. Some of the objects will be migrated with no change, some of them need manual intervention.

Data migration can also be done using AWS DMS or using batch jobs which use export and import utilities. Blob and Clob migration could be challenging while migrating the data and testing them.

Application Remediation Challenges

All db references such as direct sql statement, dynamic sql statement, stored procedure invocation, etc. in the application need to be changed. There is no tool from AWS which changes the application code. The challenge is that the technology stack could be in any language and code scanner can be used to identify where to change the code. But replacing with coverted sqls/stored procedures is manual process and time consuming too. There are 3^rd party conversion tools available which eases this task.

Testing Challenges

Database Migration Testing

                Schema Migration Testing – Ensure whether schema and its objects are correctly converted into the target system. Some of the challenges would be while testing constraints, null values, precision etc.

                Stored Procedure Testing – This testing is considered as complex testing since it involves all business logic. Some of the customer tools are available in the market to test the stored procedures separately and along with applications

                Data Validation – This is the testing which ensures whether all source data is migrated to target. Migrated data should be compared against the source data for any row level/cell value discrepancies, sum values, etc. AWS tool can be used to test them with proper test strategies and test cases. Query comparison should also be made for any result set comparison.

Application remediation testing

All the databases reference in the application are to be tested thoroughly. Direct SQL statements in the application and corresponding converted SQL statements and their fetch data should be compared for any deviation. This could be done partially by the tools. Similarly, Stored Procedure calls should be tested. Main challenge here is to test the dynamic SQLs which will be time consuming.

Reports Testing

Testing of the reports which are produced by the application and any other reporting software will be very challenging. The report data in each field needs to be tested along with formatting which is cumbersome and time consuming.

Infra Testing

Apart from application and database level testing, infrastructure level changes should be tested. This is to ensure that applications are compatible with AWS services and configurations. Verifying that applications can scale up or down as needed in the cloud is also very important.

Security Testing

When the data is moved to the target platform, the data needs to be encrypted as it was in the source system when it was transit and in rest. Also, Access control mechanisms such as IAM roles ad permissions are to be properly configured.

Performance Testing

Performance or existing critical components should be captured and should be compared against converted components. Performance should be either better than or equal to source system.

Integration Testing

Ensure that applications should be correctly integrated with AWS services or other 3 party software. Also, 3^rd party components such as payment gateways should be integrated with applications.

Implementation Challenges

There are a lot of challenges while moving the data within the cutover window. Cutover strategy prepared during assessment phase which includes parameters like zero downtime, minimal downtime, cutover windows, parallel run etc, will be used. One off migration or incremental data migration strategy based on the volume of data moved within cutover window. Ensure that all data integrity issues are sorted out. AWS takes care of high availability and minimal downtime.

Conclusion

There are a lot of challenges and risks involved while performing data base migration and application remediation for any modernization project. Proper planning, well documented migration strategy, right set of tools will be the key for success of any migration project. There are several services in AWS which could be leveraged along with custom built tools and enablers. In some cases, 3^rd party tools can be used for kind of migrations.