Introduction to AWS DMS and SCT
AWS DMS and SCT address the complexities of moving database data and schemas to the cloud.
The complexity of moving data from on-premises to the cloud, which often requires significant expertise, custom scripts, and application downtime. AWS DMS aims to simplify this process for seamless and reliable database migration.
AWS DMS is a fully managed service that makes it easier to migrate databases to AWS. It allows for continuous replication of data from various source databases to target databases.
A service that enables quick and secure database migration to Amazon Web Services (AWS). It allows you to move your databases without significant downtime.
AWS SCT converts database schemas and code objects from one database engine to another. It is crucial for heterogeneous migrations.
Data Migration Techniques
DMS supports different data migration techniques based on downtime tolerance.
This process involves stopping the source application, taking a full database backup, copying the backup to the cloud, and restoring it to AWS RDS or EC2. The application is then started with the new target database.
This process involves performing an initial full load, then AWS DMS monitors the source database transaction log for ongoing changes (inserts, updates, deletes) and applies them immediately to the target database. This allows the application to continue running on the source database while the target database catches up.
Database Migration Types
Database migrations are categorized based on whether the source and target engines are the same or different.
Moving data between the same database engines (e.g., on-premises Oracle to Oracle on Amazon RDS or EC2). Table structures and SQL code are generally compatible. Primarily requires AWS DMS for data movement; AWS SCT is typically not needed.
Moving data between different database engines (e.g., on-premises Oracle to RDS PostgreSQL or Aurora PostgreSQL). Database engines differ, meaning schemas, stored procedures, and custom SQL code are not immediately compatible. Requires both AWS DMS for data movement and AWS SCT for schema and code conversion. AWS SCT simplifies engine conversion without requiring extensive in-house expertise. This is a two-step process: Schema conversion using AWS SCT followed by Data migration using AWS DMS.
AWS DMS Features and Capabilities
AWS DMS provides a robust set of features to facilitate secure and efficient database migrations.
Key capabilities of the AWS Database Migration Service.
Managed Service
AWS handles the operational overhead of the replication instance, which is the server performing data movement.
Use Cases:
- Reducing administrative burden for users
Data Movement Versatility
Can move data securely to AWS, from AWS (to on-premises or other clouds).
Use Cases:
- Cloud migrations
- Hybrid cloud scenarios
Resilience and High Availability
Offers self-healing capabilities and a multi-AZ option for the replication instance, providing high availability and failover support.
Multi-AZ option
Available for the replication instance
Use Cases:
- Ensuring continuous migration process
Downtime Minimization
Supports online migration using Change Data Capture (CDC) for near-continuous data replication, allowing applications to keep running during the migration process.
Use Cases:
- Production database migrations
Migration Support
Supports both homogeneous and heterogeneous migrations (with SCT for the latter).
Use Cases:
- Any database migration scenario
Database Engine Versatility
Supports over 20 different database engines as sources and targets, including non-traditional targets like Amazon S3 and MongoDB.
Supported engines
Over 20 different database engines
Use Cases:
- Migrating diverse database ecosystems
Data Validation
Performs data validation to ensure integrity between source and target.
Use Cases:
- Ensuring data quality post-migration
Syntax Migration Conversion
Can perform syntax migration conversion.
Use Cases:
- Part of heterogeneous migration preparation
AWS DMS Architecture Components
The AWS DMS architecture is comprised of several key components that work together to facilitate database migrations.
The engine that performs the migration. This is the server AWS manages to handle the data movement.
Defines the specific migration process, including source and target details.
The original database to be migrated.
The destination database on AWS.
AWS DMS Supported Endpoints (Source & Target)
AWS DMS supports a wide array of database engines and services as both sources and targets for migration.
AWS DMS's versatility extends to its extensive support for various database environments.
Source Databases (On-Premises / EC2 Hosted)
DMS can pull data from a variety of on-premises databases or databases hosted on EC2 instances.
Engines
Oracle, SQL Server, MySQL, MariaDB, PostgreSQL, MongoDB, SAP ASE, IBM DB2
Use Cases:
- Migrating legacy on-premises databases
- Consolidating databases from EC2
Source Databases (Other Cloud Providers)
DMS also supports migration from databases hosted on other major cloud platforms.
Engines
Microsoft Azure SQL Database, Microsoft Azure MySQL/PostgreSQL Flexible Servers, Google Cloud MySQL/PostgreSQL, OCI MySQL HeatWave
Source Databases (AWS Managed Services)
DMS can be used to migrate data between various AWS managed database services.
Services
Amazon RDS (Oracle, SQL Server, MySQL, MariaDB, PostgreSQL, IBM DB2), Amazon Aurora (MySQL and PostgreSQL compatible), Amazon DocumentDB, Amazon S3 (non-conventional)
Use Cases:
- Consolidating AWS databases
- Data transformation to S3
Target Databases (On-Premises / EC2 Hosted)
DMS can also migrate data out of AWS to on-premises or EC2-hosted databases.
Engines
Oracle, SQL Server, MySQL, MariaDB, PostgreSQL, MongoDB, SAP ASE, DB2 (implied), RDS OSS
Use Cases:
- Data synchronization to on-premises
- Migration to self-managed EC2 databases
Target Databases (AWS Databases & Services)
A wide range of AWS databases and services are supported as targets, including relational, NoSQL, data warehousing, and streaming services.
Services
Amazon RDS (all engines, including Aurora and Aurora Serverless), Amazon Redshift and Redshift Serverless, Amazon DynamoDB, Amazon OpenSearch Service, Amazon Elasticsearch Service, Amazon Kinesis Data Stream, Amazon DocumentDB, Amazon Neptune, Apache Kafka (including Amazon MSK and self-managed)
Use Cases:
- Migrating to various AWS database types for different workloads
AWS Schema Conversion Tool (SCT)
AWS SCT is a crucial desktop application for converting database schemas and code during heterogeneous migrations.
Used for heterogeneous migrations where source and target database engines are different. It should not be used for homogeneous migrations. SCT manages the structure and code conversion, while DMS manages the data movement.
A desktop application installed on a local computer or EC2 instance.
Analyzes the source database schema; evaluates stored procedures, functions, and database structures for compatibility with the target; generates a detailed report indicating what can be automatically converted and what requires manual effort; highlights incompatible SQL functions or code (e.g., specific Oracle SQL functions not working in MySQL) and provides detailed instructions for manual handling. It also handles relational and data warehouse schemas, database objects (schemas, views, stored procedures, functions), and embedded SQL statements within application source code.
The AWS SCT supports a wide array of database conversions, including but not limited to: Oracle to Aurora, MySQL, PostgreSQL, MariaDB, and Amazon Redshift; Microsoft Azure SQL Database to Aurora, MySQL, and PostgreSQL; Microsoft SQL Server to Aurora, Amazon Redshift, MySQL, PostgreSQL, and MariaDB, and many more combinations.
How AWS DMS and SCT Work Together (Heterogeneous Migration)
procedure
A heterogeneous database migration requires a two-step process involving both AWS SCT for schema conversion and AWS DMS for data replication.
The process outlines how to convert the schema and then migrate the data, ensuring minimal disruption.
Prerequisites
- AWS SCT desktop application installed
- Connectivity to source and target databases
1
Download, install, and configure AWS SCT.
SCT is a desktop application required for schema analysis and conversion.
2
Create an SCT migration project, connecting to source and target databases.
To define the scope of the migration and establish database connections for analysis.
3
Initiate schema conversion. SCT performs assessment and compatibility analysis.
SCT needs to analyze the source schema and identify compatibility issues with the target.
4
SCT generates a schema assessment report detailing compatible and incompatible objects, with suggestions for errors.
To provide insights into the conversion effort and manual intervention required.
5
Apply the converted schema to the target database.
To prepare the target database structure for the incoming data from DMS.
6
Create a replication task in AWS DMS.
To configure the data movement parameters.
7
Disable all foreign key and referential integrity constraints on the TARGET database.
Failure to do so will interrupt the data copy process. This is NOT done on the source database.
8
Start the DMS replication job.
To begin the data transfer from source to target.
9
Perform random sampling and other measures to validate data integrity on the target.
To ensure that data has been migrated accurately and completely.
10
Re-enable the foreign key and referential integrity constraints on the target database.
To restore the database's data integrity rules after the migration is complete.
DMS in Broader AWS Migration Context
DMS plays a specific role within the larger AWS migration landscape, distinct from other migration services.
AWS Application Migration Service (formerly CloudEndure Migration) automates the actual migration of entire applications and data to AWS, using continuous block-level data replication. An agent is installed on the source server, mirroring disks to a staging area in AWS. DMS, in contrast, is specifically for migrating/replicating databases.
For migrating on-premises Oracle databases to AWS, Amazon RDS for Oracle is the correct service for a like-for-like replacement. Migration methods include AWS DMS (Database Migration Service) for minimal downtime with Change Data Capture (CDC), Oracle Data Pump for export/import, and native Oracle tools like RMAN.
DMS (Database Migration Service) is for migrating/replicating databases, not general-purpose file transfers like JSON. For transferring large amounts of daily JSON instrumentation data from an on-premises SAN to Amazon S3 for near-real-time analytics, AWS DataSync over AWS Direct Connect is the most reliable solution, as DataSync is purpose-built for securely and efficiently transferring large amounts of data from on-premises to AWS.