TW202411858A - Systems and methods for database migration, and systems for migration of an online database - Google Patents

Abstract

A system for database migration including one or more processors and one or more memory devices storing instructions The processors are configured to perform operations for database migration that include cloning the online database to a first clone database, generating export files from the first clone database and importing them to a migration database, performing a row comparison of the first clone database and the migration database after importing the export files, and initializing a double write operation replicating writing of the online database on the migration database to capture incremental data. The operations also include cloning the online database to a second clone database after performing the row comparison, identifying catch-up data by comparing the second clone database with the incremental data, and updating the migration database to include the catch-up data.

Description

Data transfer system and method

The present disclosure generally relates to computerized systems and methods for database migration. In particular, embodiments of the present disclosure relate to systems and methods for improved migration between real-time databases, using dual write, serial replication, and consistency verification while minimizing downtime and automating traffic handoff of the migrated database.

Database migration is the process of transferring data between different databases. When the migration is complete, the data in the original (or source) database is copied to the transfer (or target) database, and in some cases is reorganized. The database migration process may also include rerouting instructions (e.g., write or read instructions) from the original database to the transfer database. In some scenarios, the database migration may be homogeneous, where the original database and the transfer database have the same database management system. In other scenarios, the database migration may be heterogeneous, where the original database and the transfer database have different database management systems. Database migration may include the processes of database replication (also known as database streaming) and database transfer. Database replication copies the data in the original database to the transfer database. Database transfers will move, replace, and/or accelerate resources between databases.

Database migrations are associated with certain risks, which can be particularly problematic when migrating a real-time or instant database (i.e., a connected database that processes workloads at a constant throughput) that is extremely time-sensitive and/or accessed in parallel by clients. During a database migration, there is a risk of data loss (e.g., some of the data may not be migrated from the source system), semantic changes (e.g., semantic errors may occur, thereby creating inconsistencies), extended downtime (e.g., when the data migration process takes longer than expected, thereby preventing access to critical data), and data corruption (e.g., undesired data may be migrated to the new system, resulting in potential crashes and errors). Particularly relevant to live database migrations, additional risks must be considered. For example, migrating a database that is connected to the cloud and actively accessed and modified by users presents the additional challenges of application stability risks (e.g., applications that rely on the database can become unstable to improper development or improperly written code, resulting in power cycles or failures), orchestration issues (e.g., when the data migration process is performed out of sequence), interference issues (e.g., when multiple stakeholders utilize the application simultaneously during the migration process), and/or improper application parameterization (e.g., the target system can become incompatible with the data migration program).

Furthermore, database migration has become increasingly difficult and critical with the rise of cloud computing. Today, databases - and the information they store - can be one of a business's most valuable assets. The entire operation of a business can rely on accessing, updating, and modifying the database. For some businesses, this information must be persistent and available instantly. Downtime can cause business disruption, customer dissatisfaction, and missed opportunities. Additionally, current databases can be massive, often with terabytes or even petabytes of information, and require extremely high query throughput to maintain a large client base with high database demands. For example, a database that supports a streaming service not only has an extremely large data set but also requires the database to be delivered quickly. This combination makes real-time or instant database migration particularly complex because large databases must be migrated while the databases are operating (both providing and receiving new data) with minimal downtime and with low error tolerance.

The disclosed database migration system and method solves one or more of the problems described above and/or other problems in the prior art.

One aspect of the present disclosure relates to a database migration system. The system includes: one or more processors; and one or more memory devices, wherein the one or more memory devices store instructions that configure the one or more processors to perform operations when executed by the one or more processors. The operations may include replicating an online database with online traffic to a first replica database; generating an export file from the replica database and importing it into a migration database; performing a column comparison between the first replica database and the migration database after importing the export file (the column comparison is performed by calculating the sharding keys of a selected number of columns in the online database and the migration database); and initiating a dual write operation to replicate the write of the online database to the migration database to capture incremental data. The operation may also include replicating the online database to a second replica database after performing the row comparison, identifying catch-up data by comparing the second replica database to the delta data, and updating the staging database to include the catch-up data.

Another aspect of the present disclosure relates to a database migration method. The method may include replicating an online database to a first replica database; generating an export file from the first replica database and importing it into a transfer database; performing a column comparison of the first replica database and the transfer database after importing the export file (the column comparison is performed by calculating the shard keys of a selected number of columns in the online database and the transfer database); and initiating a dual write operation to replicate the writes of the online database to the transfer database to capture incremental data. The method may also include replicating the online database to a second replica database after performing the column comparison, identifying catch-up data by comparing the second replica database with the incremental data, and updating the transfer database to include the catch-up data.

Another aspect of the present disclosure relates to a system for migrating an online database. The system may include: a first database coupled to a network; a second database; at least one processor coupled to the first database and the second database; and at least one memory device storing instructions. When such instructions are executed by at least one processor, they configure at least one processor to: copy the first database to the first replica database; generate an export file from the first replica database and import it into the second database; perform a column comparison of the first replica database and the second database after importing the export file (the column comparison is performed by calculating a shard key of a selected number of columns in the first database and the second database); and initiate a dual write operation to copy the write of the first database to the second database to capture incremental data. The instructions may also configure the processor to: copy the first database to the second replica database after performing the row comparison; identify the catch-up data by comparing the second replica database with the incremental data; update the second database to include the catch-up data; and perform a second row comparison between the second replica database and the second database to determine whether the data is consistent. The instructions may also configure the processor to: after determining that the data is consistent, suspend the dual write operation and switch the online traffic from the first database to the second database.

The following detailed description refers to the accompanying drawings. Whenever possible, the same figure numbers are used in the drawings and the following description to refer to the same or similar parts. Although several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, the components and steps shown in the accompanying drawings may be replaced, added to, or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing, or adding steps to the disclosed methods. The following detailed description is not limited to the disclosed embodiments and examples.

The words (e.g., A or B) used to reference individual elements in a figure do not specify the number of elements or the total number of elements. Instead, they are variable references that indicate variable numbers of elements and total numbers of elements. For example, the word B does not indicate that the element with the word "B" is the second. Instead, B is a variable reference that can indicate any integer.

Embodiments of the present disclosure relate to database migration systems and methods. The disclosed systems and methods are capable of quickly and stably migrating databases using one or more replica databases and writing in parallel to facilitate data migration and traffic delivery. The disclosed systems and methods may be applicable to real-time databases used to handle workflows under constant throughput in real-time operations. For example, the disclosed methods and systems may be used to support databases for online applications of e-commerce with constantly changing and high traffic demands. The disclosed systems and methods solve the problems of traditional database migration processes. Traditional systems frequently suffer from longer than acceptable database downtime. Current online applications frequently require almost constant availability, and any downtime can be detrimental to operations. Database downtime can result in, for example, customer dissatisfaction and poor retention rates. The disclosed embodiments provide systems and methods for performing real-time database migrations with minimal or no downtime.

In addition, the disclosed systems and methods can improve the quality of data transfers that employ a catch-up operation to verify that all data has been transferred. The transfer of a real-time or online database can be cumbersome because dynamic interactions in the original database can produce differences between the original database and the transferred database. Because it is undesirable to disconnect the real-time database, the data in the online database can be modified during the transfer process. These concurrent changes can produce inconsistencies between the original database and the transferred database. The disclosed systems and methods use a method of multiple replica source databases and apply one or more catch-up and verification processes to solve these problems to ensure that the transfer results in a complete data transfer and/or replication.

Some of the disclosed embodiments are applicable to cloud-based databases. For example, some of the disclosed embodiments are applicable to cloud-based relational database services. In such embodiments, the disclosed systems and methods may include data distribution, automatic replication, and Structured Query Language (SQL) compatibility functions.

The disclosed systems and methods may also minimize downtime during a migration by using database replication. Some of the disclosed embodiments include replicating a database by creating a point-in-time copy of a production or online database. In such embodiments, online applications may continue to operate regularly during the migration operation, thereby maintaining the same data structure and content. Such embodiments of the disclosed systems and methods enable databases to be replicated in parallel to minimize downtime. For example, databases may be replicated simultaneously and/or sequentially to obtain assigned data structures that replicate the source database, and to perform server operations. Therefore, certain embodiments of the disclosed systems and methods enable database migration with virtually no downtime.

Furthermore, the disclosed systems and methods may even allow for efficient migration of databases with high traffic loads and/or large datasets. Current online applications are frequently data intensive and may require extensive data usage and transfer. Migration of such online or real-time databases presents additional challenges. For example, database migrations via traditional systems may be too long and cumbersome due to replication operations that may result in delays and/or data omissions. Therefore, for certain database migrations, it may not be possible to use traditional methods without hindering operations. The disclosed systems and methods address these issues by providing methods that allow for the transfer of databases with high traffic loads and/or large datasets while minimizing downtime and improving data accuracy.

In some embodiments, the disclosed systems and methods facilitate data migration using dual write features and catch-up operations. Traditional migrations of real-time databases may require pauses in real-time traffic to prevent changes during the migration cycle. However, such pauses can disrupt operations and lead to failed systems and/or customer dissatisfaction. The disclosed systems and methods address these issues by employing dual write features and clustering systems that allow migrations without pausing the database. Some of the disclosed systems and methods may combine clustering algorithms with dual write operations to isolate data changes, which can then be replicated in the migrated database without disconnecting the original database. In addition, some of the disclosed systems and methods may include testing dual write connections and operations to minimize data inaccuracies during the migration.

In addition, the disclosed systems and methods may facilitate automation of the migration process. Some embodiments of the disclosed systems and methods may include an automated system that coordinates multiple online tools to perform database migration. For example, some of the disclosed systems and methods may automatically capture data from a database using, for example, automatic import of SQL files. In addition, the disclosed systems and methods may configure tools to automatically import SQL files and/or metadata into other databases during the migration process without user interaction. In addition, the disclosed systems and methods may also facilitate automatic and/or parallel data validation. In some embodiments, the disclosed systems and methods may include tools for validation processes in context and data clustering to identify inconsistencies.

In addition, some embodiments of the disclosed systems and methods can address the problem of poor data consistency during migration. Particularly for large dynamic data sets (such as databases that support online applications), the data migration process can cause data inconsistencies and/or corruption. Some of the disclosed systems and methods use tools for data verification to address these problems, as well as tracking tools in specific areas to correct. For example, some of the disclosed embodiments may use tools that compare data sets to identify inconsistencies and corrupted files. These tools may use tools to check accuracy and inconsistencies after data migration. These methods may include the use of Source Data Verification (SDV) or hash-based verification, such as using MD5 file checksums. In these embodiments, data validation can help automatically and concurrently determine whether data is accurately translated, complete, and supports processes in new systems as it is transferred from one source to another. Additionally, embodiments of the disclosed systems and methods also support the use of chase tools for concurrent correction of inconsistencies and data clustering mechanisms to identify areas of discrepancy and prevent erroneous data loss.

The disclosed systems and methods may also facilitate the migration of frequent and highly used databases that support online applications. For example, online applications related to e-commerce often require frequent and smooth access to pricing databases. These types of databases handle large sets of information, are dynamically updated or modified, and have little or no tolerance for downtime. Especially in view of recent practices, such as dynamic pricing, fast and reliable access to databases is critical to business operations. The migration of these types of pricing databases is therefore complex and critical. The disclosed systems and methods allow databases to be efficiently migrated with minimal downtime and using tools for clustering, verification, and replication, which minimizes data loss and/or reduces data corruption.

Reference will now be made to the disclosed embodiments, examples of which are illustrated in the accompanying drawings.

FIG1A shows a schematic block diagram of a system 100 of an exemplary embodiment of a system including a computerized system for implementing communications for shipping, transportation, and logistics operations. As shown in FIG1A , the system 100 may include each system, wherein each system may be connected to each other via one or more networks. The systems may also be connected to each other via direct connections (e.g., using cables). The systems described include: Shipment Authority Technology (SAT) system 101, external front-end system 103, internal front-end system 105, transportation system 107, mobile devices 107A, 107B, and 107C, seller portal 109, Shipment and Order Tracking (SOT) system 111, Fulfillment Optimization (FO) system 113, Fulfillment Messaging Gateway (FMG) 115, Supply Chain Management (SCM) system 117, workforce management system 119, mobile devices 119A, 119B, and 119C (depicted as being in a fulfillment center). Center, FC) 200), third-party fulfillment system 121A, third-party fulfillment system 121B and third-party fulfillment system 121C, fulfillment center authorization system (Fulfillment Center Authorization System, FC Auth) 123 and labor management system (Labor Management System, LMS) 125.

In some embodiments, the SAT system 101 can be implemented as a computer system that monitors order status and delivery status. The SAT system 101 can also monitor data, including output (e.g., the number of packages shipped in a specific time period) and input (e.g., the number of empty cardboard boxes received for use in shipments). The SAT system 101 can also act as a gateway between different devices in the system 100, enabling communication between devices (e.g., using store-and-forward or other techniques) (e.g., using store-and-forward or other techniques).

In some embodiments, the external front-end system 103 may be implemented as a computer system that enables external users to interact with one or more systems in the system 100. For example, in an embodiment where the system 100 enables the presentation of the system to allow users to place orders for items, the external front-end system 103 may be implemented as a web server that receives search requests, presents item pages, and requests payment information. For example, the external front-end system 103 may be implemented as a computer or multiple computers running software (e.g., Apache HTTP Server, Microsoft Internet Information Service (IIS), NGINX, etc.). In other embodiments, the external front-end system 103 may run customized web server software designed to receive and process requests from external devices (e.g., mobile device 102A or computer 102B), retrieve information from databases and other data stores based on those requests, and provide responses to the received requests based on the retrieved information.

In some embodiments, the external front-end system 103 may include one or more of a web cache system, a database, a search system, or a payment system. In one aspect, the external front-end system 103 may include one or more of these systems, while in another aspect, the external front-end system 103 may include an interface (e.g., server-to-server, database-to-database, or other network connection) connected to one or more of these systems.

A set of illustrative steps shown by FIG. 1B , FIG. 1C , FIG. 1D , and FIG. 1E will help describe some operations of the external front-end system 103. The external front-end system 103 can receive information from a system or device in the system 100 for presentation and/or display. For example, the external front-end system 103 can host or provide one or more web pages, including: a search result page (Search Result Page, SRP) (e.g., FIG. 1B ), a single display page (Single Display Page, SDP) (e.g., FIG. 1C ), a shopping cart page (e.g., FIG. 1D ) or an order page (e.g., FIG. 1E ). A user device (e.g., using a mobile device 102A or a computer 102B) can navigate to the external front-end system 103 and request a search by entering information into a search box. The external front-end system 103 may request information from one or more systems in the system 100. For example, the external front-end system 103 may request information from the FO system 113 to satisfy a search request.

The external front-end system 103 may prepare an SRP based on the information (e.g., FIG. 1B ). The SRP may include information that satisfies the search request. For example, this may include pictures of products that satisfy the search request. The SRP may also include the respective prices of each product, or information related to enhanced delivery options for each product, promised delivery date (PDD), weight, size, quote, discount, etc. In some embodiments, the SRP may also include delivery options, deadlines for delivery options, and/or hypermedia elements requesting user input. The external front-end system 103 may send the SRP to the requesting user device (e.g., via a network).

The user device may then select a product from the SRP, e.g., by clicking or tapping on the user interface, or using another input device, to select a product presented on the SRP. The user device may formulate an information request for the selected product and send the request to the external front-end system 103. In response, the external front-end system 103 may request information related to the selected product. For example, the information may include additional information beyond the information presented for the product on the respective SRP. This may include, for example, a shelf life, country of origin, weight, dimensions, number of items in a package, instructions for use, a deadline for first delivery or first delivery, or other information about the product. The information may also include recommendations for similar products (e.g., based on big data and/or machine learning analysis of customers who purchased the product and at least one other product), answers to frequently asked questions, reviews from customers, manufacturer information, pictures, etc.

The external front-end system 103 may prepare a single display page (SDP) (e.g., FIG. 1C ) based on the received product information, the location of the customer's device, and the availability of delivery options. The SDP may also include other interactive elements, such as a "Buy Now" button, an "Add to Cart" button, a quantity field, a picture of the item, etc. The SDP may also include a list of sellers that offer the product. The list may be sorted in order based on the price offered by each seller, so that the seller offering the product at the lowest price may be listed at the top. The list may also be sorted based on the seller ranking, so that the highest ranked seller may be listed at the top. The seller ranking may be formulated based on multiple factors, including, for example, past tracking records of the seller meeting the promised PDD. The external front-end system 103 may deliver the SDP to the requesting user device (eg, via a network).

The requesting user device may receive an SDP listing product information. After receiving the SDP, the user device may then interact with the SDP. For example, a user of the requesting user device may click or otherwise interact with a "Place in Cart" button on the SDP. This adds the product to a shopping cart associated with the user. Alternatively or in addition, the user may interact with the SDP by providing delivery instructions. The user device may send the request to the external front-end system 103 to add the product to the shopping cart.

The external front-end system 103 may generate a shopping cart page (e.g., FIG. 1D ). In some embodiments, the shopping cart page lists products that the user has added to a virtual “shopping cart.” The user device may request the shopping cart page by clicking on an icon on the SRP, SDP, or other page or otherwise interacting with an icon on the SRP, SDP, or other page. In some embodiments, the shopping cart page may list all products that the user has added to the shopping cart, as well as information about the products in the shopping cart, such as the quantity of each product, the price per unit of each product, the price of each product based on the associated quantity, information about the promised delivery date (PDD), the delivery method, shipping costs, user interface elements for modifying products in the shopping cart (e.g., deleting or modifying the quantity), options for ordering additional products or setting up recurring deliveries of products, options for setting up interest payments, user interface elements for continuing with a purchase, and the like. A user at a user device may click on a user interface element (e.g., a button that reads "Buy Now") or otherwise interact with a user interface element to initiate the purchase of the products in the shopping cart. After doing so, the user device may send the request to the external front-end system 103 to initiate the purchase. In some embodiments, the shopping cart page may include a text box input, an interactive icon, or a recommendation message delivered for each product.

The external front-end system 103 may generate an order page (e.g., FIG. 1E ) in response to receiving a request to initiate a purchase. In some embodiments, the order page re-lists the items from the shopping cart and requests entry of payment and shipping information. For example, the order page may include a portion requesting information about the purchaser of the items in the shopping cart (e.g., name, address, email address, phone number), information about the recipient (e.g., name, address, phone number, delivery information), shipping information (e.g., speed/method of delivery and/or pickup), payment information (e.g., credit card, bank transfer, check, prepaid card), and a user interface element requesting a cash receipt (e.g., for tax purposes), etc. The external front-end system 103 may send the order page to the user device.

The user device may enter information on the order page and click or otherwise interact with user interface elements that send information to the external front end system 103. From there, the external front end system 103 may send information to different systems in the system 100 to initiate the creation and processing of a new order with the products in the shopping cart. In some embodiments, the external front end system 103 may also be further configured to enable the seller to send and receive information related to the order.

In some embodiments, the internal front-end system 105 may be implemented as a computer system that enables internal users (e.g., employees of an organization that owns, operates, or leases the system 100) to interact with one or more systems in the system 100. For example, in an embodiment where the SAT system 101 enables the presentation of the system to allow users to place orders for items, the internal front-end system 105 may be implemented as a web server that enables internal users to: view diagnostic and statistical information about orders, modify item information, or review statistics related to orders. For example, the internal front-end system 105 may be implemented as a computer or multiple computers running software (e.g., Apache HTTP Server, Microsoft Internet Information Services (IIS), NGINX, etc.). In other embodiments, the internal front-end system 105 may run customized web server software that is designed to receive and process requests from systems or devices described in system 100 (as well as other devices not shown), obtain information from databases and other data stores based on those requests, and provide responses to the received requests based on the obtained information.

In some embodiments, the internal front-end system 105 may include one or more of a web cache system, a database, a search system, a payment system, an analysis system, an order monitoring system, etc. In one aspect, the internal front-end system 105 may include one or more of these systems, while in another aspect, the internal front-end system 105 may include an interface (e.g., server-to-server, database-to-database, or other network connection) connected to one or more of these systems.

In some embodiments, the transport system 107 may be implemented as a computer system capable of communicating between the systems or devices of the system 100 and the mobile devices 107A-107C. In some embodiments, the transport system 107 may receive information from one or more mobile devices 107A-107C (e.g., mobile phones, smart phones, PDAs, etc.). For example, in some embodiments, the mobile devices 107A-107C may include devices operated by delivery workers. Delivery workers (who may be permanent, temporary, or shift employees) may utilize mobile devices 107A-107C to implement the delivery of packages including products ordered by users. For example, in order to deliver a package, the delivery worker may receive a notification on the mobile device indicating which package will be delivered and where it will be delivered. Upon arriving at the delivery location, the delivery worker may locate the package (e.g., in the back of a truck or in the package's carton), use the mobile device to scan or otherwise capture data associated with an identifier on the package (e.g., a barcode, image, text string, RFID tag, etc.), and deliver the package (e.g., by leaving the package at the front door, handing the package to security, delivering the package to the recipient, etc.). In some embodiments, the delivery worker using the mobile device may capture a photo of the package and/or may obtain a signature. The mobile device may transmit information including information about the delivery (including, for example, time, date, GPS location, photo, identifier associated with the delivery worker, identifier associated with the mobile device, etc.) to the transportation system 107. The transport system 107 may store the information in a database (not shown) for access by other systems in the system 100. In some embodiments, the transport system 107 may use the information to prepare and send tracking data to other systems, the tracking data indicating the location of a particular package.

In some embodiments, certain users may use one type of mobile device (e.g., a permanent worker may use a specialized PDA with customized hardware (e.g., a barcode scanner, stylus, and other devices)) while other users may use other types of mobile devices (e.g., temporary or shift workers may utilize off-the-shelf cell phones and/or smartphones).

In some embodiments, the transportation system 107 may associate a user with each device. For example, the transportation system 107 may store an association between a user (represented by, for example, a user ID, an employee ID, or a phone number) and a mobile device (represented by, for example, an International Mobile Equipment Identity (IMEI), an International Mobile Subscription Identifier (IMSI), a phone number, a Universal Unique Identifier (UUID), or a Globally Unique Identifier (GUID)). The transportation system 107 may use the association in conjunction with data received during delivery to analyze data stored in a database to determine, among other things, the location of a worker, the efficiency of a worker, or the speed of a worker.

In some embodiments, the seller portal 109 may be implemented as a computer system that enables a seller or other external entity to electronically communicate with one or more systems in the system 100. For example, a seller may utilize a computer system (not shown) to upload or provide product information, order information, contact information, etc. for products that the seller wishes to sell through the system 100 using the seller portal 109.

In some embodiments, the shipping and order tracking system 111 may be implemented as a computer system that receives, stores, and forwards information about the location of packages that include products ordered by customers (e.g., by users using devices 102A-102B). In some embodiments, the shipping and order tracking system 111 may request or store information from a web server (not shown) operated by a shipping company that delivers packages that include products ordered by customers.

In some embodiments, the shipping and order tracking system 111 can request and store information from the systems described in the system 100. For example, the shipping and order tracking system 111 can request information from the transportation system 107. As described above, the transportation system 107 can receive information from one or more mobile devices 107A-107C (e.g., cell phones, smartphones, PDAs, etc.) associated with one or more users (e.g., delivery workers) or vehicles (e.g., delivery trucks). In some embodiments, the shipping and order tracking system 111 can also request information from a workforce management system (WMS) 119 to determine the location of various products within a fulfillment center (e.g., fulfillment center 200). The shipping and order tracking system 111 may request data from one or more of the transportation systems 107 or the WMS 119, process the data, and present the data to devices (e.g., user device 102A and user device 102B) based on the request.

In some embodiments, the fulfillment optimization (FO) system 113 may be implemented as a computer system that stores information about customer orders from other systems (e.g., the external front-end system 103 and/or the shipping and order tracking system 111). The FO system 113 may also store information describing where specific items are held or stored. For example, certain items may be stored in only one fulfillment center, while certain other items may be stored in multiple fulfillment centers. In other embodiments, certain fulfillment centers may be designed to store only a specific group of items (e.g., fresh produce or frozen produce). The FO system 113 stores this information as well as associated information (e.g., quantity, size, receipt date, expiration date, etc.).

In some embodiments, the fulfillment messaging gateway (FMG) 115 may be implemented as a computer system that receives requests or responses from one or more systems in the system 100 (e.g., the FO system 113) in one format or protocol, converts the requests or responses to another format or protocol, and forwards the requests or responses to other systems (e.g., the WMS 119 or third-party fulfillment systems 121A, 121B, or 121C) in the converted format or protocol, or vice versa.

In some embodiments, the supply chain management (SCM) system 117 may be implemented as a computer system that performs forecasting functions. For example, the SCM system 117 may forecast the demand level for a specific product based on, for example, past demand for the product, expected demand for the product, past demand across the entire network, expected demand across the entire network, counted products stored in each fulfillment center 200, expected orders or current orders for each product, etc. In response to the forecasted level and the quantity of each product across all fulfillment centers, the SCM system 117 may generate one or more purchase orders to purchase and restock sufficient quantities to meet the forecasted demand for the specific product.

In some embodiments, a workforce management system (WMS) 119 may be implemented as a computer system that monitors a workflow. For example, the WMS 119 may receive event data indicating discrete events from various devices (e.g., devices 107A-107C or devices 119A-119C). For example, the WMS 119 may receive event data indicating that a package was scanned using one of these devices. As discussed below with respect to the fulfillment center 200 and FIG. 2 , during the fulfillment process, a package identifier (e.g., a barcode or RFID tag data) may be scanned or read by a machine (e.g., an automated or handheld barcode scanner, an RFID reader, a high-speed camera, a device (e.g., a tablet 119A), a mobile device/PDA 119B, a computer 119C, etc.) at a particular stage. The WMS 119 may store each event indicating a scan or read of a package identifier in a corresponding database (not shown) along with the package identifier, time, date, location, user identifier, or other information, and may provide the information to other systems (e.g., the shipping and order tracking system 111).

In some embodiments, the WMS 119 may store information associating one or more devices (e.g., devices 107A-107C or devices 119A-119C) with one or more users associated with the system 100. For example, in some cases, a user (e.g., a part-time or full-time employee) may be associated with a mobile device because the user owns the mobile device (e.g., the mobile device is a smartphone). In other cases, a user may be associated with a mobile device because the user temporarily keeps the mobile device (e.g., the user checks out the mobile device at the beginning of the day, will use it during the day, and will return it at the end of the day).

In some embodiments, WMS 119 may maintain a work log for each user associated with system 100. For example, WMS 119 may store information associated with each employee, including any specified process (e.g., unloading a truck, picking items from a picking area, rebin wall work, packaging items), user identification, location (e.g., floor or zone in fulfillment center 200), number of cells moved through the system by the employee (e.g., number of items picked, number of items packaged), identification associated with equipment (e.g., equipment 119A-119C), etc. In some embodiments, WMS 119 may receive check-in and check-out information from a timekeeping system (e.g., a timekeeping system operating on equipment 119A-119C).

In some embodiments, third-party fulfillment (3PL) systems 121A-121C represent computer systems associated with third-party suppliers of logistics and products. For example, while some products are stored in fulfillment center 200 (as discussed below with reference to FIG. 2 ), other products may be stored off-site, may be produced on demand, or may not be originally stored in fulfillment center 200. 3PL systems 121A-121C may be configured to receive orders from FO system 113 (e.g., via FMG 115) and may provide products and/or services directly to customers (e.g., delivery or installation). In some embodiments, one or more of the 3PL systems 121A-121C may be part of the system 100, while in other embodiments, one or more of the 3PL systems 121A-121C may be external to the system 100 (e.g., owned or operated by a third-party provider).

In some embodiments, fulfillment center authorization system (FC Auth) 123 may be implemented as a computer system having various functions. For example, in some embodiments, FC Auth 123 may act as a single sign-on (SSO) service for one or more other systems in system 100. For example, FC Auth 123 may enable a user to log in via internal front-end system 105, determine that the user has similar privileges to access resources at shipping and order tracking system 111, and enable the user to access these privileges without a second login process. In other embodiments, FC Auth 123 may enable users (e.g., employees) to associate themselves with specific tasks. For example, some employees may not have electronic devices (e.g., devices 119A-119C) but may instead move from task to task and zone to zone over the course of a day within fulfillment center 200. FC Auth 123 may be configured to enable these employees to indicate what task they are performing and what zone they are in at different times of the day.

In some embodiments, the labor management system (LMS) 125 may be implemented as a computer system that stores attendance and overtime information for employees (including full-time and part-time employees). For example, the LMS 125 may receive information from the FC Auth 123, the WMA 119, the devices 119A-119C, the transportation system 107, and/or the devices 107A-107C.

The specific configurations described in FIG. 1A are merely examples. For example, although FIG. 1A depicts an FC authorization system 123 connected to an FO system 113, not all embodiments require the specific configuration. In practice, in some embodiments, the systems in system 100 may be connected to each other via one or more public or private networks, including the Internet, an intranet, a WAN (wide area network), a MAN (metropolitan area network), a wireless network compliant with IEEE 802.11a/b/g/n standards, a leased line, etc. In some embodiments, one or more systems in system 100 may be implemented as one or more virtual servers implemented in a data center, a server farm, etc.

FIG. 2 depicts a fulfillment center 200. A fulfillment center 200 is an example of a physical location where items are stored for shipment to customers when they are ordered. A fulfillment center (FC) 200 may be divided into a plurality of zones, each of which is depicted in FIG. 2. In some embodiments, these "zones" may be thought of as virtual divisions between different stages of the process of receiving items, storing items, retrieving items, and shipping items. Thus, while "zones" are depicted in FIG. 2, other divisions of zones are possible, and in some embodiments, the zones in FIG. 2 may be omitted, repeated, and/or modified.

Inbound area 203 represents an area of FC 200 that receives items from sellers who wish to sell products using system 100 (FIG. 1A). For example, the seller may deliver items 202A and 202B using truck 201. Item 202A may represent a single item that is large enough to occupy its own shipping pallet, while item 202B may represent a group of items that are stacked together on the same pallet to save space.

In some embodiments, as shown in FIG. 2 , one or more of the portions of the FC 200 may include a positioning sensor 217. The positioning sensor 217 may include a plurality of sensors that may be used to determine the location of a product within the FC and track the movement of the product through the FC. In such embodiments, the positioning sensor 217 may be configured to track the location of a product in the FC and estimate movement between different portions. For example, the positioning sensor 217 may be configured to capture and/or store historical data of the location and time of a product to determine movement between different areas of the FC 200. Other systems may use this information to determine the distance or estimate time between a storage area and a packaging area.

As shown in FIG. 2 , the positioning sensor 217 may include a sensor 217A in the packaging area 211, a sensor 217B in the picking area 209, and a sensor 217C in the unloading area 205. In some embodiments, more sensors may be placed in different areas of the FC 200 for the purpose of tracking and capturing the location of the item FC 200, and improving the accuracy of the estimated delivery or maximizing the availability of delivery options. In some embodiments, the positioning sensor 217 may include an optical sensor, such as a camera. In other embodiments, the positioning sensor 217 may include a wireless sensor, such as a radio frequency, Bluetooth or WiFi sensor. Additionally or alternatively, the positioning sensor 217 may include a weight sensor.

The worker will receive the items in the inbound area 203 and can optionally check the items for damage and correctness using a computer system (not shown). For example, the worker can use the computer system to compare the quantity of items 202A and 202B with the ordered quantity of the items. If the quantity does not match, the worker can reject one or more of the items 202A or 202B. If the quantity matches, the worker can move these items to the buffer area 205 (using, for example, a dolly, a hand truck, a forklift, or manually). For example, the buffer area 205 can be a temporary storage area for items that are not currently needed in the picking area because there is a sufficiently high quantity of the items in the picking area to meet the predicted demand. In some embodiments, the forklift 206 operates to move items around the buffer zone 205 and between the inbound zone 203 and the unloading zone 207. If the item 202A or 202B is needed in the picking zone (e.g., due to predicted demand), the forklift can move the item 202A or 202B to the unloading zone 207.

The unloading area 207 may be an area of the FC 200 where items are stored before they are moved to the picking area 209. A worker assigned to the picking task (a "picker") may approach the items 202A and 202B in the picking area, scan the barcode of the picking area using a mobile device (e.g., device 119B), and scan the barcodes associated with the items 202A and 202B. This event may update the real-time location system, which updates the database to indicate that the items have been moved into the FC. The picker may then bring the items to the picking area 209 (e.g., by placing the items on a shopping cart or carrying the items), and the real-time location system may request a storage location for the new items.

The picking area 209 may be an area of the FC 200 where the items 208 are stored on the storage unit 210. In some embodiments, the storage unit 210 may include one or more of physical shelves, shelves, boxes, totes, refrigerators, freezers, cold storage, etc. In some embodiments, the picking area 209 may be organized into multiple levels. In some embodiments, a worker or machine may move items into the picking area 209 in a variety of ways, including, for example, a forklift, an elevator, a conveyor belt, a cart, a hand truck, a dolly, an automatic robot or device, or manually. For example, a picker may place the items 202A and 202B on a hand truck or cart in the unloading area 207 and walk to load the items 202A and 202B to the picking area 209.

The picker may receive instructions to place (or "stack") an item at a specific point in the picking area 209 (e.g., a specific space on the storage unit 210). For example, the picker may scan the item 202A using a mobile device (e.g., device 119B). The device (e.g., using a system that indicates aisles, shelves, and locations) may instruct the picker where the item 202A should be stacked. In some embodiments, the location for stacking the item 202A may be determined based on a predictive algorithm that attempts to maximize the availability of a particular delivery option (e.g., dawn delivery). The device may then prompt the picker to scan a barcode at the location before stacking the item 202A at the location. Alternatively, a wireless sensor or camera coupled with image recognition may store the location over time. In some embodiments, the device may send data (e.g., via a wireless network) to a computer system (e.g., WMS 119 in FIG. 1A) indicating that a user using device 119B has loaded item 202A at the location.

Once the user places an order, the picker can receive instructions on the device 119B to retrieve one or more items 208 from the storage unit 210. In some embodiments, as further described in conjunction with Figure 11, the picker can receive instructions for stacking products via a placement or storage guide. The picker can retrieve the item 208, scan the barcode on the item 208, and place the item 208 on the conveying mechanism 214. In some embodiments, although the conveying mechanism 214 is shown as a slide, the conveying mechanism can be implemented as one or more of a conveyor belt, an elevator, a cart, a forklift, a hand truck, a dolly, a cart, etc. Then, the item 208 can reach the packaging area 211.

The packing area 211 may be an area of the FC 200 that receives items from the picking area 209 and packages the items into boxes or bags for final shipment to customers. In the packing area 211, a worker assigned to receive items (a "merge worker") will receive the items 208 from the picking area 209 and determine the order to which the items 208 correspond. For example, the merge worker may use a device (e.g., computer 119C) to scan a barcode on the item 208. The computer 119C may visually indicate which order the item 208 is associated with. This may include, for example, a space or "cell" on the wall 216 that corresponds to the order. Once the order is complete (e.g., because the cell includes all items for the order), the merge worker may indicate to the packing worker (or "packer") that the order is complete. The packer can retrieve the items from the unit and place them in boxes or bags for shipping. The packer can then deliver the boxes or bags to the hub 213 (e.g., via a forklift, cart, trolley, hand truck, conveyor, manually, or other means).

Hub 213 may be the area of FC 200 that receives all of the boxes or bags ("parcels") from packaging area 211. Workers and/or machines in hub 213 may retrieve parcels 218 and determine which portion of the delivery area each parcel is intended for, and route the parcels to the appropriate camp area 215. For example, if the delivery area has two smaller sub-areas, the parcels will go to one of the two camp areas 215. In some embodiments, a worker or machine may scan the parcel (e.g., using one of devices 119A-119C) to determine the final destination of the parcel. Routing the parcel to the camp area 215 may include, for example, determining (e.g., based on a zip code) a portion of a geographic area to which the parcel is destined and determining a camp area 215 associated with the portion of the geographic area.

In some embodiments, the camp area 215 may include: one or more buildings, one or more physical spaces, or one or more areas where packages are received from the hub 213 for sorting the packages into routes and/or sub-routes. In some embodiments, the camp area 215 is physically separated from the FC 200, while in other embodiments, the camp area 215 may form part of the FC 200.

Workers and/or machines in camp area 215 may determine which route and/or sub-route a package 220 should be associated with (e.g., based on a comparison of the destination to existing routes and/or sub-routes, a calculation of the workload for each route and/or sub-route, the time of day, the shipping method, the cost of shipping package 220, the PDD associated with the items in package 220, delivery options, etc.). In some embodiments, a worker or machine may scan a package (e.g., using one of devices 119A-119C) to determine the final destination of the package. Once a package 220 is assigned to a particular route and/or sub-route, the worker and/or machine may move the package 220 to be shipped. In exemplary FIG. 2 , camp area 215 includes truck 222, car 226, and delivery workers 224A and 224B. In some embodiments, truck 222 may be driven by delivery worker 224A, where delivery worker 224A is a full-time employee who delivers packages for FC 200, and truck 222 is owned, leased, or operated by the same company that owns, leases, or operates FC 200. In some embodiments, car 226 may be driven by delivery worker 224B, where delivery worker 224B is a "flexible" or temporary worker who delivers on an as-needed basis (e.g., seasonally). Car 226 may be owned, leased, or operated by delivery worker 224B.

FIG. 3 is a schematic block diagram of an exemplary system 300 consistent with the disclosed embodiments. In the system 300, a data migration (DM) system 320 may include servers, computer modules and/or data processing centers configured to manage customer and product data, control database backup processes and/or deploy recovery data. In addition to the DM system 320, the system 300 may include online resources 340, client devices 350, third-party systems 360, FC/warehouse systems 310, and databases 380. In some embodiments, as shown in FIG. 3, components of the system 300 may be connected to a network 370. However, in other embodiments, components of the system 300 may be directly connected to each other without the network 370. For example, the database 380 may be directly coupled to the DM system 320.

In some embodiments, the DM system 320 may be connected to other elements of the system 300 to provide a database migration platform. Thus, in some embodiments, the DM system 320 may have access, storage, and/or curation information for the system 300. For example, in the context of e-commerce content, the DM system 320 may be connected to internal databases, external databases, application programming interfaces, and/or cloud services to manage and coordinate migration processes, perform validation operations, and deploy (or reorganize) data structures. The DM system 320 may communicate with the database via both the external front-end system 103 and the internal front-end system 105. The DM system 320 may also be coupled to the seller portal 109. In some embodiments, the DM system 320 may be configured to handle database operations and, for example, replicate databases and handle database-related traffic. In such embodiments, DM system 320 may be configured to store and deploy the data necessary to generate the SDP of Figure 1C. DM system 320 may also coordinate security protocols and permissions for accessing or modifying data.

The DM system 320 may include a controller 323 that may be coupled to tools and databases in the system 300. The controller 323 may also be configured to generate data clusters, store data, verify data, and perform catch-up operations. The controller 323 may also manage dual writes, replication, and/or replication of databases. For example, as further discussed below in conjunction with Figures 4 and 5, the controller 323 may perform operations to deploy a database solution to production, trigger an application to write data in a new database, write to a new database while keeping the source database as primary, and perform dual write operations.

In addition, in some embodiments, the controller 323 may also perform encryption and decryption functions for data. For example, the controller 323 may implement an encryption and decryption engine that can be used to encrypt and decrypt database copies or snapshots when they are transferred. The controller 323 may also encrypt traffic pipes to prevent them from being corrupted during the transfer. In some embodiments, to improve the security of the transfer process, the controller 323 may include an Advanced Encryption Standard (AES) engine or core for 128-bit, 192-bit, and/or 256-bit encryption. In some embodiments, controller 323 may include one or more known processing devices, such as, but not limited to, microprocessors from the ^{Pentium™ or Xeon™ families} manufactured ^by Intel ^™ , ^the Turion ^™ family manufactured ^{by AMD™ ,} or any of a variety of processors from other manufacturers. However, in other embodiments, controller 323 may be multiple devices coupled and configured to perform functions consistent with the present disclosure. In some embodiments, controller 323 may be coupled to one or more databases and grouping systems, as further discussed below with respect to FIG. 4.

In addition, the DM system 320 may include ^a regional database 325 ^{that stores and manages certain information of the system 300. For example, the regional database 325 may include, for example, an Oracle TM database, a Sybase TM database, or other relational databases or non-relational databases such as Hadoop TM sequential files , HBase TM} , or ^{Cassandra TM .} In some embodiments, the regional database 325 may include a computing component (e.g., a database management system, a database server, etc.) configured to receive and process requests for data stored in a memory device of the database and provide data from the database. In other embodiments, the controller 322 may control the operation of the regional database 325 and the regional database 325 may be a storage unit. In some embodiments, the regional database 325 may include a NoSQL database such as HBase, MongoDB ^™ , or Cassandra ^™ . In addition, the regional database 325 may include a relational database such as Oracle, MySQL, and Microsoft SQL Server.

The regional database 325 may store data that may be used by other components of the system 300. For example, the regional database 325 may store data that may be requested by the client device 350 or the online resource 340. The data stored in the regional database 325 may include any suitable data associated with the delivery order and/or the deliverer, which may be used to select a pricing algorithm and determine and/or adjust payments to the deliverer. The data stored in the regional database 325 may also include financial information about the user, user credentials, and/or user preferences. Although FIG. 3 shows the regional database 325 within the DM system 320, in some embodiments, the regional database 325 may be located separately from the DM system 320 (virtually or physically). For example, in some embodiments, the regional database 325 can be an external storage unit that can be connected or disconnected by a user. In some embodiments, the regional database 325 can act as a replica database during a migration process performed by the controller 323. For example, the regional database 325 can be configured to replicate one or more of the external databases 380 during the migration process. In addition, the regional database 325 can be configured as a cache database to capture chase data, for example, during data migration.

In some embodiments, the DM system 320 may be implemented with one or more of the components of the system 100 ( FIG. 1A ). For example, the system 320 may be implemented as part of the internal front-end system 105 , the FO system 113 , the SCM system 117 , the SAT system 101 , and/or the WMS 119 ( FIG. 1A ). In other embodiments, the DM system 320 may be implemented with a database server having a client-server model for database access. In such embodiments, the DM system 320 may manage information about order data, click data, and/or product data collected via the external front-end system 103 ( FIG. 1A ). Alternatively or in addition, the DM system 320 may be implemented as an embedded database within a specific application. For example, the DM system 320 may be embedded as part of the SCM 117 ( FIG. 1A ). In some embodiments, the DM system 320 may control access to the regional database 325 via front-end/back-end assignment of tasks, where the front-end runs on the client device 350 - which displays the requested data - and the back-end runs on the server and handles tasks such as data analysis and storage.

In some embodiments, the DM system 320 can request and provide resources via an application programming interface (API) that responds to a query language. The API for the DM system 320 specifies filtering conditions and parameters. The API can also provide authentication methods and pass or share credentials. Via the API, the DM system 320 can allow the client device 350 (or other components of the system 300) to request data transfer. Via the API, the DM system 320 can also expose the transferred database and/or request information from the external database. In addition, via the API, the DM system 320 can read the database after accessing it via a known credential. The DM system 320 may respond to the API request by extracting the record from the relevant table in the database, wrapping and tagging it, and returning the file to the program as a return parameter or sending it to the address passed in the parameter.

DM system 320 can use dedicated database applications such as Oracle, DB2, Informix, and Microsoft SQL Server to implement data management. Alternatively or in addition, DM system 320 can be implemented using free software database applications including PostgreSQL; and implemented according to the GNU General Public License.

In addition to the DM system 320, the system 300 may include online resources 340, client devices 350, third-party systems 360, FC systems 310, and databases 380. In some embodiments, as shown in FIG3 , the components of the system 300 may be connected to a network 370. However, in other embodiments, the components of the system 300 may be directly connected to each other without the network 370. For example, the database 380 may be directly coupled to the DM system 320.

Online resources 340 may include one or more servers or storage services provided by an entity, such as a provider of web hosting, network connectivity, cloud, or backup services. In some embodiments, online resources 340 may be associated with a hosting service or server that stores web pages for authentication services, Domain Name System (DNS), or login pages. In other embodiments, online resources 340 may be associated with cloud computing services. In still other embodiments, online resources 340 may be associated with messaging services, such as Apple Push Notification Service, Azure Mobile Services, or Google Cloud Messaging. In such embodiments, online resources 340 may handle the delivery of messages and notifications related to the functionality of the disclosed embodiments.

In addition, the online resources 340 may include a redirection server that can be configured by the DM system 320 to switch online traffic between different elements of the system 300. In addition, the online resources 340 may include cloud services for retrieving data from a database, replicating data, or clustering database space. In some embodiments, the online resources 340 may be programmable by the DM system 320. For example, a redirection server in the online resources 340 may be programmed by the DM system 320 to redirect database queries before, during, and after a database migration. Alternatively or in addition, the online resources 340 may be configured for encryption, decryption, and/or authentication processing. As further discussed in conjunction with, for example, FIG. 11, the disclosed systems and methods may coordinate different operations for database migration and data authentication.

The client device 350 may include one or more computing devices configured to perform one or more operations consistent with the disclosed embodiments. For example, the client device 350 may include a desktop computer, a laptop computer, a server, a mobile device (e.g., a tablet computer, a smart phone, etc.), a set-top box, a gaming device, a portable computing device, or other types of computing devices. In some embodiments, the client device 350 may include the user device 102A/102B (FIG. 1A) and operate as part of the system 100. However, in other embodiments, the client device 350 may be independent of the system 100. The client device 350 may include one or more processors configured to execute software instructions stored in a memory (e.g., a memory in the client device 350) to perform operations that implement the functions described below. For example, the client device 350 may access information in the regional database 325 via the DM system 320. Furthermore, the client device 350 may be configured to perform operations according to instructions transmitted by the DM system 320. In addition, the client device 350 may be configured to trigger operations of the DM system 320. For example, the client device 350 may initiate a database migration and/or monitor the migration status. Furthermore, in some embodiments, the client device 350 may receive a data validation report and/or request a validation report during or after a database migration.

In some embodiments, the client device 350 may be configured for wired and/or wireless communications and may include software that, when executed by the processor, performs Internet-related communications (e.g., TCP/IP) and content display processes. For example, the client device 350 may execute browser software that generates and displays an interface with product information. Thus, the client device 350 may execute an application that allows the client device 350 to communicate with components via the network 370 and display content in the interface via a display device included in the client device 350.

The disclosed embodiments are not limited to any particular configuration of the client device 350. For example, the client device 350 may be a mobile device that stores and executes mobile applications to perform operations that provide the functionality provided by the DM system 320 and/or online resources 340.

Database 380 may include one or more computing devices configured with appropriate software to store, manage, provide, and modify data. Database 380 may include a physical database. In addition, database 380 may include a cloud-based database scenario. Database 380 may include, for example, an Oracle ^™ database, a Sybase ^™ database, or other relational databases or non-relational databases, such as Hidup ^™ Sequential Files, Hibex ^™ , or Cassandra ^™ . Database 380 may include computing components (e.g., a database management system, a database server, etc.) configured to receive and process requests for data stored in a memory device of the database and provide data from the database. Additionally or alternatively, database 380 may include a cloud database built and accessed via a cloud platform. In such embodiments, database 380 may be managed by other elements of system 300. For example, database 380 may be managed by controller 323. In addition, cloud-based database 380 may be configured to support relational databases (including MySQL and PostgreSQL) and NoSQL databases (including MongoDB and Apache CouchDB) and accessed via an API managed by third-party system 360 or via controller 323.

Although database 380 is depicted separately, in some embodiments, database 380 may be included in or otherwise associated with DM system 320 or online resource 340. For example, database 380 may be implemented as a regional database 325 in some embodiments. In some embodiments, database 380 may store information related to the operation of system 300. For example, database 380 may include pricing data and be connected to an online application (e.g., via online resource 340).

The third-party system 360 may include one or more elements of the system 100. For example, the third-party system 360 may include 3PL systems 121A-3PL systems 121C (Figure 1). Additionally or alternatively, the third-party system 360 may include one or more servers or storage services provided by an entity associated with the DM system 320, such as a provider of services or a fulfillment center. The third-party system 360 may also be connected to the system 300 via the network 370, but in other embodiments, the third-party system 360 may include a direct connection to some elements of the system 300. For example, to minimize latency or network congestion, the third-party system 360 may be connected to the DM system 320 in a dedicated network. In addition, the third-party system 360 may be configured to provide and/or request information from the DM system 320 or other elements of the system 300. In some embodiments, third-party system 360 may not be a client of DM system 320, although it may also be coupled to network 370. Rather, third-party system 360 may include a system that includes information of a user or client of DM system 320. For example, third-party system 360 may include a delivery contractor's server that may be used by system 320 when a product delivery involves a third-party contractor. In such embodiments, DM system 320 may be configured to generate a backup of data accessible from third-party system 360.

The FC/warehouse system 310 may include sensors and processors for determining and/or storing the location of products within the FC or warehouse and periodically updating a database (e.g., a regional database 325). For example, the FC/warehouse system 310 may include sensors 217A-sensor 217C (Figure 2). Alternatively or in addition, the FC/warehouse system 310 may include a camera that captures images of the shelves and uses image recognition methods to identify products and determine the location of the products in the FC. In addition, the FC/warehouse system 310 may be coupled to a scanning device and track the location of the product in the FC by monitoring scanning events of the product. In addition, the FC/warehouse system 310 may communicate with the DM system 320 to provide information that facilitates estimating available inventory. In some embodiments, the FC/warehouse system 310 may be configured to provide periodic updates of available inventory to the DM system 320. For example, the FC/warehouse system 310 may be configured to send available inventory to the DM system 320 at the end of each day.

The network 370 can be any type of network configured to provide communication between the components of the system 300. For example, the network 370 can be any type of network (including infrastructure) that provides communication, exchanges information, and/or facilitates information exchange, such as the Internet, a local area network, near field communication (NFC), or other suitable connections that enable information to be sent and received between the components of the system 300. In some embodiments, the network 370 can be wired, wireless, or a combination. In other embodiments, one or more components of the system 300 can communicate directly via a designated communication link. In still other embodiments, the network 370 can include multiple networks, thereby organizing, for example, one or more networks.

It should be understood that the configuration and boundaries of the functional building blocks of the system 300 have been defined herein for ease of description. Alternative boundaries may be defined so long as the specified functions and their relationships are properly performed. Alternatives (including equivalents, extensions, variations, deviations, etc. of the alternatives described herein) will be apparent. Such alternatives are within the scope of the disclosed embodiments.

FIG4 is a block diagram of an exemplary database migration system 400 consistent with disclosed embodiments. In some embodiments, migration system 400 may be part of system 300. For example, controller 323 in system 400 may be the same controller discussed as part of DM system 320 ( FIG3 ). However, in other embodiments, migration system 400 may be independent of system 300.

The migration system 400 may include a controller 323. As shown in FIG4, the controller 323 may be coupled to a replica database (replica DB) 406 and a database clustering system (DB clustering system) 420. In addition, the system 400 may include a migration tool 402 and an online database (online DB) 404.

In some embodiments, the controller 323 may be part of the DM system 320. However, in other embodiments, the controller 323 may be implemented as a separate component from the DM system 320. The controller 323 may include a series of tools for database migration. These tools may be cloud-based tools or local applications. The tools may include a data expert tool 412, a data loader tool 414, a validation tool 416, and a tracking tool 418.

The data export tool 412 may include software and/or hardware for exporting data from one or more databases. For example, the data export tool 412 may be configurable to export data from the online DB 404 and/or the replica DB 406. The data export tool 412 may export data stored in a distributed SQL database platform or a CSV data file and may be used to make a logical full backup or export. The data export tool 412 may be configured via an interface of the controller 323 and may be used to export data in a database to a SQL file, to a CSV file, or other export-type file.

In some embodiments, the data export tool 412 may also be configured to filter data that is not exported. For example, the data export tool 412 may export the tables of the entire database by default. The data export tool 412 may be configurable via the interface of the controller 323 to exclude certain tables in the database, such as tables in the system database. Additionally or alternatively, the data export tool 412 may perform filtering operations based on metadata in the table or keyword, identifier, or tags in the data itself. For example, the data export tool 412 may filter a particular database or table by specifying a table filter.

In some embodiments, the data export tool 412 can perform export operations via parallel execution. For example, the data export tool 412 can be configured to perform parallel operations when transferring or exporting data to the DB clustering system 420. In these embodiments, the data export tool 412 can be configured to specify the maximum number of records of a single file (or the number of columns in the database) and enable the parallelization of tables to improve the speed of exporting larger tables. In addition or alternatively, the data export tool 412 can verify consistency during data operations. For example, the data export tool 412 can be configurable to control the way in which the data is exported for consistency assurance. In these embodiments, the data export tool 412 can obtain a snapshot of a specific timestamp by default (that is, a -consistency snapshot), and then use the snapshot for consistency. In such embodiments, the data export tool 412 may use certain parameters in the snapshot to specify the timestamp to be backed up.

The data loader tool 414 may include software and/or hardware for importing data into one or more databases. For example, via the data loader tool 414, the controller 323 may load data into the DB clustering system 420. The data loader tool 414 may be configured as a tool for multi-threaded loading of data and backup. The data loader tool 414 may operate by connecting to a host server or database, identifying the privileges required to write and/or load data, and initiating data transfer. For example, the data loader tool 414 may open a port and establish a connection to a domain socket file to subsequently deliver the data to a specific location in the database and preserve certain structures. Other data loader tools 414 may be configurable to perform one or more data corrections. For example, the data loader tool 414 may not copy table names, deconstruct certain portions of the data, and/or overwrite data that may have been corrupted. Additionally, in some embodiments, the data loader tool 414 may generate an output file reporting the exported data, the data structure, and any modifications. Additionally or alternatively, the data loader tool 414 may be configured to transmit metadata information in the tables of the database and metadata for the data load, including, for example, the start and end times of the load and the location of the main binary log, separate files, table schemas, and binary logs.

The validation tool 416 may include hardware and/or software for validating data before, during, and/or after the database migration process. The validation tool 416 may be configured for data validation across different environments. In some embodiments, the validation tool 416 may include a framework to connect to a large number of online data sources that assist in the validation process. The validation tool 416 may perform multi-level data validation functions from the table level to the column level. For example, the validation tool 416 may be configurable to perform table-level validation, perform operations such as table column counts, grouping by column counts, row aggregation, random filtering, and restrictions. Alternatively or in addition, validation may be configurable to perform row, column, or SQL level validation. For example, a validation tool may perform operations to validate schema/row data types, perform hash comparisons, and/or run custom queries against different data sources.

In some embodiments, the verification tool 416 can be configured via the interface of the controller 323 to perform parallel operations. For example, the verification tool 416 can be connected to both the source and target databases to run range or block based verification. During this preliminary or parallel verification, the verification tool 416 can verify based on counting the number of rows in the source table and verifying that it matches the target table. Additionally or alternatively, the verification tool 416 can perform parallel verification via specific rows based on tags or custom verification.

The catch-up tool 418 may include software and/or hardware for correcting data gaps between the target database and the source database. The catch-up tool 418 may be programmed via the interface of the controller 323 and perform operations for dual writes in different databases, create caches or temporary databases to store refresh and/or gap data, and perform required iterations to confirm verification. In some embodiments, the catch-up tool 418 may use an incremental copy method, in which modified data is periodically copied from the source to the target environment using multiple passes of the source data. In such embodiments, the catch-up tool 418 may turn on dual write to write the incremental data in the target environment, create a new database block or instance to capture the incremental data, and run validation and catch-up operations to transfer the incremental data from the source to the target environment. In such embodiments, the catch-up tool 418 may transfer the incremental changes to the data processed by each subsequent channel. And in some embodiments, the catch-up tool may destroy the incremental data in the cluster for faster validation and placement in the target database.

In some embodiments, the catch-up tool 418 may additionally or alternatively apply a dual write method to store and modify two databases simultaneously. In some embodiments, the catch-up tool 418 may perform certain tests before initiating the dual write operation. For example, when applying the database solution to the DB clustering system 420, the controller 323 may be integrated with the catch-up tool 418 and then use the verification tool 416 to confirm that the dual write operation applies the same data structure and writes data simultaneously.

Additionally, in some embodiments, the catch-up tool 418 may perform concurrent monitoring of modifications or entries in a source or original database so that it does not need to periodically compare data, but identifies gap data as it is written in the source database. In such embodiments, the catch-up tool 418 may be coupled to a stream of event information from the source data, which allows the tool to monitor activity and scan for changes in the stored information. In such embodiments, the catch-up tool 418 not only identifies change gap data, but also places a continuous stream of change notifications with scanned data that has undergone changes to subsequently determine whether it has been captured by a double write operation. In such embodiments, the chase tool 418 may enable data migration to be performed even as the data set undergoes in-flight changes.

As shown in FIG. 4 , in some embodiments, the controller 323 (and the tools in the controller 323 ) may be connected to a DB clustering system 420 . The DB clustering system 420 may include a database clustering system that stores and organizes data via common shards. In some embodiments, the DB clustering system 420 may seal the shard routing logic. In such embodiments, the DB clustering system 420 may allow database queries that distribute application code and data across multiple shards. For example, the DB clustering system 420 may split and merge shards as data is entered into the database. In such embodiments, the DB clustering system 420 may select a predefined cluster size and place the data in different shards based on the data size. In addition, in some embodiments, DB clustering system 420 may include a database clustering system that is exposed or can be exposed to query routing via an application programming interface (API). In this way, DB clustering system 420 can be a migration database that handles online queries after migration. In addition, DB clustering system 420 can be connected to online resources. For example, DB clustering system 420 can be connected to online resources 340.

In some embodiments, DB clustering system 420 can provide a solution for the rapid transmission of data based on real-time data with high traffic and large data sets. For example, DB clustering system 420 can deploy, size, and manage large clusters of database situations. DB clustering system 420 can manage various types of databases including MySQL, Percona, and MariaDB databases. Other DB clustering systems 420 can be architected to effectively work on public or private cloud architectures such as on designated hardware. In some embodiments, DB clustering system 420 can be configured to share its information while keeping application changes to a minimum. For example, when loading information in DB clustering system 420, one or more clusters can be set to share with external sources or internal sources. The DB clustering system 420 can also communicate with cloud-based databases and deploy individual database instances that can be managed with independently structured files.

In some embodiments, DB clustering system 420 may include several server processing programs, command line utilities, and web-based utilities supported by a consistent metadata store. For example, during the migration process, controller 323 may set DB clustering system 420 as a target database to be used to support online operations. In these embodiments, DB clustering system 420 may use a database application to set a database server, and the database application provides database services to other computer programs or to a computer, as defined by a client-server model. In addition, in some embodiments, DB clustering system 420 may serve as a target database or a migration database during the migration process.

As shown in FIG. 4 , in addition to being coupled to the DB clustering system 420, the controller 323 may also be connected to a replica DB 406. The replica DB 406 may include hardware and/or software that generates a new cluster having the same volume and the same data as the original database. The replica DB 406 may be a cloud-based database instance, where its associated data volume is associated with the original database. In some embodiments, the replica DB 406 may be a provisioned copy from a database cluster. Alternatively or in addition, the replica DB 406 may be serverless and provisioned with the same rules as the original database.

The replica DB 406 may have the same behavior as the original database. For example, the replica DB 406 may have the same data structure, database schema, and data layout. In addition, the replica DB 406 may store the same metadata and authentication data.

In embodiments with large data sets, the replica DB 406 may use a copy and write protocol to efficiently copy the data in the replica DB 406. In such embodiments, the replica DB 406 may store the data in pages, in and below the storage volume of the replica DB 406. Other replica DBs 406 may use portions of the same physical storage media as the source portions to copy the data. The replica DB 406 may also adjust the storage required in operation. For example, during the copy of the replica DB 406, the database may determine that the storage volume is equal to the storage volume of the source database. If additional space is needed, the replica DB 406 may generate new pages when copying the data in the new database. In addition, the replica DB 406 may reference the original page for data routing and API responses.

In some embodiments, and as further discussed below in conjunction with FIG. 5 , the controller 323 may generate a replica DB 406 using one or more of the regional databases 325. The controller 323 may generate a replica DB 406 that replicates the original or source database at a new location. Additionally or alternatively, the controller 323 may generate a provisioned replica DB 406 from a provisioned previously provisioned database instance. Furthermore, the replica DB 406 may be generated as an encrypted database to improve security. Furthermore, although the replica DB 406 is illustrated in FIG. 4 as a single database, in some embodiments, the replica DB 406 may include multiple databases that may be associated via one or more management servers.

Online DB 404 can be similar to replica DB 406. However, online DB 404 can be connected to online services. In some embodiments, online DB 404 can be a database system without server requirements. In these embodiments, online DB 404 can be configured to receive and process API calls to request data from online DB 404.

In some embodiments, the online DB 404 may be a cloud-based database. However, in other embodiments, the online DB 404 may be a database connected to an online service. For example, the online DB 404 may be one or more of the regional databases 325 connected to the network 370. In addition, the online DB 404 may be implemented as a special purpose database with a specific capacity range and supporting target applications.

In some embodiments, the online DB 404 and the replica DB 406 may be one or more of the databases 380. In other embodiments, the online DB 404 and the replica DB 406 may be the regional database 325. In still other embodiments, the online DB 404 and the replica DB 406 may be a combination of the database 380 and the regional database 325. For example, while the online DB 404 may be set as one of the databases 380, it is outside the DM system 320 and connected to the network 370 to support online operations, and the replica DB 406 may be set as one of the regional databases 325.

As shown in FIG. 4 , the online DB 404 and the DB clustering system 420 may be connected to a migration tool 402. The migration tool 402 may include hardware and/or software that is connected to the elements in the migration system 400 to interface with the client device and coordinate the database migration process. For example, the migration tool 402 may interface with the client device 350 ( FIG. 3 ) and receive and/or process instructions for database migration. Subsequently, the migration tool 402 may configure the controller 323 or a tool within the controller 323 to operate on the online DB 404, generate a replica DB 406, and migrate the data to the DB clustering system 420. For example, the migration tool 402 may receive instructions to transfer data from one type of database to another type of database, or from a database to another type of data storage (e.g., a data warehouse or a data lake). In such scenarios, the migration tool 402 will configure the controller 323, the DB clustering system 420 to export, load, and then catch up with the gap data so that the data in the online DB 404 is transferred to the DB clustering system 420. The migration tool 402 may also coordinate database replication instructions to facilitate data migration, and as explained above, minimize downtime and improve data accuracy through parallel verification. As further discussed in Figures 5 to 11, the migration system 400 can facilitate the migration of databases via the migration tool 402.

The migration tool 402 may be configured as a migration application that can be accessed by a client or administrator to automate the migration process and configure and execute the migration. In some embodiments, the migration tool 402 may provide a graphical designer or mapping tool. In addition, the migration tool 402 may be equipped with log-based reporting and tools for adjusting migration targets, filtering, and data structure modification.

It should be understood that the configuration and boundaries of the functional building blocks of the system 400 have been defined herein for ease of description. Alternative boundaries may be defined so long as the specified functions and their relationships are properly performed. Alternatives (including equivalents, extensions, variations, deviations, etc. of the alternatives described herein) will be apparent. Such alternatives are within the scope of the disclosed embodiments.

FIG. 5 is a flow chart of an exemplary process 500 for database migration consistent with disclosed embodiments. In some embodiments, elements of system 300 may perform process 500. For example, as disclosed in the following step description, DM system 320 may perform process 500. In some embodiments, controller 323 may be configured to perform operations in process 500. Alternatively or additionally, online resources 340 and/or third-party systems 360 may perform process 500 or portions of process 500. Furthermore, in other embodiments, system 100 or portions of system 100 may perform process 500. For example, internal front-end system 105, SCM system 117, and/or SAT 101 may perform process 500. Furthermore, in other embodiments, system 400 may perform process 500. For example, the migration tool 402 , the controller 323 , and/or the DB clustering system 420 may perform operations in the process 500 .

The following description of process 500 illustrates an embodiment of DM system 320 performing the steps of process 500. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 500. For example, online resource 340 may perform one or more steps of process 500.

In step 502, the DM system 320 may apply the database schema to a target database, which may be set as a database clustering system. For example, the DM system 320 may apply the database schema to one of the regional databases 325. In some embodiments, the system DM system 320 may apply the database schema to the DB clustering system 420 via the controller 323. In some embodiments, as further discussed in conjunction with FIG. 9, exporting data in step 504 may include generating a replica database. To minimize downtime and avoid operational interruptions, the DM system 320 may initiate process 500 with a replication operation that allows the DM system 320 to perform data export and verification without interfering with an online database (such as online DB 404).

In step 504, the DM system 320 may export data from the database. For example, the DM system 320 may export data in one or more databases 380 to SQL files and/or CSV files. In some embodiments, the DM system 320 may export data of the online DB 404 via the controller 323. For example, as discussed in conjunction with FIG. 4, the controller 323 may include an export tool 412. In such embodiments, the DM system 320 may configure and execute operations for exporting data from the online DB 404 using the configuration and/or functions of the export tool 412.

In step 506, the DM system 320 may import the file into a target repository, which may be configured as a database clustering system. For example, the DM system 320 may import the file into one of the regional databases 325. In some embodiments, the DM system may import the data exported in step 504 into the DB clustering system 420 via the controller 323. For example, as discussed in conjunction with FIG. 4, the controller 323 may include a data loader tool 414. In such embodiments, the DM system 320 may configure and execute operations for loading data into the DB clustering system 420 using the configuration and/or functionality of the data loader tool 414.

In step 508, the DM system 320 may verify the consistency of the imported data. For example, the DM system 320 may verify the consistency and/or accuracy of the data imported into the target database in step 506. In some embodiments, the DM system 320 may verify the data imported into the DB clustering system 420 in step 506 via the controller 323. For example, as discussed in conjunction with FIG. 4, the controller 323 may include a verification tool 416. In such embodiments, the DM system 320 may configure and execute operations for verifying the operations imported into the DB clustering system 420.

Although FIG. 5 illustrates step 508 following step 506, some embodiments may perform parallel verification of data consistency. As discussed in conjunction with FIG. 4 , verification tool 416 may operate during import by, for example, operating on a particular cluster of DB clustering system 420 while completing other clusters. Parallel verification may include higher-level verification, as discussed above. However, in other embodiments, step 510 may include both parallel and sequential verification.

In step 510, the DM system 320 may turn on dual write operations. For example, as discussed above in conjunction with Figure 4, the controller 323 may include a tool for dual write that captures incremental data in two database spaces. The incremental data may be received from the online resource 340 and/or the client device 350. For example, a user may issue a database query via the client device 350 to update pricing data for a product or multiple products. In such embodiments, changes requested by the user may be replicated in both the online DB 404 and the DB clustering system 420. Alternatively or in addition, instructions to update the online DB 404 may be received from the online resource 340 and/or the migration tool 402 because the user makes changes when the DM system 320 performs the migration. In step 510, the DM system 320 may initiate a dual write operation to replicate the data modifications in the online database to the target database.

In step 512, the DM system 320 may generate a new database replica from the online database. For example, the DM system 320 may generate a new replica database to capture the latest data and data modifications in the online database. In such embodiments, the new database may capture more refreshed data. In some embodiments, the DM system 320 may use the controller 323 to generate a new replica in a new cloud situation. In some embodiments, a new or second replica of the online database may be overwritten to the second replica database via the first replica. For example, step 512 may include overwriting via a previously replicated database. However, in other embodiments, the replica database may be independent.

In step 514, the DM system 320 may identify and load the delta of the data set or modifications between the replicas generated in step 512 from the point of the dual write operation started in step 510. For example, the DM system 320 may identify the data that has changed since the dual write started to capture the incremental data. In some embodiments, the DM system 320 may identify the data via data validation. And the DM system 320 may load the delta or catch-up data to the DB clustering system 420 via synchronization of the tables from the new replica database.

In step 516, the DM system 320 may run a new consistency verification. For example, after loading the data delta into the DB clustering system 420 in step 514, the DM system 320 may run a new verification process similar to the verification process performed in step 508, but this time using the real-time database (e.g., online DB 404) data set for verification. For example, via step 516, the DM system 420 verifies that the catch-up data loaded in step 514 results in the same data in the online DB 404 as in the target DB clustering system 420. This step allows the DM system 320 to verify that the data transfer has been completed and that the two databases have the same content and structure.

In step 520, the DM system 320 may determine whether there are any inconsistencies in the migration. Based on the results of the validation from step 516, the DM system 320 may determine whether there are inconsistencies between the source database and the target database. If in step 520, the DM system 320 determines that there are data inconsistencies (step 520: yes), the DM system 320 may return to step 512, generate a new DB copy to subsequently identify, load the catch-up data (step 516), and then perform a new validation (520). In some embodiments, this iterative process may be replicated multiple times until the validation process is successful. If the DM system 320 determines that there are no data inconsistencies (step 520: no), the DM system 320 may proceed to step 522.

In step 522, the DM system 320 may determine that the migration process has completed data catch-up and/or that the migration has resulted in a complete copy of the database. In some embodiments, as further discussed in conjunction with FIG. 9, the apex of the catch-up operation that loads the data delta may then proceed to replace online traffic from the original database to the migration or target database. For example, in some embodiments, traffic that was originally routed to the online DB 404 may be sent to the DB clustering system 420. In addition, the DM system 320 may perform additional operations, such as generating notifications to the client device 350 and/or updating API processing based on the data migration.

FIG. 6 is a flow chart of an exemplary process 600 for data consistency verification for migration consistent with the disclosed embodiments. In some embodiments, elements of the system 300 may perform the process 600. For example, as disclosed in the following step description, the DM system 320 may perform the process 600. In such embodiments, the controller 323, and more specifically the verification tool 416 may be configured to perform operations in the process 600. Alternatively or in addition, the online resources 340 and/or the third party system 360 may perform the process 600 or a portion of the process 600. Furthermore, in other embodiments, the system 100 or a portion of the system 100 may perform the process 600. For example, the internal front-end system 105, the SCM system 117 and/or the SAT 101 may perform the process 600. Additionally, in other embodiments, system 400 may implement process 600. For example, transfer tool 402, controller 323, and/or DB clustering system 420 may perform operations in process 600. In some embodiments, process 600 may be part of process 500. For example, process 600 may be performed as part of step 516 in process 500 ( FIG. 5 ). In other embodiments, process 600 may be independent of process 500.

The following description of process 600 shows an embodiment in DM system 320 performing the steps of process 600. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 600. For example, online resource 340 may perform one or more steps of process 600. In such embodiments, online resource 340 may communicate and perform operations as specified by controller 323.

In step 602, the DM system 320 may initiate a validation process. For example, as part of process 500, the DM system 320 may initiate data validation during import, copy, and/or data validation of data. In some embodiments, as illustrated in FIG. 6, validation may be based on column comparison validation. However, in other embodiments, validation may be based on table level and/or column validation. In such embodiments, the DM system 320 may perform operations similar to table column counts; column count grouping; row aggregation; random filtering; and filter restrictions. Additionally, in step 602, the DM system may be configurable to perform row or SQL level validation. For example, the validation tool may perform operations such as validation schemes/row data types, perform hash comparisons, and/or run custom queries on different data sources. In addition, in some embodiments - and to improve resource allocation during migration - the DM system 320 may perform row column validation on a selected number of columns in the online database. Such columns may be selected based on data ranges identified by a user (e.g., via the migration tool 402). In such embodiments, the DM system 320 may perform a column comparison of the replica database and the migration database by comparing the hash of the row value in each column of the first replica database and the migration database.

In step 610, DM system 320 may determine whether any inconsistencies in the data are found. For example, DM system 320 may determine whether all columns between the two databases pass a validation test (e.g., using a hash function). If DM system 320 determines that an inconsistency exists as a result of the validation (step 610: yes), DM system 320 may proceed to step 612.

In step 612, the DM system 320 may identify a range of inconsistencies. For example, the DM system 320 may identify a range of columns that have inconsistent sets of data. In step 614, the DM system 320 may attempt to resolve the inconsistency by running a catch-up load for the specific range of data. For example, as discussed in conjunction with FIG. 5, the DM system 320 may generate a replica database that is subsequently used to load the delta data. In step 614, the DM system may run these processes (e.g., steps 512 and 514), but limited to the specific range identified in step 612 to resolve the inconsistency. As shown in FIG. 6, after re-running the catch-up load for the range with inconsistent data, the DM system 320 may return to step 602 and re-run the validation.

However, if in step 610, the DM system 320 determines that there is no inconsistency (step 610: No), the DM system 320 may proceed to step 622 and close the dual write operation. As discussed, in conjunction with FIG. 5, the DM system 320 may use the dual write operation to, for example, write incremental data to a target database, such as the DB clustering system 420. When the verification is complete and no inconsistency is found, the DM system 320 may stop the dual write operation.

In step 624, the DM system 320 may rerun the catch-up verification for a predetermined time window to verify that no additional data has been modified in the online database after the double write cutoff. Because the online database may have changed, even during a smaller time window, the DM system 320 may rerun the catch-up verification during the time window to identify any data changes that may have occurred since the last successful verification and may not have been captured by the double write operation. If any missing data is identified, the DM system 320 may load the catch-up data.

In step 626, the DM system 320 may switch the traffic to the target database. For example, the DM system 320 may switch the traffic from the online DB 404 to the DB clustering system 420. For example, the DM system 320 may adjust the database address, server, and/or API handler to route the data request from/to the original database to the migration database.

In step 628, DM system 320 may free up space in the replica database. For example, as discussed in conjunction with FIG. 5, DM system 320 may create a replica database, such as replica DB 406, in a regional database 325 that is used to verify, export, and manipulate data during the migration process. DM system 320 may also use regional database 325 for a replication catch-up operation, which synchronizes a target database with refresh data. In step 628, DM system 320 may free up space in such replica databases by, for example, freeing up space in regional database 325.

FIG. 7 is a flow chart of an exemplary process for migration of a real-time database consistent with the disclosed embodiments. In some embodiments, components of system 300 may perform process 700. For example, as disclosed in the following step description, DM system 320 may perform process 700. In such embodiments, controller 323 may be configured to perform operations in process 700. Alternatively or additionally, online resources 340 and/or database 380 may perform process 700 or portions of process 700. Furthermore, in other embodiments, system 100 or portions of system 100 may perform process 700. For example, internal front-end system 105, SCM system 117, and/or SAT 101 may perform process 700. Furthermore, in other embodiments, system 400 may perform process 700. For example, the migration tool 402 , the controller 323 , and/or the DB clustering system 420 may perform operations in the process 700 .

The following description of process 700 shows an embodiment in DM system 320 performing the steps of process 700. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 700. For example, online resource 340 may perform one or more steps of process 700. In such embodiments, online resource 340 may communicate and perform operations as specified by controller 323.

In step 702, the DM system 320 may apply a database schema and turn on dual write to replicate incoming data modifications to the DB clustering system. For example, the DM system 320 may apply a database schema from the online DB 404 to the DB clustering system 420. Further in step 704, the DM system 320 may turn on dual write to test replication of data via dual write operations. In some embodiments, as shown in FIG. 7, steps 702 and 704 may be performed before generating an export file. In such embodiments, the DM system 320 may apply a database schema to migrate a database; perform a test of a dual write operation; and suspend a dual write operation before generating an export file and/or replicating a live database.

In step 706, DM system 320 may generate a first replica database from the online database. For example, DM system 320 may generate a replica of online DB 404 to replica DB 406 via controller 323, as discussed in FIG. 4 .

In step 708, the DM system 320 may export the data from the replica DB 406 to a transfer file. For example, via the controller 323 and/or the data export tool 412, the DM system 320 may export the data from the replica DB 406 to a transfer file. The transfer file may include an SQL file. However, in other embodiments, the transfer file may include a CSV file or other types of formats.

In step 710, the DM system 320 may import the transfer file into a target database having a clustering system. For example, the DM system 320 may import the transfer file into the DB clustering system 420 via the controller 323 and/or the data loader tool 414.

In step 712, the DM system 320 may create a new replica that replicates the original database at a later time after beginning to import files into the target database. For example, the DM system 320 may create a new database replica in one of the regional databases 325 with refreshed data. As further discussed in conjunction with FIG. 5, the DM system 320 may perform a catch-up operation to identify and subsequently load data that was not captured during the initial replication. Through sequential replication and concurrent or iterative verification of online databases, the DM system 320 may complete the migration with no data loss and no downtime.

In step 714, DM system 320 may run a catch-up routine to complete the transfer of new data. For example, DM system 320 may perform operations for identifying deltas of information that have not been captured since the last transfer and loading them into the target database. The catch-up routine may also include operations performed by catch-up tool 418 (FIG. 4). In addition, the catch-up routine may also include concurrent and/or sequential validation of data, as discussed in steps 514-520 (FIG. 5).

In step 716, the DM system 320 may perform a final synchronization between the source database and the target database. For example, the DM system 320 may select a time window to perform traffic analysis and range verification between the databases, and export any inconsistent data from the source database to the target database.

In step 718, the DM system may transfer traffic from the source real-time database to the migration or target database. For example, the DM system 320 may switch traffic from the online DB 404 to the DB clustering system 420. In some embodiments, the DM system 320 may adjust database addresses, servers, and/or API handlers to route data requests from/to the original database to the migration database, or instruct other devices to do the same.

FIG. 8 is a flow chart of an exemplary process for database migration verification consistent with the disclosed embodiments. In some embodiments, elements of system 300 may perform process 800. For example, as disclosed in the following step description, DM system 320 may perform process 800. In such embodiments, controller 323 may be configured to perform operations in process 800. Alternatively or in addition, online resources 340, client devices 350 and/or database 380 may perform process 800 or portions of process 800. Furthermore, in other embodiments, system 100 or portions of system 100 may perform process 800. For example, internal front-end system 105, SCM system 117 and/or SAT 101 may perform process 800. Furthermore, in other embodiments, system 400 may perform process 800. For example, the transfer tool 402 , the controller 323 , and/or the DB clustering system 420 may perform operations in the process 800 .

The following description of process 800 shows an embodiment in DM system 320 performing the steps of process 800. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 800. For example, online resource 340 may perform one or more steps of process 800. In such embodiments, online resource 340 may communicate and perform operations as specified by controller 323.

In some embodiments, to execute process 800, DM system 320 may perform a series of operations to first execute the query using sequential operations of query, shard determination, and column comparison. For example, DM system 320 may execute the query using, for example, the following instructions: SELECT PK, SHARD_KEYS, crc32(CONCAT_WS(＜ALL_COLUMNS＞)) as crc FROM ＜TABLE＞ WHERE (`PK` ＞= 1 AND `PK` ＜ 1001) ORDER BY `PK`

In step 802, the DM system 320 may initiate a validation operation. For example, as part of process 700, the DM system 320 may validate data after (or during) an export/import operation (e.g., in step 710) and/or identify missing data for a catch-up operation (e.g., in step 714). In such scenarios, the DM system 320 may initiate data validation through, for example, a merge or trigger operation performed by the validation tool 416.

As shown in FIG8 , the verification process may be performed in parallel operations in the source database and the target database. For example, the verification tool 416 may be configured to operate in both the source database and the target database after the data is transferred and perform parallel operations to determine data inconsistencies. In such embodiments, steps 804-808 may be performed on the data in the source database. For example, the controller 323 may perform steps 804-806 on the online DB 404. In contrast, steps 810-812 may be performed on the data in the transfer or target database. For example, the controller 323 may perform steps 810-814 on the DB clustering system 420.

In step 804, the DM system 320 may read the table in the database. In some embodiments, as discussed above, to minimize downtime, the DM system 320 may perform a validation operation on the replica database. In this way, the DM system 320 may avoid hampering operations, particularly for databases supporting online applications. Therefore, in step 804, the DM system 320 may read the table in the replica database. For example, the DM system 320 may configure the controller 323 to read the table in the replica DB 406.

In step 806, the DM system 320 may calculate a shard key for each column in the database to determine the shards. The shard key may include a single index field and/or multiple fields covered by a composite index that determines the distribution of the collection files among the cluster shards. The DM system 320 may calculate the shard key by having a value and using this as a partition. When calculating the shard key, the DM system 320 may calculate a hash value and/or pre-configured label of the source configuration for each column. In some embodiments, the calculated shard key may be an application and metric name that defines a group of shards to be used for that application. Additionally or alternatively, the DM system 320 may calculate a hash from each column in the database object and compare it to a shard identifier representing the shard. In such embodiments, the DM system 320 may generate a log of index data on the shards. The DM system 320 may use different methods to calculate the shard keys, including but not limited to DJB2 (32 bits), SDBM (32 bits), LoseLose (32 bits), FNV-1/FNV-1a (32 bits), CRC16 (16 bits), and/or Murmur2/Murmur3 (32 bits).

In parallel with steps 804-806, DM system 320 may perform steps 810-812 on the target database. In some embodiments, steps 810-812 may correspond to steps 804-806, but the steps are applied to the target database. For example, while DM system 320 may perform the verification of steps 804-806 on replica DB 406, DM system 320 may perform the verification of steps 810-812 on DB clustering system 420. Similar to step 804, in step 810, DM system 320 may read a table in DB clustering system 420. And, as in step 806, the DM system 320 may calculate a shard key for each row to determine the shard in the DB clustering system 420. In addition, the DM system 320 may determine the shard key by using a SHA1 algorithm to generate a hash value and/or shard key. For example, the DM system 320 may use a union encoding class to convert a string into a byte array that is hashed using a SHA256 class.

In step 814, the DM system 320 may compare the row values and/or shards calculated in steps 804-812. For example, the DM system 320 may compare the shard key, shard, and hash value using an iterative operation for each bit tuple of the value.

In step 816, the DM system 320 may determine whether the records are equal. For example, in step 816, the DM system 320 may determine whether the target database has the same content and data structure as the source database based on a comparison of shards, shard keys, and/or hash values. If the DM system determines that the records are not equal (step 816: No), the DM system 320 may proceed to step 818.

In step 818, the DM system 320 may determine that a different record (e.g., a different column) weather has a newer modification after the double write operation. Because the different record may have been modified after the double write was performed, the different record may not need to be corrected because it will reflect the most recent changes. Therefore, the record with the newer modification after the double write operation (e.g., after performing step 510) may be the most recent and no differences need to be resolved or corrected. Therefore, if in step 818, the DM system 320 determines that a different record has a newer modification time, the DM system 320 may continue to step 822 and ignore the column or record with a different hash or the table with a different shard. After ignoring the record with the newer modification, the DM system 320 may proceed to step 816 and again determine whether the compared records are equal. However, if in step 818, the DM system 320 determines that the different record does not have a newer modification time, the DM system 320 may proceed to step 820 and indicate that the verification was unsuccessful. If the different record does not have a newer modification, it means that the double write process has not captured the difference and there is a problem with the data export or import. In such a scenario, the verification tool 416 may trigger a signal indicating an improper verification, and, for example, indicates an inconsistency addressed by the DM system 320, as discussed in conjunction with FIG. 5 (see, for example, step 520).

However, if in step 816, the DM system 320 determines that the records are equal (step 816: yes), the DM system 320 may proceed to step 824. In step 824, the DM system 824 may determine that the data transfer was successful. In these embodiments, the verification tool 416 may trigger a signal indicating proper verification, and for example, indicate that no inconsistencies in the transfer were found and the transfer process continues, as described in FIG. 5.

In some embodiments, the DM system 320 may perform the process 800 via a sequence of first determining whether a column in the first replica database has the same shard key as a corresponding column in the transfer database. In response to determining that one or more columns in the first replica database have a different shard key than a corresponding column in the transfer database, the DM system 320 may determine whether the column in the transfer database has a later modification time. Subsequently, the DM system 320 may ignore the columns in the first replica database that have a different shard key from the corresponding column in the transfer database. And the DM system 320 may determine data consistency when the shard keys of the non-ignored columns are the same. Via this sequence, the DM system 320 may perform a data validation process.

FIG. 9 is a flow chart of an exemplary process for online database migration consistent with the disclosed embodiments. In some embodiments, components of system 300 may perform process 900. For example, as disclosed in the following step description, DM system 320 may perform process 900. In such embodiments, controller 323 may be configured to perform operations in process 900. Alternatively or in addition, online resources 340 and/or database 380 may perform process 900 or portions of process 900. Furthermore, in other embodiments, system 100 or portions of system 100 may perform process 900. For example, internal front-end system 105, SCM system 117, and/or SAT 101 may perform process 900. Furthermore, in other embodiments, system 400 may perform process 900. For example, the transfer tool 402, the controller 323 and/or the DB clustering system 420 may perform operations in the process 900.

The following description of process 900 shows an embodiment in DM system 320 performing the steps of process 900. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 900. For example, online resource 340 may perform one or more steps of process 900. In such embodiments, online resource 340 may communicate and perform operations as specified by controller 323.

In step 902, DM system 320 may replicate the real-time database. For example, via controller 323, DM system 320 may export files from online DB 404 to generate replicated DB 406. In some embodiments, online DB 404 (which is replicated) may handle online traffic. In some embodiments, the replication of step 904 may facilitate data migration with little downtime.

In step 904, the DM system 320 may generate an export file from one or more replica databases and import it into the migration database. For example, via the controller 323, the DM system 320 may generate an export file from the replica DB 406 and then load it into the DB clustering system 420. In some embodiments, the DM system 320 may use a data export tool 412 to generate an export file from the replica DB 406. The DM system 320 may also use a data loader tool 414 to import or load the export file into the DB clustering system 420. As further explained in conjunction with Figures 4 to 5, the export file may include an SQL file, a CSV file, and/or different types of export files. In addition, as discussed in conjunction with Figure 4, the data loader tool 414 may perform loading operations in parallel to generate multiple clusters at the same time.

In step 906, the DM system 320 may perform a column comparison of the first copy database and the migration database. As shown in FIG. 9, step 906 may be performed after step 904. That is, step 906 may be performed after the export file is imported into the target database. However, in other embodiments, step 906 and step 904 may be performed in parallel. For example, as discussed in conjunction with FIG. 4, the data validation process may occur when the data is loaded into the DB clustering system 420.

In step 906, the DM system 320 may perform data validation by calculating shard keys for a selected number of columns in the replica database and the transfer database. For example, as further discussed in conjunction with FIG. 8 , the DM system 320 may perform data validation by reading tables in both the replica DB 406 and the DB clustering system 420, calculating shard keys, hash values, and/or determining shards to subsequently compare records and make a determination of whether they are equal.

In step 908, the DM system 320 may initiate a dual write operation to replicate the writes to the online database on the transfer database to capture incremental data. As discussed in conjunction with FIG. 5, the dual write operation may allow incremental data to be captured. In some embodiments, the dual write operation of step 908 may have been used prior to the initialization period. For example, as discussed with respect to process 700 (FIG. 7), the DM system 320 may apply a database scheme at the beginning of the process and then shut down the database scheme. The initial dual write operation may be used for testing and quality control. For example, the DM system 320 may perform a dual write operation and then perform data verification to confirm that the dual operation is working and replicating incremental data in the online DB 404 and the DB clustering system 420. As discussed in conjunction with FIG. 5 and FIG. 11 , the dual write operation can replicate data modifications in a real-time database (eg, online DB 404 ) to a target database or a migration database (eg, DB clustering system 420 ).

In step 910, DM system 320 may replicate the online database to a second replica database. For example, as discussed in conjunction with FIG. 5, DM system 320 may create a new replica with refreshed data that may enable identification of catch-up or delta data. In some embodiments, DM system 320 may use the same tools and procedures for creating the second replica in step 910 as for creating the first replica in step 902.

In step 912, the DM system 320 may identify the catch-up data by comparing the replica database of step 910 with the incremental data written after the initial double write operation. As discussed in conjunction with FIG5, the DM system 320 may perform operations for identifying sets of delta data that have not yet been loaded into the migration or target database. For example, the DM system 320 may identify the catch-up data (which some embodiments may identify based on modification time since the double write time).

In step 914, the DM system 320 may update the migration database to include the catch-up data. For example, via the controller 323, the DM system may load the data identified in step 912 into the DB clustering system 420. In addition, as discussed in conjunction with FIG. 5, the DM system 320 may perform iterative data validation until no inconsistencies are found between the replica database and the target or migration database. In some embodiments, the update catch-up operation in step 914 may be performed by: selecting a row group in the new replica database that includes at least a portion of the catch-up data; generating a temporary file that includes the row group; and synchronizing the migration database with the second replica database by inserting the temporary file in a location that corresponds to the row group in the second replica database. In this manner, DM system 320 can effectively copy delta or catch-up data identified in an online database (such as online DB 404) to a target database.

In step 916, the DM system 320 may determine whether the migrated data is consistent across the database. For example, the DM system 320 may employ the verify identical operation discussed in conjunction with FIG. 8 to determine whether the data in the online DB 404 and the DB clustering system 420 is consistent. If the DM system 320 determines that the migrated data is inconsistent (step 916: No), the DM system 320 may return to step 910 and copy the live database to the new replica database. For example, in some embodiments, after updating the migrated database to include the catch-up data, the DM system 320 may perform a second column comparison between the second replica database and the migrated database to determine whether the data is consistent. In response to determining that the data is inconsistent, the DM system may identify the inconsistent rows, generate a temporary file including the inconsistent rows, and update the migration database by inserting the temporary file in a position corresponding to the inconsistent row. However, if the DM system 320 determines that the migration data is consistent (step 916: yes), the DM system 320 may proceed to step 918.

In step 918, the DM system 320 may stop or pause the dual write operation. And in step 920, the DM system 320 may switch the online traffic. For example, the DM system 320 may coordinate addresses, server tables, and/or API handlers to route data requests from/to the online DB 404 to the DB clustering system 420. Therefore, in some embodiments, in response to determining that the data is consistent, the DM system 320 may pause the dual write operation and switch the online traffic from the online database to the migration database. In addition, in some embodiments, after pausing the dual write operation, the DM system 320 may perform a final synchronization between the second replica database and the migration database.

10 is a schematic representation 1000 of a database migration process using a replicated database consistent with disclosed embodiments. Schematic representation 1000 provides an example of a database migration, illustrating how some of the databases in the migration process will reflect certain information at different stages of the process.

At a first time 1010, the online DB 404 may have a record 1012. These records have been copied to the DB clustering system 420, which has a record 1014. The time 1010 may be a state after the DM system 320 has exported the file from the online DB 404 and imported the file into the DB clustering system 420.

At a second time 1020, the record in the online DB 404 has changed, there are more, and the online DB 404 now has record 1022. The record 1024 in the DB clustering system has not changed. Time 1020 may be the time when the DM system generates the first replication (see, e.g., FIG. 9 , step 902). Therefore, the replication DB 406 may have the same record as the online DB 404 at time 1020.

At a third time 1030, the record in the online DB 404 has changed again, there is an additional record, and the online DB 404 now stores record 1032. For time 1030, a double write operation may have been started. For example, as discussed in conjunction with FIG. 5, FIG. 7, and FIG. 9, a double write operation may have been initiated to copy the incremental data. Therefore, the new record also exists in the record 1034 of the DB clustering system. However, the new record cannot be used in the record 1036 of the replication DB 406 because the new record is a modification after the replication and before the double write operation.

At a fourth time 1040, there are no changes to the record 1042 in the online DB 404. However, the record 1044 in the DB clustering system 420 has been updated to include the missing records captured in the replication operation but not captured at the start. As shown in FIG. 10, some of the data is missing from the start, but can be obtained via the replication DB 406. At time 1040, as further discussed in conjunction with FIG. 5, FIG. 7, and FIG. 9, data from the replication DB 406 can be exported to the data file imported into the DB clustering system. Therefore, the record 1044 captures the missing data from the record 1046.

In the fifth time 1050, there are no changes to the records in 1054 of the DB clustering system 420, and there are no changes to the records 1056 of the replica DB 406. However, there are changes that are changed in the records 1052 of the online DB 404. These changes are not copied to the DB clustering system 420. This can occur when the double write operation has been turned off (for example, after a successful verification of the run), but the new data modifications are applied to the online DB 404. As discussed in conjunction with Figures 6 to 7, the DM system 320 can stop the double write operation after completing the import and verification data transfer. However, when the online DB 404 is a real-time database that can have multiple record changes (even within an hourly time frame), the record may have changed (even during an hourly time period).

At the sixth time 1060, the record 1062 in the online DB 404 and the record 1064 in the DB clustering system 420 remain the same. However, the record 1066 in the replica DB 406 has been updated and overwrites the record 1062. As discussed in conjunction with FIG. 6, the DM system 320 may perform a rerun catch-up within a predetermined time window. Therefore, at time 1060, the replica DB 406 may include refreshed data from the online DB 404. Although FIG. 10 illustrates updated records in the same memory space in the replica DB 406, in some embodiments, the space may be different. For example, the replica DB 406 may refer to multiple databases, such as the regional database 325 or the database 380 and each replica may be written in a different space.

In the seventh time 1070, the record 1072 in the online DB 404 and the record 1076 in the replica DB 406 remain the same. However, the record 1074 in the DB clustering system 420 has been updated to include the missing record after the double write is turned off. Similar to the process used in time 1050, the data in the replica DB 406 can be copied to the DB clustering system 420.

FIG. 11 is a timing diagram of an exemplary sequence 1100 for database migration consistent with the disclosed embodiments. As shown in FIG. 11 , the sequence 1100 may involve multiple elements of the system 400, including the online DB 404, the DB grouping system 420, and the replica DB 406. However, other elements of the system 400 may be involved in the sequence 1100. For example, although not shown, the controller 323 may perform one or more operations in the sequence 1100. In addition, others in the disclosed systems and elements may perform the sequence 1100. For example, in some embodiments, the system 300 may perform the sequence 1100. In such embodiments, the DM system 320, the database 380, and/or the online resource 340 may perform one or more operations of the sequence 1100.

In step 1102, the online DB 404 may communicate with the DB clustering system 420 to apply the solution and/or test the dual write operation. In some embodiments, the communication between the online DB 404 and the DB clustering system 420 may occur via the controller 323 in the DM system 320. For example, as discussed in conjunction with FIG. 7, the DM system 320 may perform operations for obtaining a database solution from the online DB 404, applying the solution, and turning on dual write to the DB clustering system 420.

In step 1104, online DB 404 may communicate with replica DB 406 to generate a replica. In some embodiments, step 1104 may be performed only when the dual write operation test in step 1102 succeeds. In addition, in some embodiments, as disclosed in conjunction with FIG. 5 and FIG. 7 , DM system 320 may coordinate the generation of a replica database.

In step 1106, the files in the replica DB 406 may be exported. For example, as described in conjunction with FIGS. 4-5, the export tool 412 in the controller 323 may generate an export file, which may include an SQL, CSV, or other type of file. In step 1108, the replica DB 406 may communicate with the DB clustering system 420 to load the exported data file. For example, as discussed in conjunction with FIGS. 4-5, the data loader tool 414 may load the data exported in step 1106 to the DB clustering system 420.

In step 1110 , the replica DB 406 and the DB clustering system 420 may communicate to perform a new verification of consistency, similar to the verification in steps 1110 and 1118 .

In step 1112, the online DB 404 and the DB clustering system 420 may communicate to have a dual write operation. As discussed in conjunction with FIG5 and FIG7, the DM system 320 may initiate a dual write operation from the online DB 404 to the DB clustering system 420 and capture incremental data.

In step 1114, online DB 404 and replica DB 406 may communicate to create a new replica. In some embodiments, the second replica DB may be the same as the first replica DB from step 1104. However, in other embodiments, the second replica DB may be only part of the architecture of replica DB 406. For example, replica DB 406 may be implemented using regional database 325. In such embodiments, the second replica DB may be a different instance of regional database 325.

In step 1116, the replica DB 406 and the DB clustering system 420 may communicate to identify and load catch-up data. For example, as discussed in conjunction with FIG. 5 and FIG. 9, the replica DB 406 and the DB clustering system 420 may identify the delta data of the DB clustering system 420 from the last double write time. This data delta or catch-up data may be loaded into the DB clustering system 420.

In step 1118, the replica DB 406 and the DB clustering system 420 may communicate to verify data consistency. For example, similar to step 1110, and as further discussed in conjunction with FIG. 8, the replica DB 406 and the DB clustering system 420 may communicate to determine whether the data is consistent between the different databases.

In step 1120, the replica DB 406 and the DB clustering system 420 may communicate to insert certain records in the DB clustering system 420 to complete the catch-up. For example, as further discussed in conjunction with FIG5 and FIG7, the DB clustering system 420 may synchronize with the most recent records in the second replica DB 406 used by inserting selected records that have not yet been directed to the DB clustering system 420 into a table (e.g., where the table indicates a null value).

In step 1122, online DB 404 may again communicate with replica DB 406 to provide limited window replication. For example, as further discussed in conjunction with FIG. 6, in some embodiments, after the end of the verification process and the double write operation, a new timed verification may be performed with a predetermined time window. The time window may be based on a specific number of minutes, for example. In step 1124, replica DB 406 and DB clustering system 420 may communicate to perform a new verification of consistency, similar to the verification of steps 1110 and 1118.

In step 1126, the online DB 404 and the DB clustering system 420 may communicate to switch traffic. As discussed in conjunction with FIG. 6 and FIG. 7 , the DM system 320 may coordinate addresses, server tables, and/or API handlers to route data requests from/to the online DB 404 to the DB clustering system 420. In such embodiments, the online DB 404 and the DB clustering system 420 may exchange addresses and/or other information for traffic handoff.

Another aspect of the disclosure is directed to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform a method, as discussed above. The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of computer-readable media or computer-readable storage devices. For example, the computer-readable medium may be a storage unit or memory module having stored thereon computer instructions, as disclosed. In some embodiments, the computer-readable medium may be a disk or flash drive having stored thereon computer instructions.

It will be apparent to those having ordinary knowledge in the art that various modifications and variations may be made to the disclosed systems and related methods. Other embodiments will be apparent to those having ordinary knowledge in the art from consideration of the specification and practice of the disclosed systems and related methods. It is intended that the specification and examples be considered merely as exemplary, with a true scope being indicated by the following claims and their equivalents.

Although the disclosure has been illustrated and described with reference to specific embodiments of the disclosure, it should be understood that the disclosure may be implemented in other environments without modification. The foregoing description is presented for the purpose of illustration. This is not exhaustive and is not limited to the precise form or embodiment disclosed. By considering the specification and practice of the disclosed embodiments, modifications and adjustments will be apparent to those with ordinary knowledge in the art. In addition, although aspects of the disclosed embodiments are described as being stored in memory, those with ordinary knowledge in the art will understand that these aspects may also be stored in other types of computer-readable media, such as auxiliary storage devices (e.g., hard disks or CD ROMs), or other forms of RAM or ROM, USB media, DVDs, Blu-ray discs, or other optical disc media.

Computer programs based on the written description and disclosed methods are within the skill of experienced developers. Various programs or program modules may be created using any technology known to those of ordinary skill in the art, or may be designed in conjunction with existing software. For example, program parts or program modules may be designed in or by the following methods: .Net Framework, .Net Compact Framework (and related languages, such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML/AJAX combination, XML, or HTML including Java applets.

Furthermore, although illustrative embodiments have been described herein, the scope of any and all embodiments with equivalent elements, modifications, omissions, combinations (e.g., across aspects of multiple embodiments), adaptations, and/or changes will be understood by one of ordinary skill in the art based on the skill of the present invention. The limitations in the claims are to be interpreted broadly based on the language used in the claims, and are not limited to the examples described in this specification or during the prosecution of this application. These examples should be interpreted as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including by reordering steps and/or inserting or deleting steps. Therefore, the description and examples are to be regarded as illustrative only, with the true scope and spirit being indicated by the appended claims and the full scope of their equivalents.

Therefore, the foregoing description has been presented for illustrative purposes only. The foregoing description is not exclusive and is not limiting to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those having ordinary skill in the art from consideration of this specification and practice of the disclosed embodiments.

The scope of the claims should be interpreted broadly based on the language used in the claims and not limited to the examples described in this specification, which should be considered non-exclusive. In addition, the steps of the disclosed methods may be modified in any manner, including by reordering the steps and/or inserting or deleting steps.

100: Block Diagram/System 101: Shipping Authorization Technology (SAT) System 102A: Mobile Device/Device 102B: Computer/Device 103: External Front-End System 105: Internal Front-End System 107: Transportation System 107A, 107B, 107C: Mobile Device/Device 109: Seller Portal 111: Shipping and Order Tracking (SOT) System 113: Fulfillment Optimization (FO) System 115: Fulfillment Messaging Gateway (FMG) 117: Supply Chain Management (SCM) System 119: Warehouse Management System (WMS) 119A: Mobile Device/Device/Tablet/Computer 119B: Mobile device/equipment/PDA/computer 119C: Mobile device/equipment/computer 121A, 121B, 121C: Third-party fulfillment (3PL) system 123: Fulfillment Center Authorization System (FC Auth) 125: Labor Management System (LMS) 200: Fulfillment Center (FC) 201, 222: Truck 202A, 202B, 208: Item 203: Inbound area 205: Buffer area 206: Forklift 207: Unloading area 209: Picking area 210: Storage unit 211: Packaging area 213: Hub area 214: Conveyor mechanism 215: Camp area 216: Wall/sorting device 218, 220: Package 224A, 224B: Delivery worker 226: Car 300: System 310: FC/warehouse system 320: Data transfer system 323: Controller 325: Regional database 340: Online resources 350: Client device 360: Third-party system 370: Network 380: Database 400: Transfer system 402: Transfer tool 404: Online database 406: Copy database 412: Data export tool 414: Data loader tool 416: Verification tool 418: Tracking tool 420: Database clustering system 500, 600, 700, 800, 900: process 502, 504, 506, 508, 510, 512, 514, 516, 520, 522, 602, 610, 612, 614, 622, 624, 626, 628, 702, 704, 706, 708, 710, 712, 714, 716, 718, 802, 804, 806, 810, 812, 81 4. 816, 818, 820, 822, 824, 902, 904, 906, 908, 910, 912, 914, 916, 918, 920, 1102, 1104, 1106, 1108, 1110, 1112, 1114, 1116, 1118, 1120, 1122, 1124, 1126: Steps 1000: schematic representation 1010, 1020, 1030, 1040, 1050, 1060, 1070: time 1012, 1014, 1022, 1024, 1032, 1034, 1036, 1042, 1044, 1046, 1052, 1054, 1056, 1062, 1064, 1066, 1072, 1074, 1076: record 1100: sequence

FIG. 1A is a schematic block diagram of an exemplary embodiment of a network including a computerized system for communication for implementing shipping, transportation, and logistics operations consistent with disclosed embodiments. FIG. 1B depicts a sample Search Result Page (SRP) including one or more search results satisfying a search request and interactive user interface elements consistent with disclosed embodiments. FIG. 1C depicts a sample Single Display Page (SDP) including products and information about the products and interactive user interface elements consistent with disclosed embodiments. FIG. 1D depicts a sample shopping cart page including items in a virtual shopping cart and interactive user interface elements consistent with disclosed embodiments. FIG. 1E depicts a sample order page including items from a virtual shopping cart and information about purchase and shipping and interactive user interface elements consistent with disclosed embodiments. FIG. 2 is a diagrammatic illustration of an exemplary fulfillment center configured to utilize the disclosed computerized system consistent with disclosed embodiments. FIG. 3 is a schematic block diagram of an exemplary system consistent with disclosed embodiments. FIG. 4 is a block diagram of an exemplary database migration system consistent with disclosed embodiments. FIG. 5 is a flow chart of an exemplary process for database migration consistent with disclosed embodiments. FIG. 6 is a flow chart of an exemplary process for data consistency verification for migration consistent with disclosed embodiments. FIG. 7 is a flow chart of an exemplary process for migration of a real-time database consistent with disclosed embodiments. FIG. 8 is a flow chart of an exemplary process for database migration verification consistent with the disclosed embodiment. FIG. 9 is a flow chart of an exemplary process for online database migration consistent with the disclosed embodiment. FIG. 10 is a schematic representation of a database migration process using a replicated database consistent with the disclosed embodiment. FIG. 11 is a timing diagram of an exemplary sequence for database migration consistent with the disclosed embodiment.

323: Controller

400: Migrate system

402: Transfer tool

404: Online database

406: Duplicate database

412:Data export tool

414:Data loader tool

416: Verification tool

418: Tracking tool

420: Database clustering system

Claims (20)

A system for database migration, comprising: one or more processors; and one or more memory devices storing instructions, which when executed by the one or more processors configure the one or more processors to perform operations including: copying an online database with online services to a first replica database; generating an export file from the first replica database and importing it into a migration database; performing a column comparison between the first replica database and the migration database after importing the export file, the column comparison being performed by calculating the sharding keys of a selected number of columns in the online database and the migration database; initiating a dual write operation to copy the write of the online database to the migration database to capture incremental data; After performing the column comparison, the online database is replicated to a second replica database; Identifying catch-up data by comparing the second replica database with the incremental data; and Updating the migration database to include the catch-up data. A system as described in claim 1, wherein updating the migration database includes: selecting a column group in the second replica database that includes at least a portion of the catch-up data; generating a temporary file including the column group; and synchronizing the migration database with the second replica database by inserting the temporary file into a location consistent with the column group in the second replica database. A system as described in claim 1, wherein: the row comparison is a first row comparison; and the operation further comprises: after updating the migration database to include the catch-up data, performing a second row comparison between the second replica database and the migration database to determine whether the data is consistent; in response to determining that the data is inconsistent: identifying inconsistent rows; generating a temporary file including the inconsistent rows; and updating the migration database by inserting the temporary file in a position corresponding to the inconsistent row. The system as described in claim 3, wherein the operation further comprises: In response to determining that the data is consistent: Suspending the dual write operation; and Switching online traffic from the online database to the migration database. The system as described in claim 4, wherein the operation further comprises: After pausing the dual write operation, performing a final synchronization between the second replica database and the transfer database. The system of claim 1, wherein performing the column comparison comprises: determining whether a column in the first replica database has the same shard key as a corresponding column in the transfer database; in response to determining that one or more columns in the first replica database have a different shard key than a corresponding column in the transfer database, determining whether the corresponding column in the transfer database has a later modification time; ignoring columns in the first replica database that have a different shard key than the corresponding column in the transfer database; and determining data consistency when the shard keys of non-ignored columns are the same. The system of claim 1, wherein the migration database comprises a database clustering system exposed to queries routed via an application programming interface (API) call. The system of claim 1, wherein the operation further comprises: Before generating the export file: Applying the database outline to the migration database; Performing a test of the dual write operation; and Pausing the dual write operation. A system as claimed in claim 1, wherein: the export file includes an SQL file; and copying the online database to the second replica database includes rewriting via the first replica database. The system of claim 1, wherein: the selected number of columns in the online database is selected based on a data range identified by a user; and performing the column comparison of the first replica database and the transfer database includes comparing hashes of row values in each of the first replica database and the transfer database. A method for database migration, comprising: Copying an online database with online services to a first replica database; Generating an export file from the first replica database and importing it into a transfer database; Performing a column comparison between the first replica database and the transfer database after importing the export file, the column comparison being performed by calculating the sharding keys of a selected number of columns in the online database and the transfer database; Initiating a dual write operation to copy the writes of the online database to the transfer database to capture incremental data; Copying the online database to a second replica database after performing the column comparison; Identifying catch-up data by comparing the second replica database with the incremental data; and Update the migration database to include the tracking data. A method as claimed in claim 11, wherein updating the migration database comprises: selecting a column group in the second replica database that includes at least a portion of the catch-up data; generating a temporary file including the column group; and synchronizing the migration database with the second replica database by inserting the temporary file into a location consistent with the column group in the second replica database. A method as claimed in claim 11, wherein: the row comparison is a first row comparison; and the method further comprises: after updating the migration database to include the catch-up data, performing a second row comparison between the second replica database and the migration database to determine whether the data is consistent; in response to determining that the data is inconsistent: identifying inconsistent rows; generating a temporary file including the inconsistent rows; and updating the migration database by inserting the temporary file in a position corresponding to the inconsistent rows. The method of claim 13 further includes: In response to determining that the data is consistent: Suspending the dual write operation; and Switching online traffic from the online database to the migration database. The method as described in claim 14 further includes: After pausing the dual write operation, performing a final synchronization between the second replica database and the transfer database. The method of claim 11, wherein performing the column comparison comprises: Determining whether a column in the first replica database has the same shard key as a corresponding column in the transfer database; In response to determining that one or more columns in the first replica database have a different shard key than a corresponding column in the transfer database, determining whether the corresponding column in the transfer database has a later modification time; Ignoring columns in the first replica database that have a different shard key than the corresponding column in the transfer database; and Determining data consistency when the shard keys of non-ignored columns are the same. A method as described in claim 11, wherein the migration database includes a database clustering system exposed to queries routed via an application programming interface (API) call. The method of claim 11 further comprises: Before generating the export file: Applying the database outline to the migration database; Performing a test of the dual write operation; and Pausing the dual write operation. The method of claim 11, wherein: the export file comprises an SQL file; copying the online database to the second replica database comprises rewriting via the first replica database; selecting the selected number of columns in the online database based on a data range identified by a user; and performing a column comparison of the first replica database with the transfer database comprises comparing hashes of row values in each of the first replica database and the transfer database. A system for migration of an online database, the system comprising: A first database coupled to a network; A second database; At least one processor coupled to the first database and the second database; and At least one memory device storing instructions, the instructions configuring the at least one processor to perform the following operations when executed by the at least one processor: Copying the first database to a first replica database; Generating an export file from the first replica database and importing it into the second database; Performing a column comparison between the first replica database and the second database after importing the export file, the column comparison being performed by calculating shard keys of a selected number of columns in the first database and the second database; Initiate a dual write operation to replicate the write of the first database to the second database to capture incremental data; After performing the column comparison, replicate the first database to the second replica database; Identify catch-up data by comparing the second replica database with the incremental data; Update the second database to include the catch-up data; After updating the second database to include the catch-up data, perform a second column comparison between the second replica database and the second database to determine whether the data is consistent; and After determining that the data is consistent: Suspend the dual write operation; and Switch online traffic from the first database to the second database.

Images