CN110825813B

CN110825813B - Data migration method and device

Info

Publication number: CN110825813B
Application number: CN201911112021.1A
Authority: CN
Inventors: 张倞祺; 马琛; 郑朝晖
Original assignee: China Air Clearing Co ltd; China Travelsky Technology Co Ltd
Current assignee: China Air Clearing Co ltd; China Travelsky Technology Co Ltd
Priority date: 2019-11-14
Filing date: 2019-11-14
Publication date: 2022-05-03
Anticipated expiration: 2039-11-14
Also published as: CN110825813A

Abstract

According to the data migration method and device, all the associated data of the full-service life cycle corresponding to any one main table data to be migrated are identified based on the service data relation model of the full-service life cycle, so that all the associated service data are guaranteed to be migrated in the data migration process, and service data residue is avoided. Meanwhile, after the data to be migrated is written into the historical database, the corresponding data in the production database is deleted, so that the storage space in the production database is saved.

Description

Data migration method and device

Technical Field

The invention belongs to the technical field of computers, and particularly relates to a data migration method and device.

Background

In the civil aviation passenger transport field, each passenger goes out and goes through buying tickets, changing labels or changing off, taking planes or refunding until the journey is finally completed. For the airline companies, information of all links needs to be collected, the information is finally transferred to a passenger transport settlement system, the passenger transport settlement system completes related settlement processing according to the actual situation of the data, financial data are generated and imported into the financial system. The whole process is processed around the passenger ticket, all business processing is completed and the final state is reached in the settlement link, and the corresponding passenger ticket completes the life cycle and does not perform business processing any more.

For a passenger transport settlement system, the core work is that data of each link of a passenger transport ticket is continuously received, and business data related to original data is formed after business processing is completed in the system, so the accumulated data volume of the system is continuously increased. The increasing of the original data and the service data may bring many negative effects, for example, a large amount of storage space of the production environment is occupied, the performance of the system service processing is affected, and the system backup efficiency is affected. Meanwhile, the current ticket data all use the ticket number as a unique identifier, and the ticket number is used as a unique key value in the design of a related system, but because the ticket number is repeatedly issued by using the most widely used BSP (Billing and Settlement Plan) neutral ticket data in the world for 2-3 years, the linear Settlement system needs continuous operation and maintenance, that is, the operation and maintenance of the ticket number are performed on the historical data of the repeatedly issued ticket number to solve the related business conflict.

In order to solve the above problem, it is necessary to purge the history data, which has completed the business process or has exceeded a prescribed time limit, from the production database. But for a settlement system, its historical data needs to have online query requirements. Therefore, the historical data cannot be simply cleaned up in the settlement system, but needs to be migrated to the historical database for storage. The existing historical data migration scheme is operated by operation and maintenance personnel regularly, and the historical data is backed up and cleaned by using a fixed script or a fixed processing flow; in addition, when historical data is cleaned, data which is not finished in business processing cannot be cleaned, accumulation of a large amount of incomplete business data is caused, and problems of related business operations may be caused.

Disclosure of Invention

In view of the above, an object of the present invention is to provide a data migration method and apparatus, so as to solve the problems that the existing historical data migration scheme cannot automatically migrate and a large amount of incomplete business data is accumulated when migrating data. The technical scheme is as follows:

in a first aspect, the present invention provides a data migration method, including:

identifying main table data to be migrated meeting preset migration conditions from a production database, wherein the preset migration conditions comprise migration periods;

identifying all associated data which are associated with the data of the main table to be migrated and have business association from all data tables of the production database based on a business data relationship model of a full-business life cycle, wherein the business data relationship model of the full-business life cycle is used for describing the relationship between business data which have association between all data main tables in the full-business life cycle;

based on an associated data extraction rule, reading main table data to be migrated and all associated data which are correctly verified from the production database to obtain data to be migrated, wherein the associated data extraction rule is used for describing business association relations among different data;

writing the data to be migrated which is verified to be correct into a historical database, and deleting the data to be migrated which is written into the historical database from the production database.

In a second aspect, the present invention further provides a data migration apparatus, including:

the system comprises a first identification module, a second identification module and a migration module, wherein the first identification module is used for identifying main table data to be migrated which meet preset migration conditions from a production database, and the preset migration conditions comprise migration periods;

the second identification module is used for identifying all associated data which are associated with the main table data to be migrated and are in business association from all data tables of the production database based on a business data relationship model of a full-business life cycle, and the business data relationship model of the full-business life cycle is used for describing the relationship among business data which are associated with each other among all data main tables in the whole business life cycle;

the reading module is used for reading the main table data to be migrated and all the associated data which are correctly verified from the production database based on the associated data extraction rule to obtain the data to be migrated, and the associated data extraction rule is used for describing business association relations among different data;

and the migration module is used for writing the data to be migrated which is correctly verified into a historical database, and deleting the data to be migrated which is written into the historical database from the production database.

The data migration method provided by the invention realizes that all associated data of the full-service life cycle corresponding to any main table data to be migrated is identified based on the service data relation model of the full-service life cycle, thereby ensuring that all associated service data are migrated in the data migration process and avoiding service data residue. Meanwhile, after the data to be migrated is written into the historical database, the corresponding data in the production database is deleted, so that the storage space in the production database is saved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a schematic diagram of a business data relationship model of a full-service lifecycle provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a business data table association model provided by an embodiment of the present invention;

FIG. 3 is a flow chart of a data migration method according to an embodiment of the present invention;

fig. 4 is a flowchart of a process for acquiring all associated service identifiers associated in the whole service life cycle according to the embodiment of the present invention;

FIG. 5 is a flow chart of a data verification process provided by an embodiment of the present invention;

FIG. 6 is a flowchart of a process for obtaining association data extraction rules according to an embodiment of the present invention;

FIG. 7 is a flow diagram of a data migration process provided by an embodiment of the present invention;

FIG. 8 is a flow chart of another data migration method provided by embodiments of the present invention;

FIG. 9 is a block diagram of a data migration apparatus according to an embodiment of the present invention;

fig. 10 is a block diagram of another data migration apparatus according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Before data migration is performed, the following preparation work needs to be performed:

1) defining a data table to be migrated according to actual service requirements, and defining a service data table association model based on a service data relation model of a full service life cycle and an actual storage structure of service data. And defining a relation table to be migrated based on the incidence relation between the data table to be migrated and the actual data, wherein the relation table to be migrated describes the incidence relation between the data tables to be migrated. Then, based on the business data table association model and the to-be-migrated relationship table, an associated data extraction rule (i.e. extracting an associated data SQL table) of each to-be-migrated data table is defined.

In the civil aviation passenger transport settlement system, a plurality of data tables are respectively recorded aiming at different services, and the data tables can be defined as a data master table and a data slave table, and the association relationship between the data master table and the data slave table is defined. Defining a data table to be migrated according to actual business requirements, wherein the data table related to the civil aviation international passenger transport income management business system comprises the following steps: sales ticket tables, transportation ticket tables, distribution record tracking tables, etc.

The business data relationship model of the full-service life cycle is shown in fig. 1, and describes the relationship between business data with association relationship between data main tables in the whole service life cycle. The business data table association model is shown in fig. 2, and is used for describing association relations among data main tables.

The table of the relationship to be migrated is obtained by analyzing the association relationship between each table of the data to be migrated and the actual service data, as shown in table 1:

TABLE 1

The associated data extraction rule is an important basis for reading data to be migrated from the production database, for example, the production database may adopt a relational database, such as an ORACLE database, in such an application scenario, the associated data extraction rule is to extract an associated data SQL table, which includes the following information as shown in table 2:

TABLE 2

2) Creating a historical database;

the historical database and the production database are separately deployed. The production database is a database for storing data that users can perform business processing through the application system, that is, a running database, and may be a relational database, such as an ORACLE database.

The historical database is specially used for storing the historical data after migration, and the data in the historical database only can provide a query function and cannot be subjected to any business processing. The historical data refers to data that has completed a service operation specified by the system, or data that exceeds a service processing deadline. The historical database is characterized by one-time writing, no modification, little reading and relatively low system response level, so that the MySQL database can be selected in the aspect of selection of the historical database to reduce the cost of database software. The data type definitions of different types of databases have certain differences and can be mutually converted according to actual application requirements.

In addition, the tables for storing the historical data in the historical database all use the 'historical data partitioning mark' field as a partitioning key to create partitions, and generally use the year, month or year as the partitioning key, so that the partitions are conveniently deleted by taking the partitions as units when the historical data of the historical database are subjected to offline operation based on the key value, and the processing difficulty is reduced.

3) After the historical database is created, a corresponding data table is created in the historical database

And in the historical database, creating a data table with the same structure as the data table to be migrated in the production database, and adding a special field for the historical data, so as to create and obtain a historical data table for storing the migrated historical data and support the subsequent query process.

The definition of the history data specific field is shown in table 3:

TABLE 3

In each data table in the historical database, a corresponding association key (namely, a historical data tracking number) is added for each data, a new service identifier is obtained after the original service identifier and the association key of the data are combined, and the historical data tracking number is used as a leading column. The data with the association relation in the production database is migrated to the historical database and then has a uniform association key (namely, a historical data tracking number), so that the repeatedly used ticket number (namely, the service identifier) can be distinguished in the historical database in different service life cycles where the ticket number is located.

Taking a sales ticket table as an example, the business identifier of the sales ticket table in the production database is a ticket number field, and the historical data tracking number field and the ticket number field of the sales ticket table in the historical database are set as new business identifiers and take the historical data tracking number as a front column. If the ticket number field is the primary basis for subsequent queries, a non-unique index is created for the ticket number field.

4) Defining and maintaining data retention period configuration (shown in table 4), data verification rule configuration (shown in table 5), system operation parameter configuration (shown in table 6) and the like, configuring verification operation according to the configuration rules, realizing the screening and verification of migration data regularly, and recording verification results; and configuring the migration operation according to the configuration rules to realize the data migration processing at regular intervals.

Table 4 illustrates the data retention period configuration:

TABLE 4

Table 5 shows the data verification rule configuration:

TABLE 5

Migration check rules	Checking whether to apply
		1. The non-trim ratio record has completed auditing	N
2. Has completed the pin number	N
		3. Has completed sales report review	Y
4. Shipping completed accounting processing	Y
		5. Account processing completed for internal billing	Y
6. The apportionment process has been completed	Y
		7. The tax split is completed

Table 6 is a system operating parameter configuration description:

TABLE 6

Referring to fig. 3, a flowchart of a data migration method provided by an embodiment of the present invention is shown, where the method is applied in a server, and as shown in fig. 3, the method includes the following steps:

and S110, identifying the main table data to be migrated meeting the preset migration condition from the production database.

The preset migration condition includes a migration period, which may specifically be a data retention period, where the data retention period may adopt the period number shown in table 4, or may be set according to an actual requirement.

All data exceeding the corresponding data retention period need to be migrated, and in the actual use process, the data migration date needs to be calculated and recorded. Then, whether the current time is greater than the migration date of the historical data of a certain main table is compared, if so, the historical data of the main table is determined to need migration processing, namely, the historical data of the main table is identified as the data to be migrated; if so, the master table history data is still stored in the production database.

In a civil aviation passenger transport settlement system, a plurality of data tables are respectively recorded for different services, for example, the data tables related to the civil aviation international passenger transport income management service system include: sales ticket list, transportation ticket list, sharing record tracking list, etc. these data lists have master-slave list correlation relation. For example, the sales ticket table and the transportation ticket table are data master tables, the allocation record tracking table is a slave table associated with the sales ticket table, and an association relationship is established between the allocation processing number and the sales ticket table.

The data main table refers to a table which can directly acquire data through an associated key, such as a passenger ticket sales table, a baggage ticket sales table and the like.

The data slave table means that data cannot be directly obtained according to the service identifier, the data of the master table must be obtained from the data master table according to the service identifier, then the data of the slave table is obtained according to the rule in table 2 to read the SQL, and the SQL is executed to obtain the related data in the slave table.

Therefore, all the associated data secondary tables can be found according to the association keys in the data primary table, and therefore, only the data of the primary table to be migrated needs to be determined, and then the associated data of the secondary table to be migrated is found according to the data of the primary table to be migrated.

And S120, identifying all associated data which are associated with the main table data to be migrated and have business association from each data table of the production database based on the business data relation model of the full-business life cycle.

For the main table data to be migrated identified in S110, based on the service data relationship model of the full service life cycle, all associated service identifiers associated with the main table data to be migrated in the entire service life cycle are obtained, and then corresponding service data is obtained according to the associated service identifiers.

In an embodiment of the present invention, as shown in fig. 4, the process of acquiring all associated service identifiers associated in the whole service life cycle is as follows:

and S121, acquiring the service identifier to be inquired.

In this embodiment, the service identifier to be queried is a unique identifier of the service data in the production database, and may be, for example, a ticket number of BSP-neutral ticket data.

The service identifier to be queried may be a service identifier of any piece of data associated with the main table data to be migrated in the full service life cycle. For example, a service identifier associated with certain to-be-migrated main table data is searched for the first time, and the to-be-queried service identifier is the service identifier of the to-be-migrated main table data; after the service identification associated with the main table data to be migrated is found, other service identifications associated with the service identifications are inquired one by one.

S122, judging whether the service identifier array contains the service identifier to be inquired; if not, executing S123; if so, S124 is performed.

It should be noted that S122 is a specific process for determining whether the current service identifier to be queried is queried.

Recording a service identifier associated with the main table data to be migrated in the service identifier array, and if the current service identifier to be queried exists in the service identifier array, indicating that the service identifier associated with the service identifier to be queried is queried, directly querying all sales transaction numbers corresponding to the service identifier to be queried at the moment; if the current service identifier to be queried does not exist in the service identifier array, the service identifier associated with the service identifier to be queried is not queried before.

S123, storing the service identifier to be inquired into the service identifier array.

And S124, inquiring the sales transaction number matched with the service identifier to be inquired in the corresponding data table according to the service identifier to be inquired.

The data tables related to the sales data all contain a service identification field and a sales transaction number field, the step is to inquire the sales record data corresponding to the service identification to be inquired in each sales data table, and then, the sales transaction number is obtained from the sales record data.

For example, according to the service identification, data tables of sales passenger tickets, sales luggage tickets, sales MCO, sales refund bills, sales waste tickets, refund replacement old ticket unions, partial refund old tickets and the like are inquired, sales record data matched with the service identification to be inquired is obtained, and then, the sales transaction number is obtained from the sales records. That is, tracing back to the old ticket direction based on the sales message, while finding the coupon, the companion ticket, and the hybrid companion ticket, and tracing back to the old ticket direction based on the complementary refund.

S125, aiming at each sales transaction number, inquiring whether a service identifier corresponding to the sales transaction number exists in each data table in the production database; if yes, respectively using the inquired business identifications as the business identifications to be inquired, and executing S122; if not, the inquiry matching process of the current sales transaction number is ended, and the business identification corresponding to the next sales transaction number is continuously inquired.

For example, the service identifier to be queried is ticket number a, and the sales transaction number corresponding to the ticket number a is queried to include a1, a2 and a3, and for the sales transaction number a1, a corresponding data table is queried to obtain all service identifiers, e.g., b and c, associated with the transaction number a 1; then, determining whether the service identifier b is contained in the service identifier array, if the service identifier array does not contain the service identifier b, continuing to query all sales transaction numbers corresponding to the service identifier b, such as b 1; continuing to query all service identifiers, e.g., d, corresponding to the sales transaction number b 1; and continuously inquiring to find that the service identification d does not have a corresponding sales transaction number. By this point, the query process for sales transaction number a1 ends, returning a query for all of the clerk identifications associated with sales transaction number a 2.

And executing the steps shown in the steps S121 to S125 to obtain all the associated service identifiers associated with the main table data to be migrated. Then, the corresponding data to be migrated can be read according to all the identified associated service identifiers.

In one embodiment of the invention, data verification operation and data migration operation are configured respectively, and the data verification operation and the data migration operation adopt asynchronous processing, because the data verification operation and the data migration operation have different influences on the production database, the data verification operation only performs read operation on the production database, and the influence on the production database is small, so that the data verification operation and the data migration operation can be executed at any time interval; the data migration operation needs to perform read-write operation on all the data tables to be migrated, and the data migration operation is executed in parallel through multiple threads during actual processing, which may cause a large performance impact on the production database, so the data migration operation is usually executed at a non-user-use stage (for example, the operation time of monday to friday is 22:00, and the operation time of saturday and sunday is 2:00 in the morning), and the single operation time is not limited, but a single processing threshold needs to be defined, so as to avoid the phenomenon that the execution period is too long and the online user use is affected.

And the data verification operation periodically performs migration data screening and verification according to the migration period configured by the system, and records the verification result.

In one embodiment of the present invention, as shown in fig. 5, the data verification process of the data verification job is as follows:

and S21, loading the data check rule configuration.

And S22, acquiring data of a main table to be migrated from each data table to be migrated.

The data verification operation screens the data in the production database according to a certain sequence (for example, ascending system time of the data), based on the migration period configuration, and identifies a service identifier corresponding to the data to be verified; and acquiring all associated service identifications having association relations in the life cycle of the full service based on the identified service identification and the service data table association model (i.e., the processes shown in S121 to S127). And acquiring the data to be migrated aiming at all the identified associated service identifiers.

S23, based on the data verification rule configuration, verifying the obtained main table data to be migrated, if the verification is correct, executing S24; if the error is checked, S25 is executed;

and S24, recording the information of the main table data to be migrated into the key value table of the data to be migrated.

The key value table of the data to be migrated is used for subsequently performing data migration, wherein the meaning of each field in the key value table of the data to be migrated is shown in table 7:

TABLE 7

And S25, recording the data information of the check error.

And if the data is checked to be wrong, recording the error reason of the checked wrong data, providing the error reason for the user to analyze and process, recording the data range of the checked data, and providing a basis for running the check again.

It should be noted here that the data verified by the data verification job may include data that does not need to be migrated in the production database for verification; it is also possible to check only the identified data that needs to be migrated.

Further, the data verification job single-run processing range is from the date maintained by the "verification complete to date" configuration item in the system run parameter configuration (i.e., the parameters shown in table 6) to the date obtained by the number of days of the run data verification job, i.e., the "verification date buffer period (day)" maintenance. For example, the "check completion date" is 20170101, the "check date buffer period (day)" is 60, and the data check job running date is 20180506, and therefore, the end date of the check range can be calculated as 20180506 minus 60 days, that is, 20180307. Finally, the current data verification job is determined to process the production data generated during the time period having the data range of 20170101 to 20180307.

The running time of the data checking operation can be configured to be 0 point per day, and the single running time is not limited until all the data needing to be checked in the data table to be migrated in the production database are scanned. When the data checking operation is started, the operation running state of the configuration item identifier of the 'data checking running state' in the system running parameter configuration is judged, if the 'RUN' indicates that the data checking operation triggered last time is still running, at the moment, the currently submitted data checking operation is directly terminated.

And S130, reading the main table data to be migrated and all the associated data which are correctly verified from the production database based on the associated data extraction rule to obtain the data to be migrated.

The associated data extraction rule is used for describing business association relation among different data;

in one embodiment of the present invention, as shown in fig. 6, the process of obtaining the association data extraction rule is as follows:

and S31, analyzing the relationship among the system service data to obtain a service data relationship model of the full service life cycle.

The obtained service data relationship model of the full service life cycle is shown in fig. 2.

And S32, obtaining a business data table association model based on the business data relation model of the full-business life cycle and the system business data storage structure. The business data table association model is used for describing association relations among all data main tables in the system.

And S33, obtaining the associated data extraction rule based on the business data table relation model and the pre-obtained relation table to be migrated.

The to-be-migrated relationship table is obtained by analyzing the association relationship between each to-be-migrated data table and actual service data, for example, in a specific example, if the to-be-migrated data table includes a sales ticket table, a transportation ticket table, an allocation record tracking table, and the like, the to-be-migrated relationship table is as shown in table 8:

TABLE 8

For example, the data table to be migrated includes: sales ticket table, table name SALPAX; a transportation ticket table, table name UPLPAX; record tracking table, table name PRPIND, was apportioned. The sales ticket table is a main table, a main key (i.e., a service identifier) of the sales ticket table is a ticket number field (salpax. tktno), the transportation record table is also a main table, the ticket number field (upllpax. tktno) is also stored in the table, and after the associated ticket number set is confirmed, the ticket numbers can be directly used to obtain records of the two tables. The sharing record tracking table is a 2-level association table, the upper-level table directly associated with the sharing record tracking table is a sales ticket table, and the two tables are associated through sharing processing numbers, namely SALPAX. When SQL is defined, binding variables can be directly defined for tables which are directly queried by using the service identifier (namely ticket number) of the main table, and the ticket number is transmitted as a parameter when SQL is executed. For the table which does not use the ticket number query, the main table record can be obtained in the main table through the sub-query, and then the current table record can be queried by taking the sub-query result as the association condition. For 3-level and 3 or more association tables, the current table record can be positioned after the parent table association record is obtained step by step through multi-level subquery. The obtained associated data extraction rule, i.e. the extracted associated data SQL table, is shown in table 9:

TABLE 9

After the associated data extraction rule is obtained, reading corresponding main table data to be migrated and all associated data from the production database based on the associated data extraction rule, wherein the main table data to be migrated and all associated data are the data to be migrated.

And S140, writing the correctly checked data to be migrated into the historical database, and deleting the data to be migrated which is written into the historical database from the production database.

The step is realized by submitting data migration operation, when the data migration operation is started, the operation running state of the configuration item identification of the 'data migration running state' in the system running parameter configuration is judged, if the state is 'RUN', the last started data migration operation is still running, and the data migration operation submitted this time is directly terminated.

After the data migration operation is started, reading data corresponding to a service identifier from a main table of the data to be migrated according to the service identifier of the main table data to be migrated, which is recorded in a key value table of the data to be migrated and is verified correctly; reading the data of the slave table to be migrated in the full-service life cycle having the association relation with the service identifier from the slave table to be migrated according to the association data extraction rule; in order to avoid the change of the checked data caused by asynchronous processing of the data checking job and the data migration job, the data needs to be checked again, and finally, the data which is checked again is written into the corresponding data table in the historical database.

In one embodiment of the present invention, as shown in FIG. 7, the migration process for a data migration job is as follows:

and S41, reading the data of the main table to be migrated corresponding to the service identification from the production database according to the service identification of the data recorded in the key value table of the data to be migrated.

And S42, re-checking the main table data to be migrated corresponding to the service identifier, and after the re-checking is correct, extracting the sub table data to be migrated having an association relation with the service identifier from the production database based on the association data extraction rule.

And S43, after the data of the slave table to be migrated is verified to be correct again, writing the data of the master table to be migrated and the data of the slave table to be migrated, which correspond to the service identifier, and the data of the slave table to be migrated, which have the association relationship, into the historical database, and deleting the data to be migrated in the production database.

Writing the data to be migrated into a historical database, and recording the number of written data records; similarly, the number of deleted data records is recorded after the data to be migrated is deleted from the production database.

In one embodiment of the invention, distributed transactions are adopted in the process of writing the historical database and the process of deleting the production database, so that the operations of the historical database and the production database are ensured to execute transaction submission at the same time, transaction rollback is simultaneously carried out when an error occurs, if partial submission is successful and partial submission is failed, automatic transaction compensation is executed, and the data is ensured to be only stored in one of the production database or the historical database.

Before the transaction (such as a transaction for writing a historical database and a transaction for deleting a production database) is submitted, respectively acquiring the number of deleted data records (namely the number of data deleted by SQL in a relational database) of each to-be-migrated data table in the production database and the number of written data records of each historical data table in the historical database; and if the number of the deleted records is not matched with the number of the written data records, performing transaction rollback and recording an error log, so that the subsequent positioning problem is facilitated. In the transaction submitting process, the transaction of the write history database is submitted first, then the transaction of the delete production database is submitted, if the transaction of the write history database is successfully submitted and the transaction of the delete production database is failed to be submitted, compensation is carried out, namely, the history data written into the history database before being deleted is submitted and the deleting operation is submitted. If the compensation fails, the management user is notified by the mail, and manual intervention is carried out. By adopting the transaction control, the production data can not be lost, and only under the special condition that the transaction part is successfully submitted and the compensation is failed, the production database and the historical database have some data at the same time and need to be manually processed by a management user. And after the data record number checking is finished, recording a migration operation log, and calculating data of detail level and summary level of the migration data. And finally, submitting the associated operation for finishing the processing of the migration data service.

In the data migration method provided by this embodiment, based on the service data relationship model of the full-service life cycle, all the associated data of the full-service life cycle corresponding to any one of the master table data to be migrated is identified, so that all the associated service data are guaranteed to be migrated in the data migration process, and service data residue is avoided. Meanwhile, after the data to be migrated is written into the historical database, the corresponding data in the production database is deleted, so that the storage space in the production database is saved.

Referring to fig. 8, a flowchart of another data migration method provided in the embodiment of the present invention is shown, where the method may further include the following steps based on the embodiment shown in fig. 3:

s210, identifying data which do not complete system preset service processing from the data to be migrated written in the historical database.

And after the data migration operation finishes the current historical data migration, submitting an immediately executed business processing operation, wherein the business processing operation is used for carrying out corresponding business processing on the migrated historical data. The service processing operation only needs to perform corresponding service processing on data which does not complete preset service processing in the migrated historical data, so that the data which does not complete service processing, that is, the data which does not match, needs to be identified first.

S220, generating corresponding service data for the data which does not complete the preset service processing of the system, and writing the corresponding service data into the historical database.

For the data (namely, the unpaired data) which is written into the historical database and has not completed the business processing, corresponding business processing is carried out, namely, corresponding historical data migration accounts are generated mainly aiming at the data which has not completed the business processing. Generating related bill accounts of the sale miss aiming at the sale miss data; and generating related bill accounts of the missed use aiming at the missed use data.

For example, in a business scenario, there is data that is not sold, and data migration is required after the data that is not sold is exceeded. In the business process, for the data which are not sold, the transportation income of the data need to be correspondingly estimated, if the data are sold, the estimated income needs to be converted into the transportation income; however, if the sales are not available all the time, the forecast account already made needs to be converted into the profit-and-loss account in the process of migrating the data to the historical database.

In the data migration method provided by this embodiment, in the process of migrating the historical data to the historical database, corresponding service processing is performed on data that has not been subjected to service processing, so that a large amount of incomplete data of services is prevented from being accumulated in the production database in the data migration process, and meanwhile, a problem of related service operation that may be caused by data migration is prevented from occurring.

In addition, the data migration method provided by the invention can also be used for inquiring historical data; when the query condition of the query page is defined, besides the query for the service field, the historical data tracking number field and the data migration time field can be displayed in the display result, so that the use ambiguity of the query result due to the repeated use of part of the service fields is avoided. For example, in the system, the service identifier (i.e. ticket number) in the sales ticket table is repeated due to the repeated use of the upstream system, and in the historical data query process, the sales records are queried based on the ticket numbers, and the data migration time is required to be taken in the display result to distinguish the difference between the data records with the same ticket number.

In addition, the data migration method provided by the invention can also provide a statistical form, and the data volume of data migration completed in a certain time period is counted according to a certain service dimension by taking the processing period as a summary dimension.

Corresponding to the above data migration method embodiment, the present invention also provides a data migration apparatus embodiment.

Referring to fig. 9, a block diagram of a data migration apparatus provided in an embodiment of the present invention is shown, where the apparatus is applied to a server, and as shown in fig. 9, the apparatus includes: a first identification module 110, a second identification module 120, a reading module 130, and a migration module 140.

The first identification module 110 is configured to identify, from the production database, the master table data to be migrated that meets a preset migration condition. The preset migration condition comprises a migration period.

The second identifying module 120 is configured to identify, based on the service data relationship model of the full-service life cycle, all associated data associated with the service of the data of the main table to be migrated from each data table of the production database.

The business data relation model of the full-service life cycle is used for describing the relation between the business data with the incidence relation among the data main tables in the whole service life cycle.

And the reading module 130 is configured to read the correctly verified main table data to be migrated and all relevant data from the production database based on the relevant data extraction rule, so as to obtain data to be migrated.

The associated data extraction rule is used for describing business association relation among different data. In an embodiment of the present invention, the data migration apparatus further includes: the obtaining module 210 is configured to obtain the associated data extraction rule.

The obtaining module 210 is specifically configured to:

analyzing the relationship between the system service data to obtain a service data relationship model of the full service life cycle;

and obtaining a business data table association model based on a business data relation model of the full-service life cycle and a system business data storage structure. The business data table association model is used for describing association relations among all data main tables in the system.

And obtaining an associated data extraction rule based on the business data table relation model and a pre-obtained relation table to be migrated. And the relation table to be migrated is used for recording the association relation among the data tables to be migrated.

In an embodiment of the present invention, the process of determining that the correct data of the main table to be migrated and all the associated data are verified is as follows:

screening out a service identifier corresponding to data to be verified from the data in the production database based on a preset verification time period;

acquiring all associated service identifications having an association relation with the service identification of the data to be verified in the life cycle of the whole service based on the service identification corresponding to the data to be verified and the service data table association model;

identifying data to be migrated with a correlation relation based on all the correlation service identifications, wherein the data to be migrated comprises data in a master table to be migrated and data in a slave table to be migrated;

for the identified data to be migrated with the incidence relation, carrying out correctness verification based on a preset data verification rule;

and recording the information of the data in the main table to be migrated which is checked to be correct into the key value table of the data to be migrated.

In an embodiment of the present invention, based on the service identifier corresponding to the data to be verified and the service data table association model, a process of acquiring all associated service identifiers having an association relationship with the service identifier of the data to be verified in the full service life cycle is specifically as follows:

acquiring a service identifier to be inquired;

judging whether the sales transaction number corresponding to the service identifier to be inquired is inquired;

if the sales transaction number corresponding to the service identifier to be inquired is not inquired, inquiring all the sales transaction numbers corresponding to the service identifier to be inquired;

inquiring all associated service identifications corresponding to each sales transaction number;

and circularly taking each inquired associated service identifier as a new service identifier to be inquired, and returning to execute the step of judging whether the sales transaction number corresponding to the service identifier to be inquired is inquired or not until all associated service identifiers associated with the service identifier of the data to be checked in the whole service life cycle are inquired.

And the migration module 140 is used for writing the data to be migrated which is verified to be correct into the historical database, and deleting the data to be migrated which is written into the historical database from the production database.

In one embodiment of the invention, the historian and production databases are deployed separately.

In an embodiment of the present invention, the migration module 140 is configured to, when writing the correctly verified data to be migrated into the historical database, specifically:

reading main table data to be migrated corresponding to the service identification from a production database according to the service identification of the data recorded in the key value table of the data to be migrated;

re-checking the main table data to be migrated corresponding to the service identifier, and extracting the sub table data to be migrated having an association relation with the service identifier from the production database based on the association data extraction rule after the re-checking is correct;

and after the data of the slave table to be migrated is verified to be correct again, writing the data of the master table to be migrated and the data of the slave table to be migrated, which correspond to the service identifier, having the incidence relation into the historical database, and deleting the data to be migrated in the production database.

In another embodiment of the invention, in the process of writing the data to be migrated with correct verification into the historical database, an association key is created for each data to be migrated with association relationship. The association key is used for marking data with association relation in each data table of the historical database.

And for any data to be migrated which is written into the historical database, determining the service identifier of the data to be migrated in the production database and the association key of the data to be migrated as a new service identifier of the data to be migrated after combining.

In one embodiment of the present invention, as shown in fig. 10, the apparatus further comprises: a third identification module 310 and a service processing module 320.

The third identifying module 310 is configured to identify data that does not complete the preset service processing of the system from the data to be migrated written in the history database.

And the service processing module 320 is configured to generate corresponding service data for data that does not complete the preset service processing of the system, and write the corresponding service data into the historical database.

In addition, the data migration device provided by the invention can also query historical data; when the query condition of the query page is defined, besides the query for the service field, the historical data tracking number field and the data migration time field can be displayed in the display result, so that the use ambiguity of the query result due to the repeated use of part of the service fields is avoided. For example, in the system, the service identifier (i.e. ticket number) in the sales ticket table is repeated due to the repeated use of the upstream system, and in the historical data query process, the sales records are queried based on the ticket numbers, and the data migration time is required to be taken in the display result to distinguish the difference between the data records with the same ticket number.

In addition, the data migration device provided by the invention can also provide a statistical form, the processing period is taken as the summarizing dimension, and the data volume of data migration completed within a certain time period is counted according to a certain service dimension.

While, for purposes of simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present invention is not limited by the illustrated ordering of acts, as some steps may occur in other orders or concurrently with other steps in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The steps in the method of the embodiments of the present application may be sequentially adjusted, combined, and deleted according to actual needs.

The device and the modules and sub-modules in the terminal in the embodiments of the present application can be combined, divided and deleted according to actual needs.

In the several embodiments provided in the present application, it should be understood that the disclosed terminal, apparatus and method may be implemented in other manners. For example, the above-described terminal embodiments are merely illustrative, and for example, the division of a module or a sub-module is only one logical function division, and other division manners may be available in actual implementation, for example, a plurality of sub-modules or modules may be combined or integrated into another module, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The modules or sub-modules described as separate parts may or may not be physically separate, and parts that are modules or sub-modules may or may not be physical modules or sub-modules, may be located in one place, or may be distributed over a plurality of network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, each functional module or sub-module in the embodiments of the present application may be integrated into one processing module, or each module or sub-module may exist alone physically, or two or more modules or sub-modules may be integrated into one module. The integrated modules or sub-modules may be implemented in the form of hardware, or may be implemented in the form of software functional modules or sub-modules.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A method of data migration, comprising:

writing the data to be migrated which is correctly checked into a historical database, and deleting the data to be migrated which is written into the historical database from the production database;

the associated data extraction rule is obtained by the following method:

obtaining a business data table association model based on the business data relationship model of the full-business life cycle and a system business data storage structure, wherein the business data table association model is used for describing the association relationship among all data main tables in the system;

and obtaining the associated data extraction rule based on the business data table relationship model and a pre-obtained relationship table to be migrated, wherein the relationship table to be migrated is used for recording the association relationship among the data tables to be migrated.

2. The method of claim 1, wherein determining to verify correct primary table data and all associated data to be migrated comprises:

acquiring all associated service identifications having an association relation with the service identification of the data to be verified in a full service life cycle based on the service identification corresponding to the data to be verified and a service data table association model;

3. The method according to claim 2, wherein acquiring all associated service identifiers having an association relationship with the service identifier of the data to be verified in a full service life cycle based on the service identifier corresponding to the data to be verified and a service data table association model comprises:

acquiring a service identifier to be inquired;

4. The method according to claim 2, wherein the writing the data to be migrated that is verified to be correct into a history database comprises:

reading main table data to be migrated corresponding to the service identification from the production database according to the service identification of the data recorded in the key value table of the data to be migrated;

the data of the main table to be migrated corresponding to the service identification is verified again, and after the data is verified again correctly, the data of the auxiliary table to be migrated, which has an association relation with the service identification, is extracted from the production database based on an associated data extraction rule;

and after the data of the slave table to be migrated is verified to be correct again, writing the data of the master table to be migrated and the data of the slave table to be migrated, which correspond to the service identifier, having an association relationship into the historical database, and deleting the data to be migrated in the production database.

5. The method according to claim 1, wherein writing the correctly verified data to be migrated into the historical database comprises:

creating an association key for each data to be migrated with association relation in the process of writing the correctly verified data to be migrated into the historical database, wherein the association key is used for marking the data with the association relation in each data table of the historical database;

6. The method of claim 1, further comprising:

identifying data which are not finished with system preset service processing from the data to be migrated written in the historical database;

and generating corresponding service data for the data which does not complete the preset service processing of the system, and writing the corresponding service data into the historical database.

7. The method of any of claims 1-6, wherein the historical database is deployed separately from the production database.

8. A data migration apparatus, comprising:

the migration module is used for writing the data to be migrated which is correctly verified into a historical database, and deleting the data to be migrated which is written into the historical database from the production database;

the device further comprises: the acquisition module is used for acquiring the associated data extraction rule;

the acquisition module is specifically configured to: