CN117667885A - Data migration method, device, terminal equipment and storage medium - Google Patents
Data migration method, device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN117667885A CN117667885A CN202311531019.4A CN202311531019A CN117667885A CN 117667885 A CN117667885 A CN 117667885A CN 202311531019 A CN202311531019 A CN 202311531019A CN 117667885 A CN117667885 A CN 117667885A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- data table
- source system
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 164
- 238000013508 migration Methods 0.000 title claims abstract description 142
- 238000000034 method Methods 0.000 title claims abstract description 68
- 230000008859 change Effects 0.000 claims abstract description 121
- 238000012544 monitoring process Methods 0.000 claims abstract description 33
- 230000002688 persistence Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 description 14
- 230000002159 abnormal effect Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000013524 data verification Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000003780 insertion Methods 0.000 description 3
- 230000037431 insertion Effects 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000011068 loading method Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data migration method, a device, a terminal device and a storage medium, wherein the data migration method comprises the following steps: continuously monitoring a data table of a source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table; if the change of the data table of the source system is monitored, the corresponding data table change information is obtained; according to the data table change information, determining a target data field corresponding to the changed data table from a plurality of data fields; data in the target data domain is migrated from the source system to the target system. Based on the scheme, a migration mode based on the data field is adopted, and the migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
Description
Technical Field
The present disclosure relates to the field of data migration technologies, and in particular, to a data migration method, a device, a terminal device, and a storage medium.
Background
Data migration from a source system to a target system is often required for data integration, upgrades, backups, and the like. The conventional data migration method is to migrate a large amount of stock data from a source system to a target system in a table-by-table manner within a specific time window, that is, to re-migrate all data each time, including already migrated data, which causes unnecessary performance overhead. Moreover, the conventional data migration method requires a longer time window to complete migration, and the time window required for providing data migration through shutdown may cause service interruption, resulting in user dissatisfaction.
In summary, the existing data migration method has the problems of high performance overhead and influence on services.
Disclosure of Invention
The main purpose of the application is to provide a data migration method, a device, a terminal device and a storage medium, and aims to solve the problems that the performance cost of the existing data migration method is high and the service is influenced.
In order to achieve the above object, the present application provides a data migration method, including:
continuously monitoring a data table of a source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table;
If the change of the data table of the source system is monitored, the corresponding data table change information is acquired;
determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information;
and migrating the data in the target data field from the source system to a target system.
Optionally, the source system is provided with a source database, the source database stores data corresponding to the data fields, and the step of continuously monitoring the data table of the source system includes:
continuously acquiring a transaction log file of the source database, and analyzing the transaction log file to obtain data table operation information;
and continuously monitoring the data table of the source system according to the data table operation information.
Optionally, the step of determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields includes:
extracting a target data table name corresponding to the changed data table from the data table change information;
and determining the target data field corresponding to the changed data table from the plurality of data fields according to the target data table name.
Optionally, after the step of acquiring the corresponding data table change information if the data table of the source system is monitored to change, the step of determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields further includes:
generating a message corresponding to the data table change information, and storing the message corresponding to the data table change information into a preset distributed publishing and subscribing message kafka system;
if the message corresponding to the data table change information issued by the kafka system is monitored, analyzing the message corresponding to the data table change information to obtain restored data table change information;
the step of determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information comprises the following steps:
and determining the target data field corresponding to the changed data table from the plurality of data fields according to the restored data table change information.
Optionally, the step of migrating the data in the target data domain from the source system to the target system includes:
Analyzing the hierarchical relationship of the target data field, and determining a root node corresponding to the target data field;
and acquiring data corresponding to the target data field according to the root node corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system.
Optionally, the number of the target data fields is a plurality of, and the steps of obtaining data corresponding to the target data fields according to the root node corresponding to the target data fields, and migrating the data corresponding to the target data fields from the source system to the target system include:
generating corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and storing the asynchronous tasks into a preset asynchronous task queue;
based on the plurality of asynchronous tasks, acquiring data corresponding to each of the plurality of target data fields through a preset Java persistence interface, and migrating the data corresponding to the target data fields from the source system to the target system.
Optionally, after the step of migrating the data in the target data field from the source system to the target system, the method further includes:
Acquiring a key detail index corresponding to the target data field;
acquiring first-type key detail data of the target data field from the source system and second-type key detail data of the target data field from the target system according to the key detail index;
comparing the first type of key detail data with the second type of key detail data to obtain comparison result information;
and storing the comparison result information into a preset comparison record table.
The embodiment of the application also provides a data migration device, which comprises:
the monitoring module is used for continuously monitoring the data table of the source system, wherein the source system comprises a plurality of preset data fields, and any one data field comprises at least one corresponding data table;
the acquisition module is used for acquiring corresponding data table change information if the data table of the source system is monitored to change;
the determining module is used for determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information;
and the migration module is used for migrating the data in the target data field from the source system to the target system.
The embodiment of the application also provides a terminal device, which comprises a memory, a processor and a data migration program stored on the memory and capable of running on the processor, wherein the data migration program realizes the steps of the data migration method when being executed by the processor.
The embodiments of the present application also provide a computer-readable storage medium having stored thereon a data migration program which, when executed by a processor, implements the steps of the data migration method as described above.
The data migration method, the device, the terminal equipment and the storage medium provided by the embodiment of the application are used for continuously monitoring the data table of the source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table; if the change of the data table of the source system is monitored, the corresponding data table change information is acquired; determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information; and migrating the data in the target data field from the source system to a target system. Based on the scheme, a migration mode based on the data field is adopted, and the migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
Drawings
FIG. 1 is a schematic diagram of functional modules of a terminal device to which a data migration apparatus of the present application belongs;
FIG. 2 is a flowchart illustrating a first exemplary embodiment of a data migration method according to the present application;
FIG. 3 is a flowchart illustrating a second exemplary embodiment of a data migration method according to the present application;
FIG. 4 is a flowchart illustrating a third exemplary embodiment of a data migration method according to the present application;
FIG. 5 is a flowchart illustrating a fourth exemplary embodiment of a data migration method according to the present application;
FIG. 6 is a flowchart illustrating a fifth exemplary embodiment of a data migration method according to the present application;
FIG. 7 is a flowchart illustrating a sixth exemplary embodiment of a data migration method according to the present application;
FIG. 8 is a flowchart of a seventh exemplary embodiment of a data migration method of the present application;
fig. 9 is a flowchart of an eighth exemplary embodiment of a data migration method of the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The main solutions of the embodiments of the present application are: continuously monitoring a data table of a source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table; if the change of the data table of the source system is monitored, the corresponding data table change information is acquired; determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information; and migrating the data in the target data field from the source system to a target system. Based on the scheme, a migration mode based on the data field is adopted, and the migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
Specifically, referring to fig. 1, fig. 1 is a schematic functional block diagram of a terminal device to which a data migration apparatus of the present application belongs. The data migration device may be a device independent of the terminal device, capable of performing data migration, and may be carried on the terminal device in the form of hardware or software. The terminal equipment can be an intelligent mobile terminal with a data processing function such as a mobile phone and a tablet personal computer, and can also be a fixed terminal equipment or a server with a data processing function.
In this embodiment, the terminal device to which the data migration apparatus belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.
The memory 130 stores an operating system and a data migration program, and the data migration device may store information such as monitoring information and data table change information in the memory 130; the output module 110 may be a display screen or the like. The communication module 140 may include a WIFI module, a mobile communication module, a bluetooth module, and the like, and communicates with an external device or a server through the communication module 140.
Wherein the data migration program in the memory 130, when executed by the processor, performs the steps of:
continuously monitoring a data table of a source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table;
If the change of the data table of the source system is monitored, the corresponding data table change information is acquired;
determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information;
and migrating the data in the target data field from the source system to a target system.
Further, the data migration program in the memory 130, when executed by the processor, further performs the steps of:
continuously acquiring a transaction log file of the source database, and analyzing the transaction log file to obtain data table operation information;
and continuously monitoring the data table of the source system according to the data table operation information.
Further, the data migration program in the memory 130, when executed by the processor, further performs the steps of:
extracting a target data table name corresponding to the changed data table from the data table change information;
and determining the target data field corresponding to the changed data table from the plurality of data fields according to the target data table name.
Further, the data migration program in the memory 130, when executed by the processor, further performs the steps of:
Generating a message corresponding to the data table change information, and storing the message corresponding to the data table change information into a preset distributed publishing and subscribing message kafka system;
if the message corresponding to the data table change information issued by the kafka system is monitored, analyzing the message corresponding to the data table change information to obtain restored data table change information;
and determining the target data field corresponding to the changed data table from the plurality of data fields according to the restored data table change information.
Further, the data migration program in the memory 130, when executed by the processor, further performs the steps of:
analyzing the hierarchical relationship of the target data field, and determining a root node corresponding to the target data field;
and acquiring data corresponding to the target data field according to the root node corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system.
Further, the data migration program in the memory 130, when executed by the processor, further performs the steps of:
generating corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and storing the asynchronous tasks into a preset asynchronous task queue;
Based on the plurality of asynchronous tasks, acquiring data corresponding to each of the plurality of target data fields through a preset Java persistence interface, and migrating the data corresponding to the target data fields from the source system to the target system.
Further, the data migration program in the memory 130, when executed by the processor, further performs the steps of:
acquiring a key detail index corresponding to the target data field;
acquiring first-type key detail data of the target data field from the source system and second-type key detail data of the target data field from the target system according to the key detail index;
comparing the first type of key detail data with the second type of key detail data to obtain comparison result information;
and storing the comparison result information into a preset comparison record table.
According to the scheme, the data table of the source system is monitored continuously, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table; if the change of the data table of the source system is monitored, the corresponding data table change information is acquired; determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information; and migrating the data in the target data field from the source system to a target system. In this embodiment, a migration manner based on the data field is adopted, and migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
Referring to fig. 2, a first embodiment of a data migration method of the present application provides a flowchart, where the data migration method includes:
step S10, continuously monitoring a data table of a source system, wherein the source system comprises a plurality of preset data fields, and any one data field comprises at least one corresponding data table.
Specifically, the current data migration method is a stock data migration method of table pairs, and the data of each table in the source database is migrated to the corresponding table in the target database one by one, so that the problems of high performance overhead and influence on the service exist, and the method is specifically expressed as follows: (1) high performance overhead: each time the entire data set is migrated again, even if only a small portion of the data changes, a significant amount of time and resources are wasted, causing unnecessary stress on the database and network resources. (2) service interruption: typically, a downtime or a specific time window is required to perform table migration, which can lead to service interruption, affect service continuity and availability, and cause customer dissatisfaction. (3) data inconsistency: because migration is disposable, data updates occurring during shutdown cannot be reflected in the target system in time, resulting in data inconsistency problems. (4) incremental migration problem: the table-to-table approach typically does not support incremental migration, requiring all data to be re-migrated, increasing resource overhead and risk. (4) not applicable to heterogeneous systems: if the table structures of the source and target systems are not exactly the same, the table-to-table approach may not be adaptable, requiring additional data conversion and mapping work.
For this reason, this embodiment proposes migration of incremental data based on the data domain. The source system to which the present embodiment relates refers to a system that stores original data, and the target system is a destination of data migration, that is, a system in which data is finally stored and used. The goal of data migration is to transfer, transform, and load data from a source system into a target system for further processing, analysis, or storage in the target system.
The source system comprises a plurality of preset data fields, and any one data field comprises at least one corresponding data table. The data fields are understood to be the classification or range of data, and are generally classified according to business logic or functions, and the data fields related to this embodiment are preset, which means that some data fields are determined in advance, and each data field includes at least one data table. For example, for insurance business, there may be a data area corresponding to customer information, a data area corresponding to policy information, a data area corresponding to claim records, and the like.
To implement incremental data migration, the data table of the source system needs to be continuously monitored based on a database trigger, a transaction log file, or a special data change monitoring tool, and changes of the data table, such as insertion, update, or deletion operations on data in the data table, are captured.
And step S20, if the change of the data table of the source system is monitored, the corresponding data table change information is obtained.
Specifically, when it is monitored that the data table of the source system changes, for example, new data is inserted, existing data is updated or data is deleted, the corresponding data table change information is triggered to be acquired. The content of the data table change information at least comprises a table name corresponding to the data table, and can also comprise one or more of table identification, specific change types (adding, updating, deleting and the like), changed data record and record change timestamp. The data table change information may be used to decide how to perform data migration or other related operations.
And step S30, determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information.
Specifically, a dependency relationship between the data tables and the data fields is pre-established, and typically one data field will include one or more corresponding data tables. Under the condition that the data table change information is obtained, the changed data table can be determined according to the data table change information, and then the data field to which the changed data table belongs, namely the target data field, is determined according to the changed data table and the subordinate relation between the data table and the data field.
And step S40, migrating the data in the target data field from the source system to a target system.
Specifically, after the target data field is determined, data belonging to the target data field is extracted from the source system. In some cases, the data format of the source system may not match the target system, and data conversion may be required, where the process of data conversion involves mapping of the data format, conversion of values, data cleansing, etc., to ensure that the data is properly accepted and understood in the target system. Once the data is extracted and transformed, they are loaded into the corresponding data fields of the target system, the process of loading involves an insert, update or delete operation of the data table to ensure that the data is properly stored in the target system. It will be appreciated that data migration includes the three phases of extraction, conversion, and loading described above.
Optionally, after migrating the data in the target data field from the source system to the target system, the data before and after migration may be compared to ensure consistency and integrity of the data.
According to the scheme, the data table of the source system is monitored continuously, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table; if the change of the data table of the source system is monitored, the corresponding data table change information is acquired; determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information; and migrating the data in the target data field from the source system to a target system. In this embodiment, a migration manner based on the data field is adopted, and migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
Further, referring to fig. 3, a flow chart is provided in a second embodiment of the data migration method of the present application, based on the embodiment shown in fig. 2, the source system is provided with a source database, the source database stores data corresponding to the plurality of data fields, and the "continuous monitoring on a data table of the source system" in step S10 is further refined, including:
step S101, continuously obtaining a transaction log file of the source database, and analyzing the transaction log file to obtain data table operation information.
Specifically, the transaction log file of the source database records all operations such as insertion, update, deletion and the like of the source database, and when the data table in the data field of the source database changes, the data table is firstly reflected in the transaction log file. In order to monitor the data table of the source system, it is necessary to continuously acquire the transaction log file of the source database, parse the transaction log file, and convert binary or text data in the transaction log file into readable data table operation information. Wherein the data table operation information includes a table name (or table identification) of the data table related to the operation, an operation type, a time stamp of the operation, and the like.
Step S102, according to the data table operation information, continuous monitoring is carried out on the data table of the source system.
Specifically, comparing the operation information of the data table obtained this time with the operation information of the data table obtained last time can determine whether a changed data table exists, so that continuous monitoring of the data table of the source system can be realized.
If the operation information of the data table obtained at this time is consistent with the operation information of the data table obtained last time, determining that the data table which is changed does not exist, and not monitoring that the data table of the source system is changed; if the operation information of the data table obtained at this time is inconsistent with the operation information of the data table obtained at last time (for example, the operation information of the data table obtained at this time is recorded by a part more), the changed data table is determined to exist, and the change of the data table of the source system is monitored.
According to the embodiment, through the scheme, the real-time sensing of the data change of the source system is ensured, so that the real-time performance and accuracy of data migration are realized. By continuously monitoring the transaction log file of the source database, details of the data table operation including insertion, updating, deletion and the like can be immediately captured without waiting for a specific time window, so that the data migration can respond to changes more quickly, delay of the data migration is reduced, interference to the service is reduced, and meanwhile, the reliability of the data migration is improved.
Further, referring to fig. 4, a flow chart is provided in a third embodiment of the data migration method of the present application, based on the embodiment shown in fig. 2, the step S30 of determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields is further refined, including:
step S301, extracting a target data table name corresponding to the changed data table from the data table change information.
Step S302, determining the target data field corresponding to the changed data table from the plurality of data fields according to the target data table name.
Specifically, the data table change information is analyzed, and the table name corresponding to the changed data table, namely the target data table name, can be extracted from the data table change information.
The preset subordinate relation between the data table and the data field is actually embodied in the corresponding relation between the data table name and the data field number, and the number of the target data field corresponding to the target data table name can be found according to the corresponding relation between the target data table name and the number of the data table name and the data field number, so that the target data field corresponding to the changed data table is determined.
According to the scheme, the change of the data table can be accurately determined according to the change information of the data table, so that the target data field needing to be migrated is defined, the accuracy of data migration is improved, unnecessary migration operation is avoided, and resources and time are saved. Meanwhile, the migration and verification can be performed in a targeted manner by defining the target data field, so that the efficiency and manageability of data migration are improved. This helps to reduce the risk of service interruption, ensuring the success and accuracy of data migration.
Further, referring to fig. 5, a flow chart is provided according to a fourth embodiment of the data migration method of the present application, based on the embodiment shown in fig. 2, step S20, after acquiring corresponding data table change information if it is monitored that the data table of the source system changes, step S30, before determining, according to the data table change information, a target data field corresponding to the changed data table from the plurality of data fields, further includes:
and S001, generating a message corresponding to the data table change information, and storing the message corresponding to the data table change information into a preset distributed publishing and subscribing message kafka system.
And step S002, if the message corresponding to the data table change information issued by the kafka system is monitored, analyzing the message corresponding to the data table change information to obtain the restored data table change information.
The step S30 of "determining, according to the data table change information, the target data field corresponding to the changed data table from the plurality of data fields" further includes:
step S303, determining the target data field corresponding to the changed data table from the data fields according to the restored data table change information.
Specifically, in order to improve timeliness and scalability of the data migration process, the present embodiment introduces a distributed publish-subscribe message kafka system. The kafka system is directed to processing real-time data streams that employ a publish-subscribe model that allows multiple producers to publish messages to topics and allows multiple consumers to subscribe to these topics to receive messages. The Kafka has the characteristics of high throughput, scalability, fault tolerance and durability, is suitable for large-scale data processing, log collection, event-driven architecture and other applications, and is widely used for constructing real-time data pipelines in various scenes to realize reliable transmission and processing of data streams.
After the data table change information is acquired, data migration is not directly performed according to the data table change information, but a message corresponding to the data table change information is generated, and the message corresponding to the data table change information is stored in the kafka system. Storing messages corresponding to the change information of the data table in the Kafka system enables other components to subscribe to the messages and respond accordingly.
On the other hand, the message issued by the kafka system may be continuously monitored (i.e. subscribed to the kafka system), and if the message corresponding to the data table change information issued by the kafka system is monitored, the message corresponding to the data table change information may be further parsed to obtain the restored data table change information. Then, according to the restored data table change information, the target data field corresponding to the changed data table can be determined from the plurality of data fields.
According to the embodiment, through the scheme, a Kafka system is introduced to provide a reliable message transmission mechanism, the data table change information is issued and transmitted in a message form, the expandability and the stability of the system are enhanced, and the reliable transmission of the data change information is ensured. Meanwhile, kafka allows multiple components to monitor and consume the messages in real time, so that a distributed and real-time event-driven mode is realized, and data migration can respond to changes of a source system more efficiently and in real time. This helps to reduce the delay of data migration, improving the efficiency and reliability of data migration.
Further, referring to fig. 6, a flow chart is provided in a fifth embodiment of the data migration method of the present application, based on the embodiment shown in fig. 2, the "migrating data in the target data area from the source system to the target system" in step S40 is further refined, and includes:
step S401, analyzing the hierarchical relationship of the target data field, and determining a root node corresponding to the target data field.
Step S402, according to the root node corresponding to the target data field, obtaining the data corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system.
In particular, the data field typically has a tree structure including a root node, an intermediate node, and leaf nodes, and this tree structure represents the hierarchical relationship of nodes in the data field. Wherein the types of the nodes include: (1) root node: the root node is a top level node of the data field hierarchical structure, is independent of other nodes, and generally represents one data field and is a starting point of other nodes of the data field; (2) an intermediate node: intermediate nodes are located between the root node and leaf nodes, which may depend on the root node or other intermediate nodes, and may contain child nodes. (3) leaf node: leaf nodes are end nodes of the hierarchy that typically no longer contain other child nodes.
By analyzing the hierarchical relationship of the target data field, the root node corresponding to the target data field can be determined, and the root node corresponding to the target data field is the starting point of the hierarchical structure of the whole target data field, so that the organization and understanding of the structure and relationship of the target data field are facilitated.
Then, according to the root node corresponding to the target data field, the data (namely, the whole data) corresponding to the target data field can be obtained, and the data corresponding to the target data field is further migrated from the source system to the target system, so that the data migration of the data field level can be realized. Compared with the conventional table-to-table stock data migration, the incremental data migration based on the data field adopted in the embodiment has the following advantages: (1) The data migration in the data field layer is finer and more accurate, the consistency and the integrity of the migrated data and the service field can be ensured, and the risk of inconsistent data is reduced. (2) Data migration in the data field layer allows data to be organized according to business logic, so that comprehensibility and maintainability of the data are improved, and continuity of business processes is facilitated. (3) The data migration in the data field layer is more flexible, the rapid adaptation is allowed when the service requirement changes, and the dependence on a source system is reduced. (4) By dividing the data according to the fields, incremental migration can be realized, performance overhead and interruption are reduced, and migration efficiency is improved.
According to the scheme, the starting point of the data migration operation, namely the root node of the domain, can be determined by analyzing the hierarchical relationship of the target data domain. The root node is found to help to transfer the data in the whole target data field in a targeted manner, so that the accuracy and the integrity of data transfer are improved, all relevant data can be successfully transferred, and missing or inconsistent data is avoided.
Further, referring to fig. 7, a flowchart is provided in a sixth embodiment of the data migration method of the present application, based on the embodiment shown in fig. 6, the number of the target data fields is a plurality, and for "according to the root node corresponding to the target data field, obtaining the data corresponding to the target data field and migrating the data corresponding to the target data field from the source system to the target system" in step S402, further refining the data includes:
step S4021, generating corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and storing the asynchronous tasks into a preset asynchronous task queue.
Step S4022, based on the plurality of asynchronous tasks, obtaining data corresponding to each of the plurality of target data fields through a preset Java persistence interface, and migrating the data corresponding to the target data fields from the source system to the target system.
Specifically, the number of target data fields may be several, and in order to improve concurrency, responsiveness and fault tolerance of data migration, the embodiment introduces a plurality of asynchronous tasks generated in the process of managing data migration by using an asynchronous task list.
Firstly, corresponding asynchronous tasks are needed to be generated according to root nodes corresponding to the target data fields, and it is understood that one-to-one correspondence exists between the asynchronous tasks and the target data fields. Then, a plurality of asynchronous tasks are stored in a preset asynchronous task queue, wherein the asynchronous task queue is a data structure for managing and scheduling the execution sequence of the asynchronous tasks.
Further, it is necessary to execute a number of asynchronous tasks among the asynchronous task queues, which are asynchronously executable. In the execution process, firstly, a plurality of root nodes corresponding to the target data fields are found, and all data of the target data fields can be found along the root nodes. Therefore, the preset Java persistence interface (Java Persistence API) can be used for acquiring data corresponding to each of a plurality of target data fields and migrating the data corresponding to the target data fields from the source system to the target system. In this way, incremental data migration based on the data domain is completed.
According to the scheme, asynchronous tasks in the multiple target data fields are processed in parallel by introducing the asynchronous task queue, data migration among different target data fields is not interfered with each other, and the overall efficiency of data migration is improved. In addition, the Java persistence interface is used for acquiring data, so that the Java persistence interface can interact with different databases and data formats more flexibly, and applicability and maintainability are enhanced.
Further, referring to fig. 8, a flow chart is provided in a seventh embodiment of the data migration method of the present application, based on the embodiment shown in fig. 2, step S40, after migrating the data in the target data area from the source system to the target system, further includes:
step S003, obtaining a key detail index corresponding to the target data field.
Step S004, according to the key detail index, acquiring first-type key detail data of the target data field from the source system and acquiring second-type key detail data of the target data field from the target system.
And S005, comparing the first type of key detail data with the second type of key detail data to obtain comparison result information.
Step S006, storing the comparison result information into a preset comparison record table.
Specifically, to ensure accuracy, integrity, and consistency of data, data verification is also required after data migration is completed.
Firstly, a key detail index corresponding to the target data field can be acquired, wherein the key detail index is a standard or rule for measuring the quality and accuracy of data. For example, for the policy data field, the key detail indicators may include: (1) policy number consistency: ensuring that the unique identifier of each policy is correct and consistent. (2) insurance amount accuracy: and verifying whether the insurance amount is consistent with the contract, so as to avoid the wrong insurance policy amount. (3) date of validation integrity: check whether each policy has a valid validation date. (4) business rules specified by insurance companies: ensuring that the policy meets certain requirements of the company, such as payment period, insurance type, etc.
Further, according to the key detail index, first-type key detail data of the target data field are acquired from the source system, and second-type key detail data of the target data field are acquired from the target system. It will be appreciated that the first type of critical detail data corresponds to the data before migration and the second type of critical detail data corresponds to the data after migration.
The first type of critical detail data and the second type of critical detail data are compared to determine if there is a discrepancy or inconsistency between them. Specifically, it is checked whether the respective fields of the two sets of data match, for example, whether the values of the fields of the name, address, telephone number, etc. of one customer are the same.
If no abnormal condition is found in the comparison process, normal comparison result information of the data is obtained, and the normal comparison result information of the data is stored in a preset comparison record table. Similarly, if an abnormal situation is found in the comparison process, such as abnormal situations of data loss, data format error, data out of range and the like, the abnormal situation is further recorded to obtain data abnormal comparison result information, and the data abnormal comparison result information is stored in a preset comparison record table. The comparison record table can provide reference information for subsequent examination, analysis and problem solving, and ensures the quality of data migration.
According to the embodiment, the link of data verification is specifically introduced, and the accuracy and consistency of data migration can be verified by acquiring the first type of key detail data and the second type of key detail data. And the storage of the comparison result information in the comparison record table is beneficial to tracking and recording the quality of data migration, finding potential problems in advance and repairing the potential problems when needed. The method is beneficial to ensuring the quality and consistency of the migrated data, reducing the risk of entering the target system by the error data and improving the reliability of the whole data migration.
Further, referring to fig. 9, an eighth embodiment of a data migration method of the present application provides a flowchart.
Specifically, firstly, a transaction log file of a source database is continuously acquired, the transaction log file is analyzed to obtain data table operation information, and the data table of the source system can be continuously monitored according to the data table operation information.
If the change of the data table of the source system is monitored, the corresponding data table change information can be further acquired. Then, a message corresponding to the data table change information is generated, the message corresponding to the data table change information is stored in the kafka system, and the message issued by the kafka system is continuously monitored. And the information corresponding to the data table change information can be further analyzed and restored to obtain the data table change information when the information corresponding to the data table change information is monitored.
The data table change information records the table name (or name of the data table) of the changed data table. According to the table names of the changed data tables and the corresponding relation between the predetermined data table names and the data fields, the target data fields corresponding to the changed data tables can be determined. Further, a root node of the target data domain can be determined according to the hierarchical relationship of the target data domain.
Since the root node is the starting point of all nodes in the target data domain, the root node in the target data domain can migrate the whole data in the target data domain from the source system to the target system. Alternatively, the number of target data fields may be plural, and the migration process may be performed in an asynchronous task manner.
After the data migration is completed, a first type of critical detail data for the target data field may be obtained from the source system, and a second type of critical detail data for the target data field may be obtained from the target system. And further comparing the first type of key detail data with the second type of key detail data, so that data verification of the target data field can be completed. In this way, data consistency before and after migration can be ensured.
In this embodiment, a migration manner based on the data field is adopted, and migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
In addition, an embodiment of the present application further provides a data migration apparatus, where the data migration apparatus includes:
The monitoring module is used for continuously monitoring the data table of the source system, wherein the source system comprises a plurality of preset data fields, and any one data field comprises at least one corresponding data table;
the acquisition module is used for acquiring corresponding data table change information if the data table of the source system is monitored to change;
the determining module is used for determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information;
and the migration module is used for migrating the data in the target data field from the source system to the target system.
The principle and implementation process of data migration are implemented in this embodiment, please refer to the above embodiments, and are not repeated here.
In addition, the embodiment of the application also provides a terminal device, which comprises a memory, a processor and a data migration program stored on the memory and capable of running on the processor, wherein the data migration program realizes the steps of the data migration method when being executed by the processor.
Because the data migration program is executed by the processor and adopts all the technical schemes of all the embodiments, the data migration program at least has all the beneficial effects brought by all the technical schemes of all the embodiments and is not described in detail herein.
In addition, the embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores a data migration program, and the data migration program realizes the steps of the data migration method when being executed by a processor.
Because the data migration program is executed by the processor and adopts all the technical schemes of all the embodiments, the data migration program at least has all the beneficial effects brought by all the technical schemes of all the embodiments and is not described in detail herein.
Compared with the prior art, the data migration method, the device, the terminal equipment and the storage medium provided by the embodiment of the application are used for continuously monitoring the data table of the source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table; if the change of the data table of the source system is monitored, the corresponding data table change information is acquired; determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information; and migrating the data in the target data field from the source system to a target system. Based on the scheme, a migration mode based on the data field is adopted, and the migration is performed only when the data table changes by continuously monitoring the change of the data table of the source system. Therefore, incremental migration can be realized, the influence on the service is reduced while the performance cost is reduced, and the efficiency and the reliability of data migration are improved.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as above, including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to perform the method of each embodiment of the present application.
The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the claims, and all equivalent structures or equivalent processes using the descriptions and drawings of the present application, or direct or indirect application in other related technical fields are included in the scope of the claims of the present application.
Claims (10)
1. A data migration method, characterized in that the data migration method comprises:
continuously monitoring a data table of a source system, wherein the source system comprises a plurality of preset data fields, and any data field comprises at least one corresponding data table;
if the change of the data table of the source system is monitored, the corresponding data table change information is acquired;
determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information;
and migrating the data in the target data field from the source system to a target system.
2. The data migration method of claim 1, wherein the source system is provided with a source database, the source database stores data corresponding to the plurality of data fields, and the step of continuously monitoring the data table of the source system includes:
Continuously acquiring a transaction log file of the source database, and analyzing the transaction log file to obtain data table operation information;
and continuously monitoring the data table of the source system according to the data table operation information.
3. The data migration method of claim 1, wherein the step of determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information comprises:
extracting a target data table name corresponding to the changed data table from the data table change information;
and determining the target data field corresponding to the changed data table from the plurality of data fields according to the target data table name.
4. The data migration method of claim 1, wherein after the step of acquiring the corresponding data table change information if it is monitored that the data table of the source system is changed, the step of determining, from the plurality of data fields, the target data field corresponding to the changed data table according to the data table change information further comprises:
generating a message corresponding to the data table change information, and storing the message corresponding to the data table change information into a preset distributed publishing and subscribing message kafka system;
If the message corresponding to the data table change information issued by the kafka system is monitored, analyzing the message corresponding to the data table change information to obtain restored data table change information;
the step of determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information comprises the following steps:
and determining the target data field corresponding to the changed data table from the plurality of data fields according to the restored data table change information.
5. The data migration method of claim 1, wherein the step of migrating data of the target data domain from the source system to the target system comprises:
analyzing the hierarchical relationship of the target data field, and determining a root node corresponding to the target data field;
and acquiring data corresponding to the target data field according to the root node corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system.
6. The data migration method of claim 5, wherein the number of the target data fields is a plurality, and the steps of obtaining the data corresponding to the target data field according to the root node corresponding to the target data field, and migrating the data corresponding to the target data field from the source system to the target system include:
Generating corresponding asynchronous tasks according to the root nodes corresponding to the target data fields respectively, and storing the asynchronous tasks into a preset asynchronous task queue;
based on the plurality of asynchronous tasks, acquiring data corresponding to each of the plurality of target data fields through a preset Java persistence interface, and migrating the data corresponding to the target data fields from the source system to the target system.
7. The data migration method of claim 1, wherein after the step of migrating the data of the target data field from the source system to the target system, further comprising:
acquiring a key detail index corresponding to the target data field;
acquiring first-type key detail data of the target data field from the source system and second-type key detail data of the target data field from the target system according to the key detail index;
comparing the first type of key detail data with the second type of key detail data to obtain comparison result information;
and storing the comparison result information into a preset comparison record table.
8. A data migration apparatus, the data migration apparatus comprising:
The monitoring module is used for continuously monitoring the data table of the source system, wherein the source system comprises a plurality of preset data fields, and any one data field comprises at least one corresponding data table;
the acquisition module is used for acquiring corresponding data table change information if the data table of the source system is monitored to change;
the determining module is used for determining the target data field corresponding to the changed data table from the plurality of data fields according to the data table change information;
and the migration module is used for migrating the data in the target data field from the source system to the target system.
9. A terminal device, characterized in that it comprises a memory, a processor and a data migration program stored on the memory and executable on the processor, which data migration program, when executed by the processor, implements the steps of the data migration method according to any one of claims 1-7.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a data migration program, which when executed by a processor, implements the steps of the data migration method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311531019.4A CN117667885A (en) | 2023-11-16 | 2023-11-16 | Data migration method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311531019.4A CN117667885A (en) | 2023-11-16 | 2023-11-16 | Data migration method, device, terminal equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117667885A true CN117667885A (en) | 2024-03-08 |
Family
ID=90076179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311531019.4A Pending CN117667885A (en) | 2023-11-16 | 2023-11-16 | Data migration method, device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117667885A (en) |
-
2023
- 2023-11-16 CN CN202311531019.4A patent/CN117667885A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8938421B2 (en) | Method and a system for synchronizing data | |
| CN107818431B (en) | Method and system for providing order track data | |
| CN108874558B (en) | Message subscription method of distributed transaction, electronic device and readable storage medium | |
| US10338958B1 (en) | Stream adapter for batch-oriented processing frameworks | |
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| CN111078504A (en) | Distributed call chain tracking method and device, computer equipment and storage medium | |
| CN109460841B (en) | User account opening method, system and storage medium | |
| CN109800207B (en) | Log analysis method, device and equipment and computer readable storage medium | |
| CN108647357B (en) | Data query method and device | |
| US9391825B1 (en) | System and method for tracking service results | |
| CN112559475B (en) | Data real-time capturing and transmitting method and system | |
| CN112905323B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111367792A (en) | Test method, test device, storage medium and electronic equipment | |
| CN114416868B (en) | Data synchronization method, device, equipment and storage medium | |
| CN109587351B (en) | Call testing method, device, equipment and storage medium | |
| CN111159142B (en) | Data processing method and device | |
| CN112597123B (en) | Data multi-version dynamic switching method and device | |
| CN113760677A (en) | Abnormal link analysis method, device, equipment and storage medium | |
| US8406401B2 (en) | Interactive voice response system to business application interface | |
| CN113672776B (en) | Fault analysis method and device | |
| CN117667885A (en) | Data migration method, device, terminal equipment and storage medium | |
| KR101888131B1 (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| CN115757304A (en) | Log storage method, device and system, electronic equipment and storage medium | |
| CN113220530B (en) | Data quality monitoring method and platform | |
| CN112597119A (en) | Method and device for generating processing log and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |