CN112256666B - Logic increment migration method - Google Patents
Logic increment migration method Download PDFInfo
- Publication number
- CN112256666B CN112256666B CN202010931781.1A CN202010931781A CN112256666B CN 112256666 B CN112256666 B CN 112256666B CN 202010931781 A CN202010931781 A CN 202010931781A CN 112256666 B CN112256666 B CN 112256666B
- Authority
- CN
- China
- Prior art keywords
- data
- logic
- bill
- migration
- migrated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a logic increment migration method, which comprises the following steps: determining a logic range of data to be migrated according to specific business logic to obtain a logic data block; removing old data corresponding to the logic range in the target table; and migrating the data in the logic data block to the target table. The logical increment migration method provided by the embodiment of the invention can realize lossless small-batch increment data migration without time stamp, and does not need to damage or modify the existing service system.
Description
Technical Field
The invention relates to the technical field of cloud data migration, in particular to a logic increment migration method.
Background
In the process from digitization to intelligent transformation, the mine needs cloud data migration (Cloud Data Migration, CDM for short) for service system data acquisition and lake entering based on a big data integration platform.
The existing data migration methods mainly comprise two types, namely full migration and incremental migration. Full volume migration is suitable for first, disposable or less than 10 ten thousand level data migration. Incremental migration requires a time stamp on the migrated library table to control the data set boundaries each time.
However, many service systems, logistics systems, production systems, shop performance systems, etc. currently in use in mines, most of the tables to be migrated do not contain time stamp fields, and if fields are added or modified, normal operation of the original service system is affected or abnormal processing logic is caused. At present, when full migration is used, the magnitude of data sets in a table is huge, and the network bandwidth, the platform computing power, the migration efficiency and the energy consumption are affected.
Disclosure of Invention
The invention aims to provide a logic increment migration method, which performs data migration under the condition of no time stamp, does not damage or modify the existing service system, can avoid the problem of huge data volume caused by using full-volume migration, reduces bandwidth consumption, saves platform computing capacity and improves migration efficiency.
In order to solve the technical problems, the embodiment of the invention provides the following scheme:
a method of logical delta migration, comprising the steps of:
determining a logic range of data to be migrated according to specific business logic to obtain a logic data block;
removing old data corresponding to the logic range in the target table;
and migrating the data in the logic data block to the target table.
Preferably, the logical data block includes changed data and unchanged data of the two migration intervals in the logical range, wherein the changed data includes new data, modified data and deleted data.
Preferably, the step of determining the logic range of the data to be migrated according to the specific service logic to obtain the logic data block specifically includes:
extracting commonalities of the change data to be migrated, and summarizing a logic range condition containing all commonalities;
verifying the logic range condition, and verifying whether the logic range condition is complete;
and supplementing the logic range condition, amplifying the limited range or reducing the limited range according to the verification result.
Preferably, the step of extracting commonalities of the change data to be migrated and summarizing a logic range condition including all commonalities specifically includes:
analyzing the change data to be migrated to obtain a data change rule, data aggregation characteristics and common characteristics of the data;
and according to the obtained data change rule, data aggregation characteristic and data common characteristic, a logic range condition containing all commonalities is generalized.
Preferably, the step of verifying the logic range condition specifically includes:
obtaining migration frequency;
judging whether data is missed when the logic data block is migrated next time according to the migration frequency;
if the omission occurs, the logic range condition is enlarged, so that the logic data block contains all the variable data.
Preferably, the step of verifying the logical range condition further comprises:
and confirming the generalized logic range condition with a service operator to obtain a verification result.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:
the logical increment migration method provided by the embodiment of the invention can realize lossless small-batch increment data migration without time stamp, does not need to damage or modify the existing service system, can reduce the magnitude of migration data volume, can reduce bandwidth consumption, saves platform computing capacity and greatly improves migration efficiency compared with full-volume migration.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for logical delta migration provided in an embodiment of the present invention;
FIG. 2 is a diagram of a pure delta migration range in accordance with an embodiment of the present invention;
FIG. 3 is a schematic diagram of a logical delta migration scope in an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a comparison of data transfer processes between logical delta migration and full delta migration according to an embodiment of the present invention;
FIG. 5 is a diagram of a full-scale migration execution time record in accordance with an embodiment of the present invention;
FIG. 6 is a schematic diagram of a record of execution time of a logical delta migration in an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
An embodiment of the present invention provides a logical delta migration method, as shown in fig. 1, including the following steps:
s1, determining a logic range of data to be migrated according to specific business logic to obtain a logic data block.
The method of the invention changes the quasi-incremental migration evolution into the logic data block containing the change data to migrate, and reasonably sets a logic range according to specific business logic under the condition of not changing the structure of the prior source database, wherein the change data all occur in the logic range.
Taking a warehouse-out bill of a mine service system as an example, the method specifically comprises the following steps:
s101, extracting commonalities of change data to be migrated, and summarizing a logic range condition (where) containing all commonalities, wherein the commonalities are analyzed by service types, occurrence time and other angles;
s102, verifying the generalized logic range condition (where) to verify whether the logic range condition (where) is complete;
s103, supplementing a logic range condition (where), amplifying a limited range or reducing the limited range according to the verification result.
Firstly, analyzing change data to be migrated to obtain a data change rule, data aggregation characteristics and common characteristics of data; and then, the obtained data change rule, data aggregation characteristic and common data characteristic are generalized and limited in a set of conditions with a certain logic range, and only abstract generalized commonalities are not specific to a certain bill. Because logical incremental data migration is an unattended scheduling task, logical data blocks should not be defined for individual documents.
Specifically, the logical data block includes changed data and unchanged data of the two migration intervals in the logical range, wherein the changed data includes new data, modified data and deleted data.
The change data of the bill of the mine business system mainly comprises the new bill (newly added bill), the bill modification (changed bill) and the bill deletion (deleted bill). FIG. 2 is a diagram of a pure delta migration scope, including only new add bill, change bill, and delete bill, all during one accounting period, i.e., the current accounting period; FIG. 3 is a schematic diagram of a logical delta migration range according to an embodiment of the present invention, which includes not only newly added bill, changed bill, and deleted bill, but also a subset of unchanged documents during the current accounting period.
It can be seen that if the logic range condition is determined as "current accounting period", then, during data migration, the unchanged documents in the graph are migrated together, so that partial redundancy is formed. However, in general, when the logic range condition in the embodiment is determined as the bill of the "current accounting period", the data volume to be migrated is slightly larger than the pure incremental migration mode, but is still far smaller than the full migration mode, so that the selected logic range condition is reasonable and effective.
Further, the step of verifying the logic range condition specifically includes:
obtaining migration frequency; judging whether the data is missed when the logic data block is migrated next time according to the migration frequency; if a miss occurs, the logic range condition is expanded so that the logic data block contains all the variable data.
The step of verifying the logical range condition further comprises:
and confirming the generalized logic range condition with a service operator to obtain a verification result.
Specifically, the step of verifying is performed to compensate for the conditional omission problem of the logical data block range that has been generalized. For example, assuming that the current period is 202008, the range of logical data blocks that are abstracted is where (period=202008), and the frequency of system migration is scheduled once a day, the first migration will occur 24 hours later after migrating the delivery form in 202008 to the target table, the next time the logical data blocks are migrated. Within this 24 hour period, the business system will for some reason check out the system to 202007 during accounting, modify and adjust the ticket for the accounting period of 202007, save after adjustment, and check out the system to 202008 again. At this time, the bill is changed during the accounting period of 202007 existing in the business system, and at this time, when the migration out of the bill form is scheduled and executed again according to the determined logic range where (period=202008), omission occurs, and the real synchronization of the data is not achieved. At this time, it is necessary to expand the sphere condition, and to divide all the delivery order data in approximately two periods into logical data blocks and migrate the delivery order data to the target table.
Through communication with business personnel, it was determined that anti-checkout occurred occasionally during one period of the day, but that operations to modify the date of arrival during 2 or more accounting periods were never performed in succession. It is concluded that: the final generalized supplemented logic conditions were: where (period=near two accounting periods).
S2, removing old data corresponding to the logic range in the target table.
FIG. 4 is a schematic diagram illustrating a comparison of data transfer processes between logical delta migration and full delta migration according to an embodiment of the present invention. The data clearing step of the target library when the data is migrated is as follows:
for the full-volume migration mode, all data in the target table is cleared, the table is subjected to truncate operation, the table is an empty table after the operation, and then the source table data is migrated in full volume.
For the delta migration mode, since the new migration is pure delta data, no piece (batch) of data exists in the target table, so that any data is not deleted.
For the logical incremental migration mode, because the migrated data volume has a certain redundancy, the migration mode not only comprises pure variable data, but also comprises non-variable data in a logical range. Before the part of data is migrated into the target table, the old data in the part of logic range in the target table should be deleted, the logic range condition is defined by a where clause, and if the logic range condition extracted from the source table is where a=b, the logic range condition to be deleted in the target table should be consistent with where a=b. However, when determining the logical range condition of a=b, it is ensured that the change data to be migrated belongs to a subset of the set of a=b.
For example: in the case of a job of migrating a job of a job ticket in a warehouse management system, only the job in the current period has change data, and in the task scheduling with a migration frequency of days, the logical range of a logical data block to be migrated is determined to be that only the current period of a document (during accounting, yyyy year mm period is often expressed, for example, 2020 year 08 period) is extracted. Because the current documents include both those documents that were changed in all source tables and those that were not changed at the current time (this is the redundant part of the logical data block of the current migration). Therefore, when the logic increment is migrated, all documents in the current period in the target library need to be deleted, and then the migrated logic data block is inserted.
S3, migrating the data in the logic data block to a target table.
The logical increment migration method provided by the embodiment of the invention can realize lossless small-batch increment data migration without time stamp, does not need to damage or modify the existing service system, can reduce the magnitude of migration data volume, can reduce bandwidth consumption, saves platform computing capacity and greatly improves migration efficiency compared with full-volume migration.
In the following, the execution process of the logical incremental migration and the execution process of the full migration are compared, fig. 5 is a schematic diagram of the execution time record of the full migration, and fig. 6 is a schematic diagram of the execution time record of the logical incremental migration according to an embodiment of the present invention.
As can be seen from a comparison of fig. 5 and 6, the execution time used 6 minutes and 4 seconds for the full migration, and 11386487 records were migrated altogether; the scheduling algorithm of the scheduling task job_ic_stock adopts a logic increment data migration method, only the data quantity in the near two accounting periods needs to be migrated during logic debugging, the time is 14 seconds, 201553 line records are migrated, the time is accelerated (6 minutes 60+4 seconds)/14 seconds=26 times before and after, 11184934 line records of network transmission quantity are saved, 11184934/11386487 =98.23% of transmission quantity is saved, and the lifting effect is very remarkable.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (1)
1. A method of logical delta migration, comprising the steps of:
s1, determining a logic range of data to be migrated according to specific business logic to obtain a logic data block;
the logic data block comprises change data and unchanged data of a twice migration interval in the logic range, wherein the change data comprises new data, modified data and deleted data;
the step of determining the logic range of the data to be migrated according to the specific business logic to obtain the logic data block specifically comprises the following steps:
extracting commonalities of the change data to be migrated, and summarizing a logic range condition containing all commonalities;
verifying the logic range condition, and verifying whether the logic range condition is complete;
supplementing the logic range condition, amplifying the limited range or reducing the limited range according to the verification result;
the step of extracting commonalities of the change data to be migrated and summarizing a logic range condition including all commonalities specifically includes:
analyzing the change data to be migrated to obtain a data change rule, data aggregation characteristics and common characteristics of the data;
according to the obtained data change rule, data aggregation characteristic and data common characteristic, a logic range condition containing all commonalities is generalized;
the logic incremental data migration is an unattended scheduling task, and logic data blocks are not defined for individual documents;
the step of verifying the logic range condition specifically includes:
obtaining migration frequency;
judging whether data is missed when the logic data block is migrated next time according to the migration frequency;
if omission occurs, expanding the logic range condition to enable the logic data block to contain all change data;
in the delivery bill of the mine service system, the step S1 specifically includes:
s101, extracting commonalities of change data to be migrated, and summarizing a logic range condition where all commonalities are included, wherein the commonalities are analyzed by service types and occurrence time angles;
s102, verifying the generalized logic range condition where, and verifying whether the logic range condition where is complete;
s103, supplementing the logic range condition where, amplifying the limited range or reducing the limited range according to the verification result;
for the bill of the mine service system, the change data comprises bill newly added, bill modified, bill changed and bill deleted; the pure delta migration range only comprises a new bill, a changed bill and a deleted bill, which are all in one accounting period, namely the current accounting period; the logical increment migration range not only comprises a new bill, a changed bill and a deleted bill, but also comprises an unchanged bill subset in the current accounting period;
when the logic range condition is determined as a bill in the current accounting period, the data volume to be migrated is larger than the pure increment migration mode, but smaller than the full quantity migration mode;
the step of verifying the logic range condition specifically includes:
expanding a sphere condition, classifying all the ex-warehouse data in nearly two periods into a logic data block, and migrating the ex-warehouse data to a target table;
the step of verifying the logical range condition further comprises:
confirming the induced logic range condition with a service operator to obtain a verification result;
through communication with business personnel, it is determined that anti-checkout occurs during one period in the course of a day, but operations for modifying the data of the past period during 2 or more accounting periods of continuous anti-checkout never occur; it is concluded that: the final generalized supplemented logic conditions were: where period = near two accounting periods;
s2, removing old data corresponding to the logic range in the target table;
the step S2 specifically comprises the following steps:
the data clearing step of the target table when the data is migrated is as follows:
for the logic increment migration mode, because the migrated data volume has a certain redundancy, the migration mode not only comprises pure variable data, but also comprises non-variable data in a logic range; before the part of data is migrated into the target table, the old data in the part of logic range in the target table should be deleted, the logic range condition is defined by a where clause, and if the logic range condition extracted from the source table is where a=b, the logic range condition to be deleted in the target table should be consistent with where a=b; but in determining the logical range condition of a=b, it is ensured that the change data to be migrated belongs to a subset of the set of a=b;
when the delivery bill task is migrated, only the delivery bill in the current period has change data, and in the task scheduling with the day as the migration frequency, the logic range of the logic data block to be migrated is determined to be only used for extracting the current period bill; the current bill comprises all the bills changed in the source list and also comprises the bill which is not changed in the current period, namely the redundant part in the logical data block of the current migration; therefore, when the logic increment is migrated, deleting all documents in the current period in the target table, and then inserting the migrated logic data block;
s3, migrating the data in the logic data block to the target table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010931781.1A CN112256666B (en) | 2020-09-08 | 2020-09-08 | Logic increment migration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010931781.1A CN112256666B (en) | 2020-09-08 | 2020-09-08 | Logic increment migration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112256666A CN112256666A (en) | 2021-01-22 |
| CN112256666B true CN112256666B (en) | 2023-06-16 |
Family
ID=74231177
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010931781.1A Active CN112256666B (en) | 2020-09-08 | 2020-09-08 | Logic increment migration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112256666B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156738A (en) * | 2011-04-13 | 2011-08-17 | 成都市华为赛门铁克科技有限公司 | Method for processing data blocks, and data block storage equipment and system |
| CN103399936A (en) * | 2013-08-09 | 2013-11-20 | 神州数码(中国)有限公司 | Incremental data synchronization method |
| CN108073688A (en) * | 2017-11-20 | 2018-05-25 | 苏宁云商集团股份有限公司 | A kind of method and device of Data Migration |
| CN110674108A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data processing method and device |
| CN111125214A (en) * | 2019-12-02 | 2020-05-08 | 武汉虹信技术服务有限责任公司 | Lightweight incremental data synchronization method and device and computer readable medium |
| CN111198914A (en) * | 2019-12-12 | 2020-05-26 | 山西云时代技术有限公司 | Whole database real-time data acquisition method based on oracle database filing log |
-
2020
- 2020-09-08 CN CN202010931781.1A patent/CN112256666B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156738A (en) * | 2011-04-13 | 2011-08-17 | 成都市华为赛门铁克科技有限公司 | Method for processing data blocks, and data block storage equipment and system |
| CN103399936A (en) * | 2013-08-09 | 2013-11-20 | 神州数码(中国)有限公司 | Incremental data synchronization method |
| CN108073688A (en) * | 2017-11-20 | 2018-05-25 | 苏宁云商集团股份有限公司 | A kind of method and device of Data Migration |
| CN110674108A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data processing method and device |
| CN111125214A (en) * | 2019-12-02 | 2020-05-08 | 武汉虹信技术服务有限责任公司 | Lightweight incremental data synchronization method and device and computer readable medium |
| CN111198914A (en) * | 2019-12-12 | 2020-05-26 | 山西云时代技术有限公司 | Whole database real-time data acquisition method based on oracle database filing log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112256666A (en) | 2021-01-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110879813B (en) | Binary log analysis-based MySQL database increment synchronization implementation method | |
| CN112445863B (en) | Data real-time synchronization method and system | |
| CN102915336A (en) | Incremental data capturing and extraction method based on timestamps and logs | |
| CN103678392A (en) | Data increment and merging method and device for achieving method | |
| CN102591864B (en) | Data updating method and device in comparison system | |
| US20230067182A1 (en) | Data Processing Device and Method, and Computer Readable Storage Medium | |
| CN113141410B (en) | Dynamically adjusted QPS control method, system, device and storage medium | |
| CN110609860A (en) | Data ETL processing method, device, equipment and storage medium | |
| CN103235811A (en) | Data storage method and device | |
| CN101770461B (en) | Data processing method and data processing system | |
| CN113961546B (en) | Real-time query library design method supporting online analysis and statistics | |
| CN112256666B (en) | Logic increment migration method | |
| CN109918623A (en) | A kind of Policy Updates compare management system automatically | |
| CN104601374A (en) | Network fault processing method and device for numerical control machine tool | |
| CN106599113B (en) | Database reading and writing method for mass performance data of network management system | |
| CN114722045A (en) | Time series data storage method and device | |
| CN104809216A (en) | Analysis model and engine system based on data mining | |
| CN116414902B (en) | Quick data source access method | |
| CN113626516A (en) | Data increment synchronization method and system | |
| CN112632173A (en) | ETL-based due diligence data analysis system and method under mass data | |
| CN111061702A (en) | Storage service method based on big data | |
| CN117235051B (en) | Database management method and device, electronic equipment and storage medium | |
| CN117591577B (en) | Nuclear power historical data comparison method and system based on file storage | |
| CN113360461B (en) | Method and storage medium for pushing overdue data to receiving system for analysis | |
| CN112612648B (en) | SQL Server database recovery method, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |