CN110851421B - Method and device for reducing time consumption of data migration, storage medium and electronic equipment - Google Patents
Method and device for reducing time consumption of data migration, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN110851421B CN110851421B CN201911078332.0A CN201911078332A CN110851421B CN 110851421 B CN110851421 B CN 110851421B CN 201911078332 A CN201911078332 A CN 201911078332A CN 110851421 B CN110851421 B CN 110851421B
- Authority
- CN
- China
- Prior art keywords
- log
- data migration
- data
- preset
- name
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005012 migration Effects 0.000 title claims abstract description 222
- 238000013508 migration Methods 0.000 title claims abstract description 221
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000004590 computer program Methods 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for reducing data migration time, a storage medium, and an electronic device. The method comprises the following steps: acquiring a data updating log table, and recording the table name of the log table after the data updating is successful in a first table; comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table; inquiring whether a table name of a log table needing data migration in the first table is recorded in a second table, wherein the table name of the log table needing data migration is recorded in the second table; performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to perform data migration in the first table; and recording the table name of the log table subjected to the data migration in the second table. The invention reduces the total time of data migration on the premise of ensuring the integrity and timeliness of data.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for reducing time consumption of data migration, a storage medium, and an electronic device.
Background
With the continuous development of computer hardware and software, under the attraction of large data capacity and expression, the usage amount and exchange amount of data are rapidly increased, and data migration can be performed in the data use and exchange process, but the total time of data migration cannot be reduced on the premise of ensuring the integrity and timeliness of the data in the prior art.
Disclosure of Invention
In view of the above problems, the present application provides a method, an apparatus, a storage medium, and an electronic device for reducing time consumed by data migration, which solve the problem that the total time of data migration cannot be reduced on the premise of ensuring integrity and timeliness of data in the prior art.
In a first aspect, the present application provides a method for reducing time consumption of data migration, the method comprising:
step S110, acquiring a data updating log table, and recording the table name of the log table after the data updating is successful in a first table;
step S120, comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table;
step S130, inquiring whether a second table records the table name of the log table needing data migration in the first table, wherein the second table records the table name of the log table with completed data migration;
step S140, performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to be subjected to data migration in the first table;
step S150, record the table name of the log table in which the data migration has been performed in the second table.
According to an embodiment of the present application, optionally, in the method for reducing data migration time consumption, comparing the first table with a preset table includes: and traversing the table names of all the log tables in a preset table, and comparing the table names with the table names of the log tables in the first table one by one to obtain the table name of the log table needing data migration in the first table and the table name of the log table not needing data migration in the preset table.
According to an embodiment of the present application, optionally, in the method for reducing time consumed by data migration, after obtaining a table name of a log table in the preset table, where data migration is not performed, the method further includes: recording the table name coverage of the log table without data migration in the preset table in a third table; re-acquiring the data updating log table, and recording the table name of the log table after the data updating is successful in the first table in a covering manner; comparing the covered first table with a preset table to obtain a table name of a log table which needs data migration in the covered first table; and performing data migration on the data corresponding to the table names of all log tables which need to be subjected to data migration in the covered first table and the covered third table.
According to an embodiment of the present application, optionally, the method for reducing time consumption of data migration further includes: and when the table names of all log tables needing data migration in the first table are recorded in the second table, returning to execute the step S110.
According to an embodiment of the present application, optionally, the method for reducing time consumption of data migration further includes:
and after the table name of the log table which needs to be subjected to data migration in the first table is obtained, performing data migration on the data corresponding to the table name of the log table which needs to be subjected to data migration in the first table.
According to an embodiment of the present application, optionally, the method for reducing time consumption of data migration further includes: comparing the second table with the preset table, and judging whether the table names of all the log tables in the preset table are recorded in the second table; and when the table names of all the preset log tables are recorded in the second table, outputting prompt information that the data migration is completed.
According to an embodiment of the present application, optionally, the method for reducing time consumption of data migration further includes: and when the table names of all the log tables in the preset are not recorded in the second table, returning to execute the step S110.
In a second aspect, the present application provides an apparatus for reducing time consumption of data migration, including:
an acquisition module configured to acquire a data update log table;
the processing module is configured to judge whether the state of the data updating log table is successful or not, compare the first table with a preset table, and inquire whether a table name of the log table needing data migration in the first table is recorded in a second table or not;
the execution module is configured to perform data migration on data corresponding to the table name of the log table, and record the table name of the log table after the data migration is completed in the second table;
the storage module is configured to store the data update log, the first table, the preset table, the second table and the third table.
In a third aspect, the present application provides a storage medium storing a computer program executable by one or more processors for implementing a method for reducing the time consumption of data migration as described above.
In a fourth aspect, the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program, and the computer program, when executed by the processor, performs the above method for reducing the time consumption of data migration.
Compared with the prior art, one or more embodiments in the above scheme can have the following advantages or beneficial effects:
according to the method, the device, the storage medium and the electronic equipment for reducing the time consumption of data migration, the data updating log table is obtained, and the table name of the log table after the data updating is successful is recorded in the first table; comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table; inquiring whether a table name of a log table needing data migration in the first table is recorded in a second table, wherein the table name of the log table needing data migration is recorded in the second table; performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to perform data migration in the first table; and recording the table name of the log table subjected to the data migration in the second table. On the premise of ensuring the integrity and timeliness of data, the overall time of data migration is reduced.
Drawings
The present application will be described in more detail hereinafter on the basis of embodiments and with reference to the accompanying drawings:
fig. 1 is a schematic flowchart illustrating a method for reducing time consumption of data migration according to an embodiment of the present disclosure;
fig. 2 is another schematic flowchart of a method for reducing time consumption of data migration according to an embodiment of the present disclosure;
fig. 3 is another schematic flowchart of a method for reducing time consumption of data migration according to an embodiment of the present disclosure;
fig. 4 is another schematic flowchart of a method for reducing time consumption of data migration according to an embodiment of the present disclosure;
FIG. 5 is another schematic flow chart illustrating a method for reducing time consumption of data migration according to an embodiment of the present disclosure;
FIG. 6 is another schematic flow chart illustrating a method for reducing time consumption of data migration according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an apparatus for reducing time consumption of data migration according to an embodiment of the present disclosure.
In the drawings, like parts are designated with like reference numerals, and the drawings are not drawn to scale.
Detailed Description
The following detailed description will be provided with reference to the accompanying drawings and embodiments, so that how to apply the technical means to solve the technical problems and achieve the corresponding technical effects can be fully understood and implemented. The embodiments and various features in the embodiments of the present application can be combined with each other without conflict, and the formed technical solutions are all within the scope of protection of the present application.
According to the method, the device, the storage medium and the electronic equipment for reducing the time consumption of data migration, the data updating log table is obtained, and the table name of the log table after the data updating is successful is recorded in the first table; comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table; inquiring whether a table name of a log table needing data migration in the first table is recorded in a second table, wherein the table name of the log table needing data migration is recorded in the second table; performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to perform data migration in the first table; and recording the table name of the log table subjected to the data migration in the second table. The problem that the total time of data migration cannot be reduced on the premise of ensuring the integrity and timeliness of data in the prior art is solved.
Example one
Fig. 1 is a schematic diagram of a method for reducing time consumption of data migration according to an embodiment of the present application, as shown in fig. 1, the method includes the following steps:
s110: and acquiring a data updating log table, and recording the table name of the log table after the data updating is successful in the first table. The method specifically comprises the following steps: presetting a timing task, when the timing task starts timing execution, acquiring a data updating log table from a relational database, wherein the log table records a table name and an updating state, the updating state only has two states of updating success and updating failure, and the table name of the log table with the updating state being successful is recorded in a first table.
The migration data source refers to the most original application data, a data table below an application layer or backed up, and specific file storage modes such as a relational database, an Oracle (Oracle RDBMS), an sql Server (Structured Query Language), and the like are generally consistent with the data processing mode. Data volume of migration data source: hundreds of thousands, millions, and billions of services vary. Time span of migrating data: the time span from the beginning of the service to the moment is the time span of the data of the table, but different service cut-in angles and statistical modes are different, and the time span is not completely consistent but is necessarily between the beginning time and the end time of the service.
Creating a new table in the relational database, wherein the new table specially stores the table to be updated, the table name, the starting time and the ending time of table updating and the table updating state are recorded in the upstream database, and the new table obtains each table updating state, namely the data updating log table, from the upstream database log table.
The data migration object has a data table or a data view, but both may be called as a data source, and in this embodiment, the data migration object is collectively referred to as a table.
Presetting a timing task for updating data, and when the timing task starts to time, executing step S110, where the timing task may be a timing task in which service data is updated, or may be a timing task in which service data is not updated, but it must be noted that the timing task is a time task in which the latest data in the service data table exists. For example, once a day update, it must be maintained once a day, and it cannot occur twice or more a day.
S120: comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the table names of all preset log tables needing data migration are recorded in the preset table. The method specifically comprises the following steps: and traversing the table names of all the log tables in the preset table, and comparing the table names with the table names of the log tables in the first table one by one to obtain the table name of the log table needing data migration in the first table and the table name of the log table not needing data migration in the preset table.
The table names of all log tables which are recorded in the preset table and need to be subjected to data migration are preset according to the needs in the service.
After the table name of the log table without data migration in the preset table is obtained, the method further includes: recording the table name coverage of the log table without data migration in the third table; re-acquiring the data updating log table, and recording the table name of the log table after the data updating is successful in the first table in a covering manner; comparing the covered first table with a preset table to obtain a table name of a log table which needs data migration in the covered first table; and performing data migration on the data corresponding to the table names of all log tables which need to be subjected to data migration in the first covered table and the third covered table.
As shown in fig. 2, a data update log table is obtained, and the table name of the data update success is recorded in table a; comparing the table A with the table B to obtain the table name of the log table without data migration in the table B; re-acquiring a data updating log, recording the table name of the log table after the data updating is successful in the table A in a covering manner, and comparing the covered table A with the table B to obtain the table name of the log table which needs to be subjected to data migration in the covered table A; and performing data migration on the data corresponding to the table names of all log tables needing data migration in the covered table D. Wherein, table A is a first table, table B is a preset table, and table D is a third table.
S130: and inquiring whether the second table records the table name of the log table needing data migration in the first table, wherein the second table records the table name of the log table with completed data migration. The method comprises the following steps:
and opening the second table by using a tool carried by Linux, dividing the letters into the individual table names by using tools such as a pipeline and the like, comparing the table names recorded in the second table with the table names in the first table one by one, and judging whether all the table names in the first table are subjected to data migration to obtain the table names which are not subjected to data migration and the table names which are subjected to data migration in the first table. Data migration is avoided from being repeated.
And performing data migration after querying the table name of the log table in the second table, wherein whether data migration is performed on the data corresponding to the table name of the log table depends on whether the table name of the log table is recorded in the second table, that is, the table name of the log table with data migration completed is recorded in the second table, so as to confirm whether data migration has been performed on the data corresponding to the table name of the log table with data migration performed each cycle. And performing data migration on the data corresponding to the table names of the log tables one by one in the background to achieve the purpose of performing data migration on a plurality of data simultaneously, reducing the total time consumed by data migration, monitoring the running state of each file until the data corresponding to the table names of all the log tables recorded in the preset table are completed, and finishing the data migration.
S140: and performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to be subjected to data migration in the first table. The method comprises the following steps:
and performing data migration on data corresponding to the table names of all log tables needing data migration in the first table according to the table name information recorded in the second table.
Wherein, the data to be migrated is placed in the database and not in the table. The process of performing data migration includes: and after obtaining an updated table name, storing data corresponding to the table name by using a shell command carried by the Linux, and migrating the data, wherein each table name corresponds to data needing data migration. And judging whether the previous data migration command is finished by using the shell command of the Linux system, and monitoring the state of data migration.
S150: and recording the table name of the log table subjected to the data migration in the second table. The method specifically comprises the following steps: and storing the table name of the log table subjected to data migration in the second table, so as to avoid repeated circulation.
The second table records the table name of the log table with completed data migration, that is, the second table records the table name of the log table with the current data migration state of "completed", the data corresponding to the table with completed data migration does not need to be migrated again, and only the data corresponding to the table without data migration is migrated, so that the data migration of the same data is avoided.
For example, as shown in fig. 3, a data update log table is obtained, the table name of the log table in which data update is successful is recorded in table a, table a and table B are compared to obtain the table name of the log table in which data migration is required in table a, and whether all the table names of the log tables in which data migration is required in table a are recorded in table C is queried.
When the table names of all the log tables in the table a requiring data migration are recorded in the table C, the step S110 is executed in return.
When the table names of all the log tables which need to be subjected to data migration in the table A are not recorded in the table C, performing data migration on data which are not recorded in the table A and correspond to the table names of the log tables which need to be subjected to data migration in the table C; the table name of the log table in which the data migration has been performed is recorded in table C. Wherein, table A is a first table, table B is a preset table, and table C is a second table.
At S120: comparing the first table with a preset table to obtain the table name of the log table needing data migration in the first table, the method further comprises the following steps: and performing data migration on data corresponding to the table name of the log table needing data migration in the first table, and recording the table name of the log table subjected to data migration in the second table.
As shown in fig. 4, a data update log table is obtained, the table name of the log table with successful data update is recorded in table a, and the table a and the table B are compared to obtain the table name of the log table with data migration required in table a; and performing data migration on the data corresponding to the table name of the log table needing data migration in the table A, and recording the table name of the log table subjected to data migration in the table C. Wherein, table A is a first table, table B is a preset table, and table C is a second table.
At S150: after recording the table name of the log table subjected to the data migration in the second table, the method further comprises the following steps: comparing the second table with a preset table, and judging whether the table names of all log tables in the preset table are recorded in the second table; when the table names of all log tables in the preset table are recorded in the second table, outputting prompt information that the data migration is completed; when the table names of all the log tables in the preset table are not recorded in the second table, the step S110 is executed in a return manner.
As shown in fig. 5, a data update log table is obtained, the table name of the log table in which data update is successful is recorded in table a, table a and table B are compared to obtain the table name of the log table in which data migration is required in table a, and whether all the table names of the log tables in which data migration is required in table a are recorded in table C is queried.
When the table names of all the log tables in the table a requiring data migration are recorded in the table C, the step S110 is executed in return.
When the table names of all the log tables which need to be subjected to data migration in the table A are not recorded in the table C, performing data migration on data which are not recorded in the table A and correspond to the table names of the log tables which need to be subjected to data migration in the table C; the table name of the log table in which the data migration has been performed is recorded in table C.
Comparing the table C with the table B, and judging whether the table C records the table names of all the log tables in the table B; when the table names of all the log tables in the table B are recorded in the table C, outputting prompt information that the data migration is completed; when the table names of all the log tables in the table B are not recorded in the table C, the execution returns to the step S110. Wherein, table A is a first table, table B is a preset table, and table C is a second table. On the premise of ensuring the correctness and the technical performance of data, the overall time of data migration is reduced.
Example two
Fig. 6 is another flowchart illustrating a method for reducing time consumption of data migration according to an embodiment of the present disclosure. As shown in fig. 6, the update log Table is acquired, the Table names Table1, Table2, and Table3 with the update status "successful" are acquired, the Table names Table1, Table2, and Table3 are recorded in Table a, Table a and Table B are compared, data corresponding to Table1, Table2, and Table3 in Table a needs to be migrated, and Table4 and Table5 are the Table names of the log Table in Table B that needs to be migrated. Wherein, the table B records the table name of the log table that needs data migration: table1, Table2, Table3, Table4, and Table 5.
Splitting tables 1, 2 and 3 into single Table names, performing data migration on data corresponding to tables 1, 2 and 3 respectively, and recording tables 1, 2 and 3 which finish the data migration in a Table C; table C records the table name of the log table in which data migration has been performed.
Querying whether Table1, Table2 and Table3 are recorded in Table C, comparing Table C with Table B when Table1, Table2 and Table3 are recorded in Table C, outputting prompt information that data migration is completed when the Table names of the log Table recorded in Table C and the log Table recorded in Table B are all Table1, Table2, Table3, Table4 and Table5, otherwise, continuously acquiring the updated log Table, taking the Table names Table1, Table2 and Table3 whose update state is "successful", recording the Table names Table1, Table2 and Table3 in Table a, and circulating.
Storing Table4 and Table5 into Table D, acquiring the update log Table again, taking the Table names Table1, Table2, Table3, Table4 and Table5 with the update state of "success", and recording the Table names Table1, Table2, Table3, Table4 and Table5 in the Table A in a covering manner. In this case, Table1, Table2, Table3, Table4 and Table5 are recorded in Table a. Comparing the Table a with the Table B again, data corresponding to Table1, Table2, Table3, Table4 and Table5 in the Table a needs to be migrated, and data corresponding to Table1, Table2, Table3, Table4 and Table5 recorded in the Table a and each Table name in Table4 and Table5 recorded in the Table D needs to be migrated. Storing the tables 1, 2, 3, 4 and 5 which complete data migration into the Table C, comparing the Table C with the Table B, outputting the prompt information that the data migration is completed when the Table names of the log Table recorded in the Table C and the log Table recorded in the Table B are all Table1, Table2, Table3, Table4 and Table5, otherwise, continuously acquiring the updated log Table, taking the Table names of Table1, Table2 and Table3 whose update state is "successful", recording the Table names of Table1, Table2 and Table3 into the Table a, and circulating. On the premise of ensuring the integrity and timeliness of data, the overall time of data migration is reduced.
EXAMPLE III
Fig. 7 is a schematic structural diagram of an apparatus 20 for reducing time consumption of data migration according to an embodiment of the present application, and as shown in fig. 7, an apparatus 20 for reducing time consumption of data migration includes:
an obtaining module 21 configured to obtain a data update log table;
the processing module 22 is configured to determine whether the status of the data update log table is successful, compare the first table with a preset table, and query whether a table name of a log table needing data migration in the first table is recorded in the second table;
the execution module 23 is configured to perform a data migration program on data corresponding to the table name of the log table, and record the table name of the log table after the data migration is completed in the second table;
the storage module 24 is configured to store the data update log, the first table, the preset table, the second table, and the third table.
The obtaining module 21 obtains a data update log table, the processing module 22 determines whether the state of the data update log table is successful, the storage module 24 stores the log table in the first table, the processing module 22 compares the first table with a preset table and inquires whether a table name of the log table in the first table, which needs data migration, is recorded in the second table, the executing module 23 performs data migration on data corresponding to the table name of the log table according to a processing result of the processing module 22, and records the table name of the log table after data migration in the second table.
The specific embodiment process of the above method steps can be referred to as embodiment one, and the detailed description of this embodiment is not repeated herein.
Example four
The present embodiments also provide a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App, etc., having stored thereon a computer program that when executed by a processor, performs the following method steps:
step S110, acquiring a data updating log table, and recording the table name of the log table after the data updating is successful in a first table;
step S120, comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table;
step S130, inquiring whether a table name of a log table needing data migration in the first table is recorded in a second table, wherein the table name of the log table with the data migration completed is recorded in the second table;
step S140, data migration is carried out on data which are not recorded in the first table and correspond to the table names of the log tables needing data migration in the second table;
step S150, record the table name of the log table in which the data migration has been completed in the second table.
The specific embodiment process of the above method steps can be referred to as embodiment one, and the detailed description of this embodiment is not repeated herein.
EXAMPLE five
An embodiment of the present application provides an electronic device, which may be a mobile phone, a computer, or a tablet computer, and the electronic device includes a memory and a processor, where the memory stores a computer program, and the computer program is executed by the processor to reduce time consumption of data migration as in the first embodiment. It is understood that the electronic device may also include multimedia components, input/output (I/O) interfaces, and communication components.
The processor is configured to perform all or part of the steps of the method for reducing the time consumption of data migration according to the first embodiment. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.
The Processor may be an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, and is configured to perform the method for reducing the time consumption of data migration in the first embodiment.
The Memory may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk.
In summary, according to the method, the apparatus, the storage medium, and the electronic device for reducing the time consumption of data migration provided by the present application, the data update log table is obtained, and the table name of the log table after the data update is successful is recorded in the first table; comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table; inquiring whether a table name of a log table needing data migration in the first table is recorded in a second table, wherein the table name of the log table needing data migration is recorded in the second table; manual checking is reduced, and working efficiency is improved. Performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to perform data migration in the first table; the method and the device run in the background without waiting for the program to finish execution and then execute the next program, thereby reducing the waiting of data migration tasks. And recording the table name of the log table subjected to the data migration in the second table. Repeated execution of executed data migration tasks is avoided. By dynamically detecting the updating state of the data of the service layer and timely following the step of data migration to the data warehouse, the total time consumed by data migration is greatly reduced, and the problem of queuing of migration tasks is reduced on the premise of ensuring the integrity and the effectiveness of the data in the data migration process.
In the embodiments provided in the present application, it should be understood that the method disclosed in the present invention can be implemented in other ways. The above-described method embodiments are merely illustrative.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Although the embodiments disclosed in the present application are described above, the above descriptions are only for the convenience of understanding the present application, and are not intended to limit the present application. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims.
Claims (8)
1. A method for reducing the time consumed by data migration, the method comprising:
step S110, acquiring a data updating log table, and recording the table name of the log table after the data updating is successful in a first table;
step S120, comparing the first table with a preset table to obtain a table name of a log table needing data migration in the first table; the method comprises the steps that a preset table is recorded in a preset table, and table names of all log tables needing data migration are recorded in the preset table;
step S130, inquiring whether a second table records the table name of the log table needing data migration in the first table, wherein the second table records the table name of the log table with completed data migration;
step S140, performing data migration on data corresponding to the table name of the log table which is not recorded in the second table and needs to be subjected to data migration in the first table;
step S150, recording the table name of the log table with the completed data migration in the second table;
wherein comparing the first table to a preset table comprises:
traversing table names of all log tables in a preset table, and comparing the table names with the table names of the log tables in the first table one by one to obtain the table names of the log tables needing data migration in the first table and the table names of the log tables not needing data migration in the preset table;
after the table name of the log table without data migration in the preset table is obtained, the method further includes:
recording the table name coverage of the log table without data migration in the preset table in a third table;
re-acquiring the data updating log table, and recording the table name of the log table after the data updating is successful in the first table in a covering manner;
comparing the covered first table with a preset table to obtain a table name of a log table which needs data migration in the covered first table;
and performing data migration on the data corresponding to the table names of all log tables which need to be subjected to data migration in the covered first table and the covered third table.
2. The method of claim 1, further comprising:
and when the table names of all log tables needing data migration in the first table are recorded in the second table, returning to execute the step S110.
3. The method of claim 1, further comprising:
and after the table name of the log table which needs to be subjected to data migration in the first table is obtained, performing data migration on the data corresponding to the table name of the log table which needs to be subjected to data migration in the first table.
4. The method of claim 1, further comprising:
comparing the second table with the preset table, and judging whether the table names of all the log tables in the preset table are recorded in the second table;
and when the table names of all the preset log tables are recorded in the second table, outputting prompt information that the data migration is completed.
5. The method of claim 4, further comprising:
and when the table names of all the log tables in the preset are not recorded in the second table, returning to execute the step S110.
6. An apparatus for reducing the time consumed by data migration, comprising:
an acquisition module configured to acquire a data update log table;
the processing module is configured to judge whether the state of the data updating log table is successful, compare the first table with a preset table, and inquire whether a table name of the log table needing data migration in the first table is recorded in a second table;
the execution module is configured to perform data migration on data corresponding to the table name of the log table, and record the table name of the log table after the data migration is completed in the second table;
a storage module configured to store a data update log, a first table, a preset table, a second table, and a third table;
wherein, comparing the first table with the preset table comprises:
traversing table names of all log tables in a preset table, and comparing the table names with the table names of the log tables in the first table one by one to obtain the table names of the log tables needing data migration in the first table and the table names of the log tables not needing data migration in the preset table;
after the table name of the log table without data migration in the preset table is obtained, the method further includes:
recording the table name coverage of the log table without data migration in the preset table in a third table;
re-acquiring the data updating log table, and recording the table name of the log table after the data updating is successful in the first table in a covering manner;
comparing the covered first table with a preset table to obtain a table name of a log table which needs data migration in the covered first table;
and performing data migration on the data corresponding to the table names of all log tables which need to be subjected to data migration in the covered first table and the covered third table.
7. A storage medium storing a computer program executable by one or more processors for performing a method of reducing the time required for data migration as claimed in any one of claims 1 to 5.
8. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program that, when executed by the processor, performs the method of reducing the time consumed for data migration according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911078332.0A CN110851421B (en) | 2019-11-06 | 2019-11-06 | Method and device for reducing time consumption of data migration, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911078332.0A CN110851421B (en) | 2019-11-06 | 2019-11-06 | Method and device for reducing time consumption of data migration, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110851421A CN110851421A (en) | 2020-02-28 |
| CN110851421B true CN110851421B (en) | 2022-04-08 |
Family
ID=69598776
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911078332.0A Active CN110851421B (en) | 2019-11-06 | 2019-11-06 | Method and device for reducing time consumption of data migration, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110851421B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148714B (en) * | 2020-09-30 | 2023-12-05 | 珠海格力电器股份有限公司 | Data monitoring method, system, storage medium and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914458A (en) * | 2012-12-29 | 2014-07-09 | 中国移动通信集团河北有限公司 | Mass data migration method and device |
| CN105930493A (en) * | 2016-05-04 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Method and system for data synchronization between different databases |
| CN105956191A (en) * | 2016-06-13 | 2016-09-21 | 浪潮(北京)电子信息产业有限公司 | Data migration method and system |
| CN106598977A (en) * | 2015-10-15 | 2017-04-26 | 北京国双科技有限公司 | Data migration method and device |
| CN108304553A (en) * | 2018-02-01 | 2018-07-20 | 平安普惠企业管理有限公司 | Data migration method, device, computer equipment and storage medium |
| CN110413593A (en) * | 2019-06-19 | 2019-11-05 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5029113B2 (en) * | 2007-04-13 | 2012-09-19 | ソニー株式会社 | Backup system, backup device, backup request device, and backup method |
| US20120124081A1 (en) * | 2010-11-17 | 2012-05-17 | Verizon Patent And Licensing Inc. | Method and system for providing data migration |
-
2019
- 2019-11-06 CN CN201911078332.0A patent/CN110851421B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914458A (en) * | 2012-12-29 | 2014-07-09 | 中国移动通信集团河北有限公司 | Mass data migration method and device |
| CN106598977A (en) * | 2015-10-15 | 2017-04-26 | 北京国双科技有限公司 | Data migration method and device |
| CN105930493A (en) * | 2016-05-04 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Method and system for data synchronization between different databases |
| CN105956191A (en) * | 2016-06-13 | 2016-09-21 | 浪潮(北京)电子信息产业有限公司 | Data migration method and system |
| CN108304553A (en) * | 2018-02-01 | 2018-07-20 | 平安普惠企业管理有限公司 | Data migration method, device, computer equipment and storage medium |
| CN110413593A (en) * | 2019-06-19 | 2019-11-05 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110851421A (en) | 2020-02-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8996499B2 (en) | Using temporary performance objects for enhanced query performance | |
| WO2022143045A1 (en) | Method and apparatus for determining data blood relationship, and storage medium and electronic apparatus | |
| US20160125022A1 (en) | Efficient maintenance of column store indexes on memory-optimized tables | |
| US20170329836A1 (en) | Database transfer of changes | |
| WO2019109854A1 (en) | Data processing method and device for distributed database, storage medium, and electronic device | |
| CN108573019B (en) | Data migration method and device, electronic equipment and readable storage medium | |
| CN109885642B (en) | Hierarchical storage method and device for full-text retrieval | |
| CN101093454A (en) | Method and device for executing SQL script file in distributed system | |
| CN110807145A (en) | Query engine acquisition method, device and computer-readable storage medium | |
| US20220004664A1 (en) | Data integrity procedure | |
| CN110851421B (en) | Method and device for reducing time consumption of data migration, storage medium and electronic equipment | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| CN109344164B (en) | Data storage method and device | |
| CN113065084B (en) | Data loading method and device, computer equipment and storage medium | |
| CN109829678B (en) | Rollback processing method and device and electronic equipment | |
| CN107247784B (en) | Distributed transaction control method and transaction manager | |
| US20220229821A1 (en) | Data restoration using dynamic data structure altering | |
| CN116521363B (en) | Code packaging method, computer equipment and storage medium | |
| CN116842244A (en) | Search engine data synchronization method, system, device and storage medium | |
| CN110633300A (en) | Breakpoint continuous operation method and device for nested query | |
| CN116089518A (en) | Data model extraction method and system, terminal and medium | |
| US20230359494A1 (en) | Disabling of memory allocators | |
| CN112765126B (en) | Database transaction management method, device, computer equipment and storage medium | |
| CN114564621A (en) | Method, device and equipment for associating data and readable storage medium | |
| WO2022001627A1 (en) | Data clearing method and apparatus, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |