CN109933632B - Data migration method, device and equipment for database - Google Patents
Data migration method, device and equipment for database Download PDFInfo
- Publication number
- CN109933632B CN109933632B CN201910271970.8A CN201910271970A CN109933632B CN 109933632 B CN109933632 B CN 109933632B CN 201910271970 A CN201910271970 A CN 201910271970A CN 109933632 B CN109933632 B CN 109933632B
- Authority
- CN
- China
- Prior art keywords
- data
- source unit
- unit
- database
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data migration method, a device, equipment and a computer readable storage medium of a database, wherein the method comprises the following steps: determining a source unit and a destination unit; performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to a source unit, and acquiring identification information corresponding to all data currently contained in the source unit; performing an unlock operation on the source unit to allow data to be written to the source unit; reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; and writing the incremental data of the source unit into the destination unit in real time after the query command is issued based on the identification information. The data migration method ensures the data consistency of the target unit and the source unit, and greatly reduces the influence of data migration on the database service continuity.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a method, an apparatus, a device, and a computer-readable storage medium for migrating data of a database.
Background
The database is a very core component in computer software, and when the application scale is gradually increased, the service is gradually wide, the scale of the database is also gradually increased, and the requirement on the performance is also higher and higher. The single database node may not satisfy the upper layer application in terms of service implementation, performance and service reliability, and thus a database real-time synchronization technology is used.
Disclosure of Invention
The invention aims to provide a method, a device and equipment for data migration of a database and a computer readable storage medium, which can realize the real-time synchronization technology of the database on the premise of avoiding influencing the service continuity of the database needing to realize the data migration.
In order to achieve the above purpose, the invention provides the following technical scheme:
a method of data migration of a database, comprising:
determining a storage unit to which data migration needs to be realized as a source unit, and determining a storage unit to which data needs to be migrated in the source unit as a target unit;
performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to the source unit, and acquiring identification information corresponding to all data currently contained in the source unit;
performing an unlock operation on the source unit to allow data to be written to the source unit; reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; writing the incremental data of the source unit into the target unit in real time after the query command is issued based on the identification information; the database corresponding to the source unit has a multi-version concurrency characteristic that data contained in the read data set is consistent with data contained in the source unit when the search command is issued.
Preferably, the obtaining of the identification information corresponding to all data currently contained in the source unit includes:
inquiring and recording the current system change number value of the database corresponding to the source unit; wherein the system change number value is an identification value corresponding to all data that the corresponding time in the source unit has;
writing the incremental data of the source unit into the destination unit in real time after the query command is issued based on the identification information, wherein the incremental data comprises:
and writing the incremental data of the source unit correspondingly generated when the real-time system change number value of the database is larger than the recorded system change number value into the target unit in real time.
Preferably, writing the incremental data of the source unit into the destination unit in real time after issuing the query command includes:
and grabbing an incremental data log of the source unit after the query command is issued by using the LogMinr of the database, and writing corresponding incremental data into the target unit in real time based on the incremental data log.
Preferably, the method further comprises the following steps:
receiving a synchronization termination instruction input from the outside, and stopping executing the step of writing the incremental data of the source unit into the destination unit in real time under the trigger of the synchronization termination instruction.
Preferably, after writing the data set into the destination unit, the method further includes:
and outputting a prompt message that the stock data migration is finished.
Preferably, outputting a prompt that the stock data migration has been completed includes:
and sending the prompt information of the completed stock data migration to a terminal corresponding to the appointed user.
Preferably, the source unit and the destination unit are both database tables.
A data migration apparatus of a database, comprising:
a unit determination module to: determining a storage unit to which data migration needs to be realized as a source unit, and determining a storage unit to which data needs to be migrated in the source unit as a target unit;
a prepare migration module to: performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to the source unit, and acquiring identification information corresponding to all data currently contained in the source unit; performing an unlock operation on the source unit to allow writing of data to the source unit
A migration synchronization module to: reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; writing the incremental data of the source unit into the target unit in real time after the query command is issued based on the identification information; the database corresponding to the source unit has a multi-version concurrency characteristic that data contained in the read data set is consistent with data contained in the source unit when the search command is issued.
A data migration apparatus of a database, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data migration method of the database as described in any one of the above when the computer program is executed.
A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data migration method of a database according to any one of the preceding claims.
In the data migration scheme of the database disclosed by the invention, when the data of a source unit needs to be migrated to a destination unit, the source unit is subjected to locking operation and unlocking operation, a search command for inquiring all the data of the source unit is issued between two operations, and identification information corresponding to all the data currently contained in the source unit is acquired, so that no new data is written into the source unit in the process of indicating the source unit to inquire all the data and acquire the corresponding identification information through issuing the search command, and the consistency between the data corresponding to the identification information and the data inquired by indicating the source unit through issuing the search command is ensured; then reading a data set obtained by inquiring of the source unit and writing the data into the destination unit, wherein the data contained in the data set obtained by reading is all data contained in the source unit when the inquiry command is issued because the database corresponding to the source unit has the MVCC characteristic; and because the identification information corresponds to all data contained in the source unit when the query command is issued, the incremental data generated by the source unit relative to the queried data set can be determined based on the identification information, and then the incremental data is written into the destination unit. Therefore, the technical scheme disclosed by the application can ensure the consistency of the data in the target unit and the data in the source unit, and the action of issuing the command of the command and acquiring the identification information is only needed to be realized between the locking operation and the unlocking operation, so that the time for writing the data service which cannot be provided by the source unit is very short, and even the perception of the upper-layer service cannot be realized, thereby greatly reducing the influence of data migration on the continuity of the database service.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data migration method for a database according to an embodiment of the present invention;
fig. 2 is an exemplary implementation process of a data migration method for a database according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data migration apparatus for a database according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to implement the database real-time synchronization technology in the related art, the source library service is suspended when the synchronization of the data in the database is completed in consideration of data consistency. Among them, a database to which data migration is to be implemented is referred to as a source library, and a database to which data of the source library is to be migrated is referred to as a destination library. Specifically, the upper layer writing service of the source library is suspended, and the data of the source library is exported and then imported into the target library. After the stock data migration is completed, the data of the two libraries are ensured to be consistent, incremental data synchronization is started, and the source library service is recovered. However, the influence degree of the method for suspending the source library service on the source library is positively correlated with the data volume, and the larger the data volume is, the longer the influence time on the source library is. Namely, the method has the problem of influencing the service continuity of the database needing to realize data migration.
Referring to fig. 1, a flowchart of a data migration method for a database according to an embodiment of the present invention is shown, where the method includes:
s11: determining a storage unit to which data migration needs to be realized as a source unit, and determining a storage unit to which data needs to be migrated in the source unit as a target unit.
It should be noted that the execution subject of the data migration method for the database provided in the embodiment of the present invention may be a corresponding data migration apparatus. The database consists of one or more data tables, each data table consists of one or more rows of records, and each row of records consists of one or more fields; correspondingly, the storage unit in the present application may refer to a database itself, or may refer to a data table or a record, and may be specifically set according to actual needs. And after determining the source unit needing to realize the data migration and the target unit needing to migrate the data of the source unit, starting to realize the step of migrating the data of the source unit to the target unit. The database in this embodiment may be an Oracle database, but is not limited thereto, and the embodiment is described by taking the Oracle database as an example.
S12: performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to a source unit, and acquiring identification information corresponding to all data currently contained in the source unit; an unlock operation is performed on the source unit to allow data to be written to the source unit.
Locking the source unit in order to block new data from falling into the source unit, i.e. to prohibit writing data into the source unit; then, issuing a search command to the source unit, instructing the source unit to search and return all the data of the source unit, and acquiring identification information corresponding to all the data currently contained in the source unit (the identification information may be any information corresponding to all the data currently contained in the source unit, which is set according to actual needs, such as a system Change number scn (system Change number) or a number corresponding to all the data currently contained in the source unit, and the like), therefore, through the locking operation, all the data of the source unit obtained through query are ensured to be the data from a single time point corresponding to the identification information, namely, through the locking and unlocking operation of the source unit, all the data contained in the source unit when the query command is issued to the source unit is ensured to be consistent with all the data contained in the source unit when the identification information corresponding to all the data currently contained in the source unit is acquired; and then, unlocking the source unit, thereby allowing the data to be written into the source unit and recovering the data writing service of the source unit. In addition, it should be noted that, after the source unit is locked, the source unit can be unlocked and fixed after the issuing of the query command and the acquiring of the identification information are completed, that is, in the present application, the issuing of the query command and the acquiring of the identification information only need to be completed between the locking operation and the unlocking and fixing operation of the source unit, and the issuing of the query command and the acquiring of the identification information are two simple actions requiring a very short time, so that the time required for locking the source unit is very short, correspondingly, the service continuity of the source unit only needs to be affected in the very short time, the basic perception of the upper service cannot be achieved, and the normal operation of the upper service can be ensured to a certain extent; in addition, only data writing to the source unit is prohibited during the time from the locking operation to the unlocking operation of the source unit, but the query service is not affected, so that the normal operation of the upper layer service is ensured from the viewpoint.
The locking operation and the unlocking operation of the source unit can be realized by issuing a locking command and an unlocking command to the source unit respectively, and the locking command and the unlocking command can be compiled and set according to actual needs; the following statements are used as an example for explanation, where the LOCK TABLE-Name IN SHARE MODE is a LOCK command, and meanwhile, the source unit may be set to be automatically unlocked after the following three statements are executed (i.e., after the query command issuing and the identification information obtaining are realized), that is, the following three statements are executed as an unlock command.
LOCK TABLE table-Name IN SHARE MODE；
SELECT CURRENT_SCN FROM V$DATABASE；
SELECT FROM table-Name. S13: reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; writing the incremental data of the source unit into the target unit in real time after issuing the search command based on the identification information; the database corresponding to the source unit has the multi-version concurrency characteristic that the data contained in the read data set is consistent with the data contained in the source unit when the search command is issued.
The source-unit-corresponding database has an MVCC characteristic (multi-version Concurrency characteristic) for making the data included in the read data set and the data included in the source unit when the search command is issued coincide with each other, and therefore the data set obtained by reading the source unit in response to the search command is also a data set of all data corresponding to the identification information. The MVCC characteristic is a characteristic of an oracle database, corresponding data consistency is realized based on the characteristic, and the read data is guaranteed to come from a single time point when an inquiry command is issued. Specifically, during the locking of the source unit, the source unit returns the query result data set after receiving the query command, and after the unlocking of the source unit, the data migration apparatus consumes the data set, i.e., reads the data set and writes the read result to the destination unit. Because the identification information is information corresponding to all data contained in the source unit at the corresponding moment, that is, all data contained in the source unit at the moment corresponding to the identification information can be determined as long as the identification information is determined, after all data contained in the source unit at the moment corresponding to the identification information is determined, all data contained in the source unit at the current moment can be compared with all data contained in the source unit at the moment corresponding to the identification information, and different data existing in all data contained in the source units at two moments is incremental data generated by the source unit after the moment corresponding to the identification information; meanwhile, the acquisition of the identification information and the issuing of the search command are both carried out when the source unit is locked and data writing into the source unit is prohibited, so that all data contained in the source unit at the time corresponding to the identification information and the search command are the same, and the incremental data generated by the source unit after the time corresponding to the identification information is also the incremental data generated by the source unit after the search command is issued, and then the incremental data is written into the target unit to realize the synchronization of the corresponding incremental data. Writing the incremental data into the destination unit is to synchronize the incremental data to the destination unit, and includes specific operations such as data addition, modification, deletion and the like. Therefore, stock data migration of the source unit is realized through the data set returned by the consumption source unit, and incremental data synchronization of the source unit is realized through synchronization of the incremental data.
In the data migration scheme of the database disclosed by the invention, when the data of a source unit needs to be migrated to a destination unit, the source unit is subjected to locking operation and unlocking operation, a search command for inquiring all the data of the source unit is issued between two operations, and identification information corresponding to all the data currently contained in the source unit is acquired, so that no new data is written into the source unit in the process of indicating the source unit to inquire all the data and acquire the corresponding identification information through issuing the search command, and the consistency between the data corresponding to the identification information and the data inquired by indicating the source unit through issuing the search command is ensured; then reading a data set obtained by inquiring of the source unit and writing the data into the destination unit, wherein the data contained in the data set obtained by reading is all data contained in the source unit when the inquiry command is issued because the database corresponding to the source unit has the MVCC characteristic; and because the identification information corresponds to all data contained in the source unit when the query command is issued, the incremental data generated by the source unit relative to the queried data set can be determined based on the identification information, and then the incremental data is synchronized to the target unit. Therefore, the technical scheme disclosed by the application can ensure the consistency of the data in the target unit and the data in the source unit, the data migration of the stock and the actions of issuing the command of checking and acquiring the identification information only need to be realized between the locking operation and the unlocking operation, so that the time for writing the data service which cannot be provided by the source unit is very short, and even the upper-layer service cannot be perceived, thereby greatly reducing the influence of the data migration on the continuity of the database service.
The data migration method for a database, provided by the embodiment of the present invention, acquiring identification information corresponding to all data currently contained in a source unit, may include:
inquiring and recording the current system change number value of the database corresponding to the source unit; wherein, the system change number value is an identification value corresponding to all data in the source unit at the corresponding moment;
writing the incremental data of the source unit into the destination unit in real time after issuing the search command based on the identification information, which may include:
and writing the incremental data of the source unit correspondingly generated when the real-time system change number value of the database is larger than the recorded system change number value into the target unit in real time.
It should be noted that, in the data migration method for a database provided in the embodiment of the present invention, the database may specifically be an Oracle database, and a System Change Number (SCN) is a Number that is automatically maintained by the DBMS to accumulate and increment after the Oracle database is updated, so that the value of the SCN increases every time the database is updated. In the application, the SCN value is used as identification information, the SCN value corresponding to the source unit before the source unit is unlocked is queried and recorded (the SCN of the source unit is the SCN of the database corresponding to the source unit), and it can be determined that data in the source unit is changed as long as the SCN value at any subsequent time is greater than the recorded SCN value, and at this time, corresponding incremental data is synchronized to the destination unit. Of course, other settings may be performed according to actual needs, such as using a timestamp as the identification information, which are within the protection scope of the present invention.
The data migration method for a database provided in the embodiments of the present invention writes incremental data of a source unit into a destination unit in real time after issuing a search command, and may include:
and grabbing an incremental data log of the source unit after issuing the searching command by using the LogMinr of the database, and writing corresponding incremental data into the target unit in real time based on the incremental data log.
It should be noted that, the specific process of implementing the real-time synchronization of the incremental data in the present invention may be selected according to actual needs, and the synchronization of the incremental data is implemented in the present embodiment by using a manner according to an incremental data log, so that the synchronization of the incremental data can be implemented quickly; the LogMinr is a software tool provided in the oracle database, and can capture logs in the oracle database, track transaction change operations, track table operations, and the like. And the LogMinr of the oracle database is used for capturing the incremental data logs so as to realize incremental data synchronization, and stock data migration is realized in a query mode, so that data migration and synchronous connection are realized on the premise of not invading the source unit (the source unit is not required to be installed with a stationing program, and the consumption of the stationing program on the performance of the source unit is avoided), and the structure in the source unit is not required to be used, so that the method has the advantage of structural independence of the source unit.
The data migration method for the database provided by the embodiment of the invention can further comprise the following steps:
receiving a synchronization termination instruction input from the outside, and stopping executing the step of writing the incremental data of the source unit into the destination unit in real time under the trigger of the synchronization termination instruction.
It should be noted that, if the outside needs to stop the incremental data synchronization, a synchronization termination instruction may be input to the data migration apparatus, and after receiving the synchronization termination instruction, the data migration apparatus stops the real-time synchronization of the incremental data of the source unit, so that an external worker can flexibly control the data migration, and the use experience of a user is improved.
In the data migration method for a database provided in the embodiment of the present invention, after writing the data set into the destination unit, the method may further include:
and outputting a prompt message that the stock data migration is finished.
After the stock data migration of the source unit is realized, the prompt information for finishing the stock data migration can be output to the outside, so that outside workers can timely know the information, and further can perform corresponding operation under the prompt of the information, for example, a synchronous termination instruction or other control instructions for a database are input, and the user experience is improved.
The data migration method for a database table provided in the embodiment of the present invention outputs the prompt information that stock data migration has been completed, and may include:
and sending the prompt information of the completed stock data migration to a terminal corresponding to the appointed user.
It should be noted that, a specific implementation manner of outputting the prompt information may be set according to actual needs, and in this embodiment, the prompt information is preferably sent to a terminal corresponding to a designated user, where the designated user may be an administrator or other user who needs to pay attention to the stock data migration process, so that the specific user can know the stock data migration condition in time, and user experience is further improved.
According to the data migration method of the database provided by the embodiment of the invention, both the source unit and the destination unit can be database tables.
It should be noted that the storage unit in the present invention may be a database table (may be referred to as a table for short) or a database (i.e., a database table set), and in the present application, the preferred storage unit is the database table, which not only meets the actual requirement for daily implementing data migration of the database, but also can avoid the situation that the data migration is required only by repeating multiple steps when implementing data migration by using records as a unit, thereby further facilitating data migration of the database.
In addition, the database in the data migration method of the database provided by the embodiment of the present invention may be an Oracle database (the database is fully called Oracle RDBMS), and the storage unit may be a database table, and at this time, the implementation process of the technical solution provided by the embodiment of the present invention may be as shown in fig. 2, where the precondition is to open an Oracle LogMinr of the source unit, so as to implement capture of an incremental data log by using the LogMinr at a later stage; the data table is a database table; the stock migration is stock data migration, and the incremental synchronization is incremental data synchronization. Specifically, the following steps may be included: locking the source table (locking the source unit); issuing a full table checking command (corresponding to the command for issuing all data of a query source unit), and inquiring and recording the SCN (corresponding to the recorded SCN value) of the oracle database corresponding to the source table; unlocking the source table (unlocking the source unit); the result set of the consumption and full-checking table is written into a target table (a data set obtained by corresponding acquisition and query is written into a target unit); and writing the SCN consumption increment log into a destination table (correspondingly writing the increment data of the source unit correspondingly generated when the real-time SCN value of the database is greater than the recorded SCN value into the destination unit in real time).
An embodiment of the present invention further provides a data migration apparatus for a database, as shown in fig. 3, which may include:
a unit determination module 11 configured to: determining a storage unit to which data migration needs to be realized as a source unit, and determining a storage unit to which data needs to be migrated in the source unit as a target unit;
a preparatory migration module 12 for: performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to a source unit, and acquiring identification information corresponding to all data currently contained in the source unit; performing an unlock operation on the source unit to allow data to be written to the source unit;
a migration synchronization module 13 configured to: reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; writing the incremental data of the source unit into the target unit in real time after issuing the search command based on the identification information; the database corresponding to the source unit has the multi-version concurrency characteristic that the data contained in the read data set is consistent with the data contained in the source unit when the search command is issued.
In an embodiment of the present invention, a data migration apparatus for a database, where a preparatory migration module includes:
a query logging unit to: inquiring and recording the current system change number value of the database corresponding to the source unit; wherein, the system change number value is an identification value corresponding to all data in the source unit at the corresponding moment;
the migration synchronization module may include:
a data synchronization unit to: and writing the incremental data of the source unit correspondingly generated when the real-time system change number value of the database is larger than the recorded system change number value into the target unit in real time.
In the data migration apparatus for a database provided in an embodiment of the present invention, the data synchronization unit may include:
a data synchronization subunit to: and grabbing an incremental data log of the source unit after issuing the searching command by using the LogMinr of the database, and writing corresponding incremental data into the target unit in real time based on the incremental data log.
The data migration apparatus for a database provided in the embodiments of the present invention may further include:
an instruction response module to: receiving a synchronization termination instruction input from the outside, and stopping executing the step of writing the incremental data of the source unit into the destination unit in real time under the trigger of the synchronization termination instruction.
The data migration apparatus for a database provided in the embodiments of the present invention may further include:
a prompt output module for: and after the data set is written into the destination unit, outputting prompt information that the stock data migration is finished.
In the data migration apparatus for a database provided in an embodiment of the present invention, the prompt output module may include:
a prompt output unit to: and sending the prompt information of the completed stock data migration to a terminal corresponding to the appointed user.
According to the data migration device for the database provided by the embodiment of the invention, both the source unit and the destination unit can be database tables.
An embodiment of the present invention further provides a data migration device for a database, which may include:
a memory for storing a computer program;
a processor for implementing the steps of the data migration method of the database as any one of the above when executing the computer program.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data migration method for a database as described above may be implemented.
It should be noted that for the description of the relevant parts in the data migration apparatus, the device and the computer readable storage medium of the database provided in the embodiment of the present invention, reference is made to the detailed description of the corresponding parts in the data migration method of the database provided in the embodiment of the present invention, and details are not repeated here. In addition, parts of the technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for migrating data of a database, comprising:
determining a storage unit to which data migration needs to be realized as a source unit, and determining a storage unit to which data needs to be migrated in the source unit as a target unit;
performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to the source unit, and acquiring identification information corresponding to all data currently contained in the source unit; performing an unlock operation on the source unit to allow data to be written to the source unit;
reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; writing the incremental data of the source unit into the target unit in real time after the query command is issued based on the identification information; the database corresponding to the source unit has a multi-version concurrency characteristic that data contained in the read data set is consistent with data contained in the source unit when the search command is issued;
acquiring identification information corresponding to all data currently contained in the source unit, including:
inquiring and recording the current system change number value of the database corresponding to the source unit; wherein the system change number value is an identification value corresponding to all data that the corresponding time in the source unit has.
2. The method of claim 1,
writing the incremental data of the source unit into the destination unit in real time after the query command is issued based on the identification information, wherein the incremental data comprises:
and writing the incremental data of the source unit correspondingly generated when the real-time system change number value of the database is larger than the recorded system change number value into the target unit in real time.
3. The method of claim 2, wherein writing the incremental data of the source unit into the destination unit in real time after issuing the lookup command comprises:
and grabbing an incremental data log of the source unit after the query command is issued by using the LogMinr of the database, and writing corresponding incremental data into the target unit in real time based on the incremental data log.
4. The method of claim 3, further comprising:
receiving a synchronization termination instruction input from the outside, and stopping executing the step of writing the incremental data of the source unit into the destination unit in real time under the trigger of the synchronization termination instruction.
5. The method of claim 4, wherein after writing the data set into the destination cell, further comprising:
and outputting a prompt message that the stock data migration is finished.
6. The method of claim 5, wherein outputting a prompt that inventory data migration has been completed comprises:
and sending the prompt information of the completed stock data migration to a terminal corresponding to the appointed user.
7. The method of claim 6, wherein the source unit and the destination unit are both database tables.
8. An apparatus for migrating data in a database, comprising:
a unit determination module to: determining a storage unit to which data migration needs to be realized as a source unit, and determining a storage unit to which data needs to be migrated in the source unit as a target unit;
a prepare migration module to: performing a locking operation on the source unit to block writing of data to the source unit; issuing a query command for querying all data to the source unit, and acquiring identification information corresponding to all data currently contained in the source unit; performing an unlock operation on the source unit to allow data to be written to the source unit;
a migration synchronization module to: reading a data set obtained by the source unit responding to the search command, and writing the data set into the destination unit; writing the incremental data of the source unit into the target unit in real time after the query command is issued based on the identification information; the database corresponding to the source unit has MVCC characteristics which enable data contained in the read data set to be consistent with data contained in the source unit when an inspection command is issued;
the preparation migration module comprises:
a query logging unit to: inquiring and recording the current system change number value of the database corresponding to the source unit; wherein the system change number value is an identification value corresponding to all data that the corresponding time in the source unit has.
9. A data migration apparatus of a database, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data migration method of the database according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of a data migration method of a database according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910271970.8A CN109933632B (en) | 2019-04-04 | 2019-04-04 | Data migration method, device and equipment for database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910271970.8A CN109933632B (en) | 2019-04-04 | 2019-04-04 | Data migration method, device and equipment for database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109933632A CN109933632A (en) | 2019-06-25 |
| CN109933632B true CN109933632B (en) | 2021-04-27 |
Family
ID=66989469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910271970.8A Active CN109933632B (en) | 2019-04-04 | 2019-04-04 | Data migration method, device and equipment for database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109933632B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110673941B (en) * | 2019-09-27 | 2020-07-17 | 掌阅科技股份有限公司 | Migration method of micro-services in multiple computer rooms, electronic equipment and storage medium |
| CN111651117B (en) * | 2020-04-24 | 2023-07-21 | 广东睿江云计算股份有限公司 | Method and device for migration of stored data |
| CN111984421B (en) * | 2020-09-03 | 2022-09-16 | 深圳壹账通智能科技有限公司 | Data processing method, device and storage medium |
| CN112612852A (en) * | 2020-12-29 | 2021-04-06 | 中国农业银行股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113568886A (en) * | 2021-07-21 | 2021-10-29 | 上海淇玥信息技术有限公司 | Data migration method and device and electronic equipment |
| CN116414907A (en) * | 2021-12-30 | 2023-07-11 | 中兴通讯股份有限公司 | Database table copying method and device, electronic equipment and storage medium |
| CN115454932A (en) * | 2022-09-16 | 2022-12-09 | 北京火山引擎科技有限公司 | Data processing method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107077495A (en) * | 2014-10-19 | 2017-08-18 | 微软技术许可有限责任公司 | High performance transaction in data base management system |
| CN107844566A (en) * | 2017-11-02 | 2018-03-27 | 杭州时趣信息技术有限公司 | A kind of dump control methods and its system |
| CN108132949A (en) * | 2016-12-01 | 2018-06-08 | 腾讯科技（深圳）有限公司 | The method and device of Data Migration in data-base cluster |
| CN108573019A (en) * | 2018-01-23 | 2018-09-25 | 北京金山云网络技术有限公司 | A kind of data migration method, device, electronic equipment and readable storage medium storing program for executing |
| CN108932282A (en) * | 2018-05-18 | 2018-12-04 | 腾讯科技（深圳）有限公司 | A kind of database migration method, apparatus and storage medium |
| CN109213817A (en) * | 2018-08-10 | 2019-01-15 | 杭州数梦工场科技有限公司 | Incremental data abstracting method, device and server |
| CN109284339A (en) * | 2018-11-30 | 2019-01-29 | 安徽继远软件有限公司 | A kind of method and apparatus of database data real-time synchronization |
| CN109426585A (en) * | 2017-08-22 | 2019-03-05 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for backing up, restoring database data |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10951706B2 (en) * | 2016-12-09 | 2021-03-16 | Google Llc | High-throughput algorithm for multiversion concurrency control with globally synchronized time |
| CN107357883A (en) * | 2017-06-30 | 2017-11-17 | 北京奇虎科技有限公司 | Data migration method and device |
| CN108399256B (en) * | 2018-03-06 | 2020-08-04 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method and device and middleware |
-
2019
- 2019-04-04 CN CN201910271970.8A patent/CN109933632B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107077495A (en) * | 2014-10-19 | 2017-08-18 | 微软技术许可有限责任公司 | High performance transaction in data base management system |
| CN108132949A (en) * | 2016-12-01 | 2018-06-08 | 腾讯科技（深圳）有限公司 | The method and device of Data Migration in data-base cluster |
| CN109426585A (en) * | 2017-08-22 | 2019-03-05 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for backing up, restoring database data |
| CN107844566A (en) * | 2017-11-02 | 2018-03-27 | 杭州时趣信息技术有限公司 | A kind of dump control methods and its system |
| CN108573019A (en) * | 2018-01-23 | 2018-09-25 | 北京金山云网络技术有限公司 | A kind of data migration method, device, electronic equipment and readable storage medium storing program for executing |
| CN108932282A (en) * | 2018-05-18 | 2018-12-04 | 腾讯科技（深圳）有限公司 | A kind of database migration method, apparatus and storage medium |
| CN109213817A (en) * | 2018-08-10 | 2019-01-15 | 杭州数梦工场科技有限公司 | Incremental data abstracting method, device and server |
| CN109284339A (en) * | 2018-11-30 | 2019-01-29 | 安徽继远软件有限公司 | A kind of method and apparatus of database data real-time synchronization |
Non-Patent Citations (2)
| Title |
|---|
| "Data management in cloud environments: NoSQL and NewSQL data stores";Katarina Grolinger et al.;《Journal of Cloud Computing: Advances, Systems and Applications》;20131218;第1-24页 * |
| "分布式存储系统主动容错机制研究";籍鑫璞;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170315;第I137-302页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109933632A (en) | 2019-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109933632B (en) | Data migration method, device and equipment for database | |
| CN109460349B (en) | Test case generation method and device based on log | |
| US11520753B1 (en) | Time reservations for ensuring consistent reads in a distributed database without logging | |
| US9984140B1 (en) | Lease based leader election system | |
| CN108932282B (en) | Database migration method and device and storage medium | |
| CN109241175A (en) | Method of data synchronization, device, storage medium and electronic equipment | |
| CN113396407A (en) | System and method for augmenting database applications using blockchain techniques | |
| CN105868343B (en) | Database migration method and system | |
| CN104809201B (en) | A kind of method and apparatus of database synchronization | |
| US8626765B2 (en) | Processing database operation requests | |
| CN110990432B (en) | Device and method for synchronizing distributed cache clusters across machine room | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| CN107515874A (en) | The method and apparatus of synchronous incremental data in a kind of distributed non-relational database | |
| CN103970833A (en) | Method for achieving two-way synchronous data circulation in heterogeneous database synchronizing system based on logs | |
| CN107391634B (en) | Data migration method and device | |
| CN109101423A (en) | A kind of automated testing method based on server-side log playback | |
| CN110019469A (en) | Distributed data base data processing method, device, storage medium and electronic device | |
| CN105279285A (en) | Synchronization system and method of relation-type database and non-relation-type database | |
| CN108319617A (en) | Determine the method, apparatus and method for handover control, device of database principal and subordinate's difference | |
| CN111930850A (en) | Data verification method and device, computer equipment and storage medium | |
| CN110019527A (en) | From library read method, relevant apparatus and equipment | |
| CN116932649A (en) | Database synchronization method, database synchronization device, and readable storage medium | |
| CN102339305A (en) | Database cluster failure recovery method on basis of partial ordering relation logs | |
| CN115587141A (en) | Database synchronization method and device | |
| KR102123616B1 (en) | Method and apparatus for parallel journaling using conflict page list |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |