CN113792024A

CN113792024A - Method, device, equipment and storage medium for migrating data

Info

Publication number: CN113792024A
Application number: CN202110227512.1A
Authority: CN
Inventors: 刘泽广; 闫创
Original assignee: Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2021-03-02
Filing date: 2021-03-02
Publication date: 2021-12-14

Abstract

The application discloses a method, a device, equipment and a storage medium for migrating data, and relates to the technical field of distribution. The specific implementation scheme is as follows: acquiring an identifier of target data; marking the mark; copying the target data from the source database to the target database; in response to determining that the target data copy is complete, the mark is removed. The implementation mode provides a data migration scheme, and data in a source database can be migrated to a target database.

Description

Method, device, equipment and storage medium for migrating data

Technical Field

The present application relates to the field of computer technologies, and in particular, to the field of distributed technologies, and in particular, to a method, an apparatus, a device, and a storage medium for migrating data.

Background

In the electronic commerce service, advertisement service data is dispersedly stored in a plurality of data tables according to identification, and in order to prevent the data from increasing without limit, each advertiser has respective upper limit limitation in different table dimensions. However, as new advertisers and active advertisers increase, data in the material library is accelerated more and more quickly, and the storage in the MySQL single library mode cannot meet the requirement, and the read-write pressure needs to be dispersed in a sub-library and sub-table mode.

After new elastic database resources are obtained, the data volume of the original physical database is large, and the data requirements are high in availability and strong in consistency. The risk of migrating to a new library at once is large, so migration by identification dimension is required, and the migration process is a continuous work with a large time span. In addition, a new table is created in the process of continuous iteration of advertisement putting requirements, so that in the data migration stage, a new table is maintained by a new library and an old library at the same time, and therefore the migration of the new table can be flexibly accessed by the library migration scheme, and repeated migration development work does not need to be carried out on the new table.

Disclosure of Invention

A method, apparatus, device, and storage medium for migrating data are provided.

According to a first aspect, there is provided a method for migrating data, comprising: acquiring an identifier of target data; marking the mark; copying the target data from the source database to the target database; in response to determining that the target data copy is complete, the mark is removed.

According to a second aspect, there is provided an apparatus for migrating data, comprising: an acquisition unit configured to acquire an identification of target data; a marking unit configured to mark the identifier; a copying unit configured to copy the target data from the source database to the target database; a removal unit configured to remove the mark in response to determining that the target data replication is complete.

According to a third aspect, there is provided an electronic device for migrating data, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

According to a fourth aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method as described in the first aspect.

According to a fifth aspect, a computer program product comprising a computer program which, when executed by a processor, implements the method as described in the first aspect.

According to the data migration scheme provided by the technology of the application, the data in the source database can be migrated to the target database.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:

FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present application may be applied;

FIG. 2 is a flow diagram for one embodiment of a method for migrating data, according to the present application;

FIG. 3 is a schematic illustration of an application scenario of a method for migrating data according to the present application;

FIG. 4 is a flow diagram of another embodiment of a method for migrating data according to the present application;

FIG. 5 is a schematic block diagram illustrating one embodiment of an apparatus for migrating data according to the present application;

FIG. 6 is a block diagram of an electronic device used to implement a method for migrating data according to an embodiment of the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the present method for migrating data or apparatus for migrating data may be applied.

As shown in fig. 1, the system architecture 100 may include a source database 101, a server 102, and a target database 103. The source database 101 and the server 102 are communicatively coupled via a network, as well as the server 102 and the target database 103. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The source database 101 and the target database 103 may be used for storing data. A large amount of data may be stored in the source database 101, and the server 102 may migrate the data in the source database 101 to the target database 103 after receiving a migration instruction from a user. The source database 101 and the target database 103 may be distributed databases, each of which may include a plurality of database shards. The data may be stored in the data table described above.

The server 102 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the server 105 is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

It should be noted that the method for migrating data provided by the embodiment of the present application is generally executed by the server 102. Accordingly, the means for migrating data is generally disposed in the server 102.

It should be understood that the number of databases and servers in fig. 1 is merely illustrative. There may be any number of databases and servers, as desired for implementation.

With continued reference to FIG. 2, a flow 200 of one embodiment of a method for migrating data in accordance with the present application is shown. The method for migrating data of the embodiment comprises the following steps:

step 201, acquiring the identifier of the target data.

In this embodiment, an execution subject of the method for migrating data (e.g., the server 102 shown in fig. 1) may obtain the identification of the target data in various ways. For example, the executing agent may receive an identification of target data sent by the user through the terminal device. The identification of the target data may be an identification of the advertiser, such as the advertiser's name, or the like. In some specific applications, the execution agent may add the identification of the target data to the target queue each time after receiving a migration instruction sent by the user. The execution agent may obtain an identification of the target data from the target queue. Thus, if the execution subject is a distributed device, the identification of the target data can be obtained from the target queue respectively.

Step 202, marking the identification.

After acquiring the identifier of the target data, the execution subject may mark the identifier. The flag acts as a "black list" to suspend writing of data indicated by the flag. And data migration errors caused by data writing in the data migration process are avoided. When the data reading and writing device reads and writes data, it can firstly inquire whether the data mark has a mark, if so, it indicates that the data can not be written at this time.

And step 203, copying the target data from the source database to the target database.

After marking the target data, the execution body may copy the target data from the source database to the target database. Specifically, the execution agent may execute the copy task by a plurality of threads. Each thread may be responsible for the replication of data in a single data table.

Step 204, in response to determining that the target data replication is complete, removing the mark.

The execution body may remove the mark after copying the target data. Therefore, the data in the source database and the data migrated to the target database can be both read and written by users. It can be understood that, in order to ensure consistency between data in the source database and data in the target database, a read-write log may be generated in the process of reading and writing data in the source database and the target database.

With continued reference to FIG. 3, a schematic diagram of one application scenario of a method for migrating data according to the present application is shown. In the application scenario of fig. 3, after receiving a migration instruction sent by a user, a server 301 obtains an identifier abc of target data from a source database 302. The identification abc is then marked as unwritable. The target data is then copied to the target database 303303. Finally, after the target data is copied, the 'unwritable' mark is removed.

The method for migrating data provided by the above embodiment of the present application can mark data in the process of migrating data, so that a situation of data migration error caused by writing data in the process of migrating data can be avoided.

With continued reference to FIG. 4, a flow 400 of another embodiment of a method for migrating data in accordance with the present application is shown. As shown in fig. 4, the method of the present embodiment may include the following steps:

step 401, acquiring an identifier of target data.

Step 402, inquiring the identifier in a preset data state list, and checking whether the state corresponding to the identifier is to be migrated; and marking the identifier in response to determining that the state corresponding to the identifier is to be migrated.

In this embodiment, after acquiring the identifier of the target data, the execution subject may first query a preset data state list, where the list includes a correspondence between the identifier and the state. And if the state corresponding to the identifier is inquired to be transferred, marking the identifier. If the state corresponding to the identifier is migrated, it indicates that the data has been migrated, and the identifier of the new data may be acquired.

In step 403, the target data is copied from the source database to the target database.

Step 404, in response to determining that the target data replication is complete, removing the flag and updating the data status list.

If the target data replication is complete, the execution agent may remove the flag and update the update data status list. Specifically, the executing entity may update the state corresponding to the identifier in the data state list.

Step 405, generating a migration log in the copying process; after the copying is finished, comparing the data copied to the target database with the target data; and in response to determining that the two are different, performing rollback operation according to the migration log and deleting the data copied to the target database.

In this embodiment, the execution subject may also generate a migration log in the replication process. The migration log includes information such as the source of the data, the destination path of the data, and the time of copying. After the copying is complete, the executing entity may compare the data copied into the target database with the target data. If the two are not the same, it indicates that an error occurs in the migration process. The execution principal may perform a rollback operation from the migration log. And delete the data copied to the target database.

In some optional implementations of this embodiment, the migration log includes a migration batch. The migration batch refers to data which are migrated in the same batch and are initiated by a graph operator. The aromatic hydrocarbon may also include the following steps not shown in fig. 4: in response to determining that the data in the target database is in error, other data of the same migration batch as the erroneous data is checked.

In this implementation, if the execution subject determines that the data in the target database is erroneous, other data in the same migration batch as the erroneous data may be checked. The error may be various errors, such as a read/write error, and the like.

Step 406, in response to determining that the target data migration completion time and the current time are the same, determining the duration between the target data migration completion time and the current time; and deleting the target data in the source database in response to the fact that the duration is larger than the preset duration.

If the two are the same, the data migration is successful, and the execution subject may determine the duration between the target data migration completion time and the current time. And if the duration is longer than the preset duration, deleting the target data in the source database. In this way, memory space can be saved.

Step 407, in response to determining that the incremental data of the target data is detected after the copying is completed, copying the incremental data to the target database; in response to determining that the incremental data copy is complete, the incremental data is assembled with the target data.

In this embodiment, if the execution subject detects incremental data of the target data after the replication is completed, it indicates that the user writes data in the source database after the target data is migrated to the target database. The incremental data needs to be copied to the target database. If the incremental data copying is completed, the incremental data and the target data can be assembled, and the consistency of the data is ensured.

The method for migrating data provided by the embodiment of the application can verify the data after the migration is completed, so that the accuracy of the data is ensured. The incremental data can also be migrated to a target database, and the incremental data and the target data are assembled after the migration is completed.

With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, the present application provides an embodiment of an apparatus for migrating data, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be applied to various electronic devices.

As shown in fig. 5, the apparatus 500 for migrating data of the present embodiment includes: an acquisition unit 501, a marking unit 502, a copying unit 503, and a removing unit 504.

An obtaining unit 501 configured to obtain an identification of target data.

A marking unit 502 configured to mark the identity.

A copying unit 503 configured to copy the target data from the source database to the target database.

A removal unit 504 configured to remove the mark in response to determining that the target data replication is complete.

In some optional implementations of this embodiment, the marking unit 502 may be further configured to: inquiring an identifier in a preset data state list, and checking whether the state corresponding to the identifier is to be migrated; marking the identifier in response to determining that the state corresponding to the identifier is to be migrated

In some optional implementations of this embodiment, the apparatus 500 may further include an updating unit, not shown in fig. 5, configured to: in response to determining that the target data copy is complete, the data status list is updated.

In some optional implementations of this embodiment, the apparatus 500 may further include, not shown in fig. 5: the device comprises a generating unit, a comparing unit and a rollback unit.

A generation unit configured to generate a migration log during the copying.

And the comparison unit is configured to compare the data copied to the target database with the target data after the copying is completed.

And the rollback unit is configured to perform rollback operation according to the migration log and delete the data copied to the target database in response to determining that the two are different.

In some optional implementations of this embodiment, the migration log includes a migration batch. The apparatus 500 may further comprise a verification unit, not shown in fig. 5, configured to: in response to determining that the data in the target database is in error, other data of the same migration batch as the erroneous data is checked.

In some optional implementations of this embodiment, the apparatus 500 may further include a deletion unit, not shown in fig. 5, configured to: in response to determining that the target data migration completion time and the current time are the same, determining the duration between the target data migration completion time and the current time; and deleting the target data in the source database in response to the fact that the duration is larger than the preset duration.

In some optional implementations of this embodiment, the apparatus 500 may further include an assembling unit, not shown in fig. 5, configured to: in response to determining that incremental data of the target data is detected after the copying is complete, copying the incremental data to the target database; in response to determining that the incremental data copy is complete, the incremental data is assembled with the target data.

It should be understood that units 501 to 504 recited in the apparatus 500 for migrating data correspond to respective steps in the method described with reference to fig. 2. Thus, the operations and features described above with respect to the method for migrating data are equally applicable to the apparatus 500 and the units contained therein and will not be described again here.

The application also provides an electronic device, a readable storage medium and a computer program product according to the embodiment of the application.

FIG. 6 shows a block diagram of an electronic device 600 that performs a method for migrating data according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.

As shown in fig. 6, the device 600 includes a processor 601 that may perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a memory 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I/O interface (input/output interface) 605 is also connected to the bus 604.

A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a memory 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

Processor 601 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of processor 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 601 performs the various methods and processes described above, such as the method for migrating data. For example, in some embodiments, the method for migrating data may be implemented as a computer software program tangibly embodied in a machine-readable storage medium, such as memory 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into RAM 603 and executed by the processor 601, one or more steps of the method for migrating data described above may be performed. Alternatively, in other embodiments, the processor 601 may be configured by any other suitable means (e.g., by way of firmware) to perform the method for migrating data.

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. The program code described above may be packaged as a computer program product. These program code or computer program products may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor 601, causes the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this application, a machine-readable storage medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable storage medium may be a machine-readable signal storage medium or a machine-readable storage medium. A machine-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server can be a cloud Server, also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server may also be a server of a distributed system, or a server incorporating a blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of the present application can be achieved, and the present invention is not limited thereto.

The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for migrating data, comprising:

acquiring an identifier of target data;

marking the identification;

copying the target data from the source database to the target database;

removing the mark in response to determining that the target data replication is complete.

2. The method of claim 1, wherein said tagging the identity comprises:

inquiring the identifier in a preset data state list, and checking whether the state corresponding to the identifier is to be migrated;

and marking the identifier in response to determining that the state corresponding to the identifier is to be migrated.

3. The method of claim 2, wherein the method further comprises:

in response to determining that the target data replication is complete, updating the data status list.

4. The method of claim 1, wherein the method further comprises:

generating a migration log in the copying process;

after the copying is finished, comparing the data copied into the target database with the target data;

and in response to determining that the two are different, performing rollback operation and deleting the data copied to the target database according to the migration log.

5. The method of claim 4, wherein the migration log comprises a migration batch; and

the method further comprises the following steps:

in response to determining that the data in the target database is in error, other data of the same migration batch as the erroneous data is checked.

6. The method of claim 4, wherein the method further comprises:

in response to determining that the target data migration completion time and the current time are the same, determining the duration between the target data migration completion time and the current time;

and deleting the target data in the source database in response to the fact that the duration is larger than the preset duration.

7. The method of claim 1, wherein the method further comprises:

in response to determining that incremental data of the target data is detected after copying is complete, copying the incremental data to the target database;

assembling the incremental data with the target data in response to determining that the incremental data copy is complete.

8. An apparatus for migrating data, comprising:

an acquisition unit configured to acquire an identification of target data;

a marking unit configured to mark the identification;

a replication unit configured to replicate the target data from a source database to a target database;

a removal unit configured to remove the mark in response to determining that the target data replication is complete.

9. The apparatus of claim 8, wherein the tagging unit is further configured to:

10. The apparatus of claim 9, wherein the apparatus further comprises an update unit configured to:

11. The apparatus of claim 8, wherein the apparatus further comprises:

a generation unit configured to generate a migration log in a replication process;

a comparison unit configured to compare the data copied into the target database with the target data after completion of the copying;

and the rollback unit is configured to perform rollback operation and delete the data copied to the target database according to the migration log in response to determining that the two are different.

12. The apparatus of claim 11, wherein the migration log comprises a migration batch; and

the apparatus further comprises a verification unit configured to:

13. The apparatus of claim 11, wherein the apparatus further comprises a deletion unit configured to:

14. The apparatus of claim 8, wherein the apparatus further comprises an assembly unit configured to:

15. An electronic device that performs a method for migrating data, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.