CN112527899A - Data synchronization method, device, equipment and storage medium - Google Patents
Data synchronization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112527899A CN112527899A CN202011398414.6A CN202011398414A CN112527899A CN 112527899 A CN112527899 A CN 112527899A CN 202011398414 A CN202011398414 A CN 202011398414A CN 112527899 A CN112527899 A CN 112527899A
- Authority
- CN
- China
- Prior art keywords
- data
- message
- change information
- message queue
- source database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data synchronization method, a data synchronization device, data synchronization equipment and a data synchronization storage medium, and relates to the field of cloud computing such as cloud storage and cloud service. One embodiment of the data synchronization method includes: acquiring the change information of data in a source database; converting the change information into a message adapted to be stored in a message queue, and storing the message in the message queue; responding to the monitored message stored in the message queue, analyzing the change information of the data from the message, and forming an instruction for synchronizing the data changed in the source database to the target database according to the change information of the data; the method and the system send the instruction to the target database so that the target database synchronizes the changed data in the source database according to the instruction, and therefore when the data are synchronized, only the data change event is concerned, the synchronization is realized without change and synchronization without change, the operation of consuming resources such as reading the data in full quantity and writing the data in is avoided, and the resources of the server are saved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to the field of cloud computing such as cloud storage and cloud service, and in particular, to a method, an apparatus, a device, and a storage medium for data synchronization.
Background
In the current internet industry, the normal operation of a network system depends on the stable operation of a data center system, and once data is lost for some reason, certain functions are paralyzed, so that immeasurable loss is caused. In order to ensure the data security in the source database, a data backup method is usually adopted to ensure the continuity of the data, and the data of the source database on the server is remotely synchronized to the target database, so that the data in the target database can be used when the server fails.
When the data volume is too large, the problems of too much occupied resources (such as network broadband, disk IO, central processing unit and the like) and too long consumed time can occur during data synchronization, and further, the problem that the incremental data cannot be synchronized in real time during data synchronization, so that the synchronized data is inaccurate exists.
Disclosure of Invention
The embodiment of the application provides a method, a device, equipment and a storage medium for data synchronization.
In a first aspect, an embodiment of the present application provides a data synchronization method, including: acquiring the change information of data in a source database; converting the change information into a message adapted to be stored in a message queue, and storing the message in the message queue; responding to the monitored message stored in the message queue, analyzing the change information of the data from the message, and forming an instruction for synchronizing the data changed in the source database to the target database according to the change information of the data; and sending an instruction to the target database so that the target database synchronizes the changed data in the source database according to the instruction.
In a second aspect, an embodiment of the present application provides an apparatus for data synchronization, including: the acquisition module is configured to acquire the change information of the data in the source database; a conversion and storage module configured to convert the change information into a message adapted to be stored in a message queue, and store the message in the message queue; the analysis and formation module is configured to respond to the monitoring of the message queue storage message, analyze the change information of the data from the message, and form an instruction for synchronizing the data changed in the source database to the target database according to the change information of the data; and the synchronization module is configured to send an instruction to the target database so that the target database synchronizes the changed data in the source database according to the instruction.
In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of the implementations of the first aspect.
In a fourth aspect, embodiments of the present application propose a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described in any one of the implementations of the first aspect.
According to the method, the device, the equipment and the storage medium for data synchronization provided by the embodiment of the application, firstly, the change information of data in a source database is obtained; then converting the change information into a message which is suitable for being stored in a message queue, and storing the message in the message queue; then, responding to the monitored message queue to store the message, analyzing the change information of the data from the message, and forming an instruction for synchronizing the data changed in the source database to the target database according to the change information of the data; and finally, an instruction is sent to the target database so that the target database synchronizes the changed data in the source database according to the instruction, so that only data change events are concerned during data synchronization, synchronization is realized without change and synchronization without change, the operation of consuming resources such as reading data in full and writing data in is avoided, and the resources of the server are saved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings. The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a schematic flow chart diagram illustrating one embodiment of a method for data synchronization according to the present application;
FIG. 3 is a schematic flow chart diagram illustrating another embodiment of a method for data synchronization in accordance with the present application;
FIG. 4 is a schematic diagram of an application scenario of one embodiment of a method of data synchronization according to the present application;
FIG. 5 is a schematic block diagram of an embodiment of an apparatus for data synchronization of the present application;
fig. 6 is a block diagram of an electronic device for implementing a method of data synchronization according to an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
For ease of understanding, the terminology used in this application is now explained:
the source database, also called the upstream database, stores the data that is needed for synchronization.
The target database, also called downstream database, is used to hold the data that is to be synchronized.
The existing data synchronization technology implementation schemes mainly include the following two types:
the first scheme mainly comprises three steps: the method comprises the following steps: deleting data in the target database; step two: reading data in a source database, and storing the data into a data file; step three: and all data in the data file are input into the target database.
The second scheme mainly comprises three steps: the method comprises the following steps: reading data in a source database, and storing the data into a data file A; step two: reading data in a target database, and storing the data in a data file B; step three: and checking the data in the data file A and the data file B item by item, and performing corresponding operation according to different conditions.
For the first scheme, each time data synchronization needs to import the full amount of data, and if the data amount is too large, the data synchronization occupies too much network bandwidth and disk IO resources. Because the full amount of data needs to be written in each data synchronization, the synchronization time is too long due to single data synchronization, and the problem that incremental data is inaccurate during synchronization can occur.
For the second scheme, reading the full data occupies too much disk IO of the machine, and each piece of data is checked one by one, and occupies too much central processing unit of the machine. As the amount of data increases, the single synchronization increases linearly, and the incremental data is inaccurate during synchronization.
In view of the above-mentioned drawbacks of the prior art, embodiments of the present application provide a method for data synchronization.
Fig. 1 shows an exemplary system architecture 100 to which embodiments of the method of data synchronization or the apparatus of data synchronization of the present application may be applied.
As shown in fig. 1, the system architecture 100 may include a source database 101, a server 102, and a target database 103. The media comprising communication links between the source database 101 and the server 102 and between the target database 103 and the server 102 may comprise various types of connections, such as wired, wireless communication links, or fiber optic cables, among others.
The source database 101 may interact with the server 102 over a network. The source database 101 may provide change information for data in the source database.
The server 102 may provide various services, for example, the server 102 may analyze data, such as the change information of the data in the source database, acquired from the source database 101, and generate a processing result (for example, send an instruction to the target database 103 to enable the target database 103 to synchronize the changed data in the source database 101 according to the instruction).
The server 102 may be hardware or software. When the server 102 is hardware, it may be implemented as a distributed server cluster composed of multiple servers, or may be implemented as a single server. When the server 102 is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be noted that the method for data synchronization provided in the embodiment of the present application is generally performed by the server 102, and accordingly, the apparatus for data synchronization is generally disposed in the server 102.
It should be understood that the number of source databases, target databases, and servers in FIG. 1 are merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of a method of data synchronization in accordance with the present application is shown. The method comprises the following steps:
In this embodiment, an executing entity (e.g., the server 102 shown in fig. 1) of the data synchronization method may obtain the variation information of the data in the source database.
The data change information includes addition of data, update of data, deletion of data, and the like. When events such as adding data, updating data, deleting data and the like occur in the source database, the execution main body can monitor related events. The monitoring of the data change event in the source database can be realized according to the developed producer component. Producer components may be developed using a development language such as Python, JAVA, etc.
The producer component is configured to monitor change information of data in a source database and process and push the change information of the data to a message queue, and the producer component provides different implementations for different message queues, such as a Kafka component. The producer component interface (ipriducer) is used to define the external interfaces of the component: (a) send (topic, key, message, partition): and sending the message, namely sending the message to a specified theme in the message queue, and supporting the function of sending the theme partition. The sent message can designate a key value, and when the partition is not designated, automatic partition can be performed according to the key value; (b) flush () function: and refreshing the cache, and refreshing and sending the message currently cached in the sending queue.
In this embodiment, the execution body may convert the change information into a message adapted to be stored in a message queue, and store the message in the message queue.
Wherein the change information may be encapsulated as a message of fixed structure. Types of messages include, but are not limited to, insert type messages, delete type messages, and the like. Illustratively, the sample of plug-in messages is as follows: 'host': xxxx ',' values ':' project _ id ': xx', 'host': xx ',' v central processor s ': 1', 'memory _ mb': xx ',' root _ gb ',' geographic _ gb ',' 0',' vm _ state ': xx', 'task _ state': xx ',' os _ type ': xx', 'operation _ name': xx ',' error _ id ',' error _ p ': etc.' location ': 0', 'map': etc. 'map', 'map' etc. 'map' and 'map' etc. 'map' and map 'etc.' map 'and map' etc. 'map' are of '0', 'map' type ',' map 'and' map 'data' etc. 'map' of 'location'; examples of delete type messages are as follows: ' host ', ' xxxxxxxx ', ' user _ id ', ' project _ id ', ' xx ', ' host ', ' xx ', ' v central processor s ', '1', ' memory _ mb ', ' root _ gb ', ' geographic _ gb ', '0', ' vm _ state ', ' task _ state ', ' os _ type ', ' go _ name ', ' pn ', ' node ', etc. host ', ' go _ state ', ' os _ type ', ' host ', etc.
The message queue is also called a message-oriented middleware, and is a container for storing messages, when data in a source database changes (for example, data in the database is added, modified, deleted, and moved), the execution body can write corresponding messages into the message queue by using a producer, and when a consumer needs to use a message, the message can be taken out from the message queue for consumption and processing. Currently, the more used message queue types include RabbitMQ, Kafka, RockketMQ, Redis, ActiveMQ and the like.
In this embodiment, the execution body may perform an operation of parsing the change information of the data from the message in response to monitoring the message queue storing message, and forming an instruction to synchronize the changed data in the source database with the target database according to the change information of the data.
The execution subject can monitor the message queue according to the developed consumer component. When a new message appears in the message queue, the data change information described in the message body can be analyzed. The consumer component can be developed by using development languages such as Python and JAVA.
The consumer component is used for acquiring messages to be processed from the message queue and injecting the messages into the streaming data processing pipeline for data processing, such as the Kafka component. The consumer component interface (iconparent) is used to define the external interfaces of the component: (a) start _ stream (stream): starting a data stream, starting to subscribe the specified data stream from the message queue, starting to process new data by the data stream after the data stream is started, and supporting topic partition data stream subscription; (b) stop _ stream (stream): and stopping the data flow, canceling the subscription of the specified data flow from the message queue, and after the subscription is canceled, the data flow does not process new data any more. The consumer component will cooperate with the data flow component. And the consumer component acquires the subscription topic information from the data stream component and injects the message obtained through subscription into the data stream component for processing.
The execution main body utilizes the consumer component to form an instruction which can directly synchronize the data changed in the source database with the target database according to the analyzed data change information.
In this embodiment, by using the producer/consumer model, the problem of too long time delay for data synchronization between the source database and the target database can be reduced, and the real-time performance and accuracy of data synchronization are improved.
In this embodiment, the execution body may send an instruction to the target database, so that the target database synchronizes the data that changes in the source database according to the instruction.
The target database can be connected and the target database executes the instruction, so that the target database synchronizes the changed data in the source database.
According to the data synchronization method provided by the embodiment of the application, only data change events (namely, the data change events are not changed and are not synchronized, and the data change events are synchronized) are concerned, so that the operation that resources are consumed by reading data and writing data in a whole amount in the existing scheme is avoided, and the resources of a server, such as network bandwidth, disk IO, a central processing unit and the like, are saved.
With further reference to FIG. 3, shown is a flow diagram of another embodiment of a method of data synchronization, the method comprising the steps of:
Wherein, a system-level (Master) database in the source database writes the data change event into a binary log (binary log). The source database here may be a relational database management system, such as MySQL. The execution body may analyze the binary log in the source database to obtain data change information of the source database.
Step 302 is substantially the same as step 202, and therefore is not described in detail.
Step 303 is substantially the same as step 203, and therefore is not described in detail.
Step 304 is substantially the same as step 204 and thus will not be described again.
In some optional implementations of this embodiment, the message includes at least one of: insert messages, delete messages, update messages.
The inserted message represents that a plurality of data are newly added in the source database; the deletion message represents that a plurality of data are deleted from the source database; the update message characterizes a change in the source database.
In some optional implementations of this embodiment, the type of data includes at least one of: a cloud server instance, a virtual private cloud instance, a load balancing instance.
The cloud server instance refers to a virtual computing environment, and includes a central processing unit, a memory, an operating system, a broadband, a disk and other basic computing components. The virtual private cloud instance refers to a virtual computing environment based on a private cloud, and comprises a central processing unit, a memory, an operating system, a broadband, a disk and other basic computing components. The load balancing instance is a running load balancing service entity. Load balancing, called Load Balance in english, means that a Load (work task) is balanced and distributed to a plurality of operation units for operation, such as an FTP server, a Web server, an enterprise core application server, and other main task servers, so as to cooperatively complete the work task.
The data types in the source database and the target database can be any one of a cloud server instance, a virtual private cloud instance and a load balancing instance.
For ease of understanding, fig. 4 shows a schematic application scenario of an embodiment of a method of data synchronization according to the present application.
As shown in fig. 4, firstly, a producer monitors data change information of a cloud server instance, a virtual private cloud instance, and a load balancing instance as a source database; then the producer packages the data change information in the source database into a message, and stores the message into a message queue; then the consumer monitors the new message in the message queue, and analyzes the data change information entity in the new message, and forms an instruction which can be directly input to the target database; and finally, the consumer inputs an instruction to the target database, and the target database executes the instruction, so that the synchronous updating of the data is realized.
With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, the present application provides an embodiment of an apparatus for data synchronization, which corresponds to the embodiment of the method shown in fig. 2, and which can be applied to various electronic devices.
As shown in fig. 5, the apparatus 500 for data synchronization of the present embodiment may include: an acquisition module 501, a conversion and storage module 502, an analysis and formation module 503, and a synchronization module 504. The obtaining module 501 is configured to obtain change information of data in a source database; a conversion and storage module 502 configured to convert the change information into a message adapted to be stored in a message queue, and store the message in the message queue; a parsing and forming module 503 configured to parse change information of data from the message in response to monitoring that the message queue stores the message, and form an instruction to synchronize changed data in the source database with the target database according to the change information of the data; a synchronization module 504 configured to send an instruction to the target database to enable the target database to synchronize the changed data in the source database according to the instruction.
In the present embodiment, in the data synchronization apparatus 500: the specific processing and the technical effects thereof of the obtaining module 501, the converting and storing module 502, the analyzing and forming module 503, and the synchronizing module 504 can refer to the related descriptions of step 201 and step 204 in the corresponding embodiment of fig. 2, which are not described herein again.
In some optional implementations of this embodiment, the obtaining module 501 is further configured to: and analyzing the log file of the source database to obtain the change information of the data of the source database.
In some optional implementations of this embodiment, the message queue includes any one of: redis message queue, Rabbitmq message queue, Active message queue, Kafka message middleware.
In some optional implementations of this embodiment, the message includes at least one of: insert messages, delete messages, update messages.
In some optional implementations of this embodiment, the type of data includes at least one of: a cloud server instance, a virtual private cloud instance, a load balancing instance.
Fig. 6 is a block diagram of an electronic device according to an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 6, the electronic apparatus includes: one or more processors 601, memory 602, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 6, one processor 601 is taken as an example.
The memory 602 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method for data synchronization provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method of data synchronization provided herein.
The memory 602, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the method for data synchronization in the embodiments of the present application (e.g., the obtaining module 501, the transforming and storing module 502, the parsing and forming module 503, and the synchronizing module 504 shown in fig. 5). The processor 601 executes various functional applications of the server and data processing, i.e., a method for implementing data synchronization in the above-described method embodiments, by executing non-transitory software programs, instructions, and modules stored in the memory 602.
The memory 602 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device of the method of data synchronization, and the like. Further, the memory 602 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 602 optionally includes memory located remotely from the processor 601, and these remote memories may be connected over a network to the electronics of the method of data synchronization. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the method of data synchronization may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected by a bus or other means, and fig. 6 illustrates the connection by a bus as an example.
The input device 603 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device of the method of data synchronization, such as an input device of a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, etc. The output devices 604 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service extensibility in the conventional physical host and Virtual Private Server (VPS) service.
Cloud Computing (Cloud Computing) refers to a technology architecture that accesses a flexibly scalable pool of shared physical or virtual resources through a network, where the resources may include servers, operating systems, networks, software, applications, storage devices, and the like, and may be deployed and managed in an on-demand, self-service manner. Through the cloud computing technology, high-efficiency and strong data processing capacity can be provided for technical application and model training of artificial intelligence, block chains and the like.
According to the technical scheme of the application, firstly, the change information of data in a source database is obtained; then converting the change information into a message which is suitable for being stored in a message queue, and storing the message in the message queue; then, responding to the monitored message queue to store the message, analyzing the change information of the data from the message, and forming an instruction for synchronizing the data changed in the source database to the target database according to the change information of the data; and finally, an instruction is sent to the target database so that the target database synchronizes the changed data in the source database according to the instruction, so that only data change events are concerned during data synchronization, synchronization is realized without change and synchronization without change, the operation of consuming resources such as reading data in full and writing data in is avoided, and the resources of the server are saved.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (12)
1. A method of data synchronization, comprising:
acquiring the change information of data in a source database;
converting the change information into a message adapted to be stored in a message queue, and storing the message in the message queue;
responding to the monitored message stored in the message queue, analyzing the change information of the data from the message, and forming an instruction for synchronizing the data changed in the source database to the target database according to the change information of the data;
and sending the instruction to a target database so that the target database synchronizes the changed data in the source database according to the instruction.
2. The method of claim 1, wherein the obtaining the variation information of the data in the source database comprises:
and analyzing the log file of the source database to obtain the change information of the data of the source database.
3. The method of claim 1, wherein the message queue comprises any one of:
redis message queue, Rabbitmq message queue, Active message queue, Kafka message middleware.
4. The method of claim 1, wherein the message comprises at least one of:
insert messages, delete messages, update messages.
5. The method of claim 1, wherein the type of data comprises at least one of:
a cloud server instance, a virtual private cloud instance, a load balancing instance.
6. An apparatus for data synchronization, wherein the apparatus comprises:
the acquisition module is configured to acquire the change information of the data in the source database;
a conversion and storage module configured to convert the change information into a message adapted to be stored in a message queue, and store the message in the message queue;
the analysis and formation module is configured to respond to the monitoring that the message queue stores the message, analyze the change information of the data from the message, and form an instruction for synchronizing the changed data in the source database to the target database according to the change information of the data;
and the synchronization module is configured to send the instruction to the target database so that the target database synchronizes the changed data in the source database according to the instruction.
7. The apparatus of claim 6, wherein the acquisition module is further configured to:
and analyzing the log file of the source database to obtain the change information of the data of the source database.
8. The apparatus of claim 6, wherein the message queue comprises any one of:
redis message queue, Rabbitmq message queue, Active message queue, Kafka message middleware.
9. The apparatus of claim 6, wherein the message comprises at least one of:
insert messages, delete messages, update messages.
10. The apparatus of claim 6, wherein the type of data comprises at least one of:
a cloud server instance, a virtual private cloud instance, a load balancing instance.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011398414.6A CN112527899A (en) | 2020-12-03 | 2020-12-03 | Data synchronization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011398414.6A CN112527899A (en) | 2020-12-03 | 2020-12-03 | Data synchronization method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527899A true CN112527899A (en) | 2021-03-19 |
Family
ID=74997366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011398414.6A Pending CN112527899A (en) | 2020-12-03 | 2020-12-03 | Data synchronization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527899A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113032491A (en) * | 2021-04-07 | 2021-06-25 | 工银科技有限公司 | Method, device, electronic equipment and medium for realizing static data synchronization |
| CN113095874A (en) * | 2021-03-31 | 2021-07-09 | 北京沃东天骏信息技术有限公司 | Material increment processing method, device, equipment and storage medium |
| CN113312192A (en) * | 2021-06-07 | 2021-08-27 | 平安证券股份有限公司 | Data synchronization method and device based on window, electronic equipment and storage medium |
| CN113448647A (en) * | 2021-06-28 | 2021-09-28 | 华云数据控股集团有限公司 | Resource synchronization method, implementation equipment and electronic equipment |
| CN113704354A (en) * | 2021-08-30 | 2021-11-26 | 康键信息技术（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN113722401A (en) * | 2021-11-04 | 2021-11-30 | 树根互联股份有限公司 | Data caching method and device, computer equipment and readable storage medium |
| CN114442959A (en) * | 2022-01-29 | 2022-05-06 | 北京百度网讯科技有限公司 | Data writing method, device and system of multi-region storage system |
| CN115640100A (en) * | 2022-11-08 | 2023-01-24 | 安超云软件有限公司 | Virtual machine information synchronization method and computer readable medium |
| CN116795932A (en) * | 2023-08-29 | 2023-09-22 | 天津旗领机电科技有限公司 | Data synchronization method and system based on speed reducer |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694857A (en) * | 2020-06-12 | 2020-09-22 | 北京百度网讯科技有限公司 | Method and device for storing resource data, electronic equipment and computer readable medium |
| CN111767296A (en) * | 2020-06-30 | 2020-10-13 | 北京百度网讯科技有限公司 | Method, device, electronic equipment and readable storage medium for synchronizing data |
-
2020
- 2020-12-03 CN CN202011398414.6A patent/CN112527899A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694857A (en) * | 2020-06-12 | 2020-09-22 | 北京百度网讯科技有限公司 | Method and device for storing resource data, electronic equipment and computer readable medium |
| CN111767296A (en) * | 2020-06-30 | 2020-10-13 | 北京百度网讯科技有限公司 | Method, device, electronic equipment and readable storage medium for synchronizing data |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095874A (en) * | 2021-03-31 | 2021-07-09 | 北京沃东天骏信息技术有限公司 | Material increment processing method, device, equipment and storage medium |
| CN113095874B (en) * | 2021-03-31 | 2024-06-21 | 北京沃东天骏信息技术有限公司 | Material increment processing method, device, equipment and storage medium |
| CN113032491A (en) * | 2021-04-07 | 2021-06-25 | 工银科技有限公司 | Method, device, electronic equipment and medium for realizing static data synchronization |
| CN113032491B (en) * | 2021-04-07 | 2023-01-06 | 工银科技有限公司 | Method, device, electronic equipment and medium for realizing static data synchronization |
| CN113312192A (en) * | 2021-06-07 | 2021-08-27 | 平安证券股份有限公司 | Data synchronization method and device based on window, electronic equipment and storage medium |
| CN113448647A (en) * | 2021-06-28 | 2021-09-28 | 华云数据控股集团有限公司 | Resource synchronization method, implementation equipment and electronic equipment |
| CN113448647B (en) * | 2021-06-28 | 2024-02-09 | 华云数据控股集团有限公司 | Resource synchronization method, implementation equipment and electronic equipment |
| CN113704354B (en) * | 2021-08-30 | 2023-10-20 | 康键信息技术（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN113704354A (en) * | 2021-08-30 | 2021-11-26 | 康键信息技术（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN113722401A (en) * | 2021-11-04 | 2021-11-30 | 树根互联股份有限公司 | Data caching method and device, computer equipment and readable storage medium |
| CN114442959A (en) * | 2022-01-29 | 2022-05-06 | 北京百度网讯科技有限公司 | Data writing method, device and system of multi-region storage system |
| CN115640100B (en) * | 2022-11-08 | 2023-05-23 | 安超云软件有限公司 | Virtual machine information synchronization method and computer readable medium |
| CN115640100A (en) * | 2022-11-08 | 2023-01-24 | 安超云软件有限公司 | Virtual machine information synchronization method and computer readable medium |
| CN116795932A (en) * | 2023-08-29 | 2023-09-22 | 天津旗领机电科技有限公司 | Data synchronization method and system based on speed reducer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112527899A (en) | Data synchronization method, device, equipment and storage medium | |
| CN108519914B (en) | Big data calculation method and system and computer equipment | |
| CN111753997B (en) | Distributed training method, system, device and storage medium | |
| US20210248060A1 (en) | Method and apparatus for testing map service | |
| CN109522029A (en) | A kind of method and device for disposing cloud platform technology component | |
| CN112069201A (en) | Target data acquisition method and device | |
| CN110795503A (en) | Multi-cluster data synchronization method and related device of distributed storage system | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN111782365B (en) | Timed task processing method, device, equipment and storage medium | |
| CN111694857B (en) | Method, device, electronic equipment and computer readable medium for storing resource data | |
| CN113364877B (en) | Data processing method, device, electronic equipment and medium | |
| CN111966677A (en) | Data report processing method and device, electronic equipment and storage medium | |
| CN112214547A (en) | Data processing method, data server, electronic device and storage medium | |
| CN111740859B (en) | Test environment deployment method and device, electronic equipment and storage medium | |
| CN111782147A (en) | Method and apparatus for cluster scale-up | |
| CN111259066A (en) | Server cluster data synchronization method and device | |
| CN111782341A (en) | Method and apparatus for managing clusters | |
| CN111782357A (en) | Label control method and device, electronic equipment and readable storage medium | |
| CN112306413B (en) | Method, device, equipment and storage medium for accessing memory | |
| CN114070889A (en) | Configuration method, traffic forwarding method, device, storage medium, and program product | |
| CN113297327A (en) | System and method for generating distributed ID | |
| CN113821232A (en) | Model updating method and device | |
| CN111292223A (en) | Graph calculation processing method and device, electronic equipment and storage medium | |
| CN106411547B (en) | A kind of redis calls monitoring method and device | |
| CN111930748B (en) | Method, device, equipment and storage medium for tracking data of streaming computing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |