CN111309747A - Data synchronization method, system and device - Google Patents
Data synchronization method, system and device Download PDFInfo
- Publication number
- CN111309747A CN111309747A CN202010099242.6A CN202010099242A CN111309747A CN 111309747 A CN111309747 A CN 111309747A CN 202010099242 A CN202010099242 A CN 202010099242A CN 111309747 A CN111309747 A CN 111309747A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- server
- request
- timestamp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure discloses a data synchronization method, a system and a device. One embodiment of the method comprises: the method comprises the steps of responding to a received data synchronization request sent by a client, analyzing the synchronization request to obtain data to be synchronized indicated by the synchronization request, sending a timestamp request to a global clock server, responding to the received global timestamp returned by the global clock server, sending a first storage request to a first server, responding to a received storage success message corresponding to the first storage request returned by the first server, sending a data request to the first server, receiving target synchronization data corresponding to the data request returned by the first server, sending a second storage request to a second server, responding to the received data synchronization result returned by the second server, and forwarding the data synchronization result to the client. The implementation mode improves the accuracy of data synchronization and improves the efficiency of data synchronization.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to the technical field of data processing, and particularly relates to a data synchronization method, system and device.
Background
The distributed storage system Hbase and a data search and analysis engine (ES for short) are a commonly used mass data storage and real-time search scheme, and because data needs to be stored in the distributed storage system Hbase and the data search and analysis engine ES respectively, how to synchronize the data to the distributed storage system Hbase and the data search and analysis engine ES becomes a problem to be solved.
The current scheme for synchronizing data to the distributed storage system Hbase and the data search and analysis engine ES includes: one is that data is firstly stored in a distributed storage system Hbase, and then the data is written into a data search analysis engine ES from the distributed storage system Hbase according to the sequence of the data stored in the Hbase; the other is to store the data in the message queue to the distributed storage system Hbase and the data search analysis engine ES in sequence by using the message queue, and the other is to realize the data synchronization of the distributed storage system ES and the distributed storage system Hbase by using a coprocessor of the distributed storage system Hbase.
Disclosure of Invention
The embodiment of the disclosure provides a data synchronization method, a system and a device.
In a first aspect, an embodiment of the present disclosure provides a data synchronization method, where the method includes: in response to receiving a data synchronization request sent by a client, analyzing the synchronization request to acquire data to be synchronized indicated by the synchronization request; sending a timestamp request to a global clock server; sending a first storage request to a first server in response to receiving a global timestamp returned by a global clock server, wherein the first storage request comprises the global timestamp and data to be synchronized, and the global timestamp is a group of numbers indicating the current time and a sequence number; sending a data request to the first server in response to receiving a storage success message corresponding to the first storage request returned by the first server; receiving target synchronous data corresponding to a data request returned by a first server, and sending a second storage request to a second server, wherein the second storage request comprises a global timestamp and the target synchronous data; and responding to the received data synchronization result returned by the second server and forwarding the data synchronization result to the client.
In some embodiments, after receiving target synchronization data corresponding to a data request returned by the first server and sending a second storage request to the second server, the method further includes: in response to receiving a storage failure message corresponding to the second storage request returned by the second server, storing the target synchronous data and the global timestamp in a data recovery table; based on the data recovery table, a data recovery operation is performed.
In some embodiments, the target synchronization data is full data corresponding to a data identifier in the first server database, where the data identifier is a unique identifier of a row of records corresponding to data to be synchronized in the first server database.
In a second aspect, an embodiment of the present disclosure provides a data synchronization method, including: receiving a storage request sent by a synchronous processing server, and analyzing the storage request to acquire target synchronous data and a global timestamp, wherein the global timestamp is a group of monotonically increasing numbers indicating the current time and the sequence number; and updating the data corresponding to the target synchronous data and the version number of the data corresponding to the data ID based on the global timestamp, and returning a data synchronization result to the synchronous processing server, wherein the data ID is the unique identifier of the data corresponding to the target synchronous data in the local database.
In some embodiments, based on the global timestamp, updating data corresponding to the target synchronization data and updating a version number of the data corresponding to the target synchronization data, and returning a data synchronization result to the synchronization processing server includes: judging whether the global timestamp is larger than the version number of the data corresponding to the data ID or not; if the global timestamp is greater than the version number of the data corresponding to the data ID, updating the data corresponding to the target synchronous data and updating the version number of the data corresponding to the data ID; and responding to the success of updating the data corresponding to the target synchronous data, and returning a data synchronization result to the synchronous processing server.
In some embodiments, after determining whether the global timestamp is greater than the version number of the data corresponding to the data ID, the method further comprises: and if the global timestamp is not greater than the version number of the data corresponding to the data ID, returning a data synchronization result to the synchronization processing server.
In some embodiments, after updating the data corresponding to the target synchronization data and updating the version number of the data corresponding to the data ID if the global timestamp is greater than the version number of the data corresponding to the data ID, the method further includes: and responding to the data failure corresponding to the updated target synchronous data, and sending a storage failure message to the synchronous processing server.
In a third aspect, an embodiment of the present disclosure provides a data synchronization system, including: a synchronization processing server, wherein the synchronization processing server is configured to execute the data synchronization method of the first aspect.
In some embodiments, the system further comprises a second server; the second server is configured to execute the data synchronization method of the second aspect.
In some embodiments, the system further comprises a first server and a global clock server; the first server is used for receiving a first storage request sent by the synchronous processing server, analyzing the first storage request to obtain data to be synchronized and a global timestamp, updating the timestamp of the data corresponding to the data to be synchronized and the data corresponding to the updated data identifier based on the global timestamp, wherein the data identifier is a unique identifier of a row of records corresponding to the data to be synchronized in the library, responding to the success of updating the data corresponding to the data to be synchronized, sending a storage success message corresponding to the first storage request to the synchronous processing server, receiving the data request sent by the synchronous processing server, obtaining target synchronous data corresponding to the data request in the library, and sending the target synchronous data to the synchronous processing server. And the global clock server is used for receiving the timestamp request sent by the synchronous processing server, acquiring a global timestamp corresponding to the current timestamp, and sending the global timestamp to the synchronous processing server.
In some embodiments, the first service end is further configured to determine whether the global timestamp is greater than a timestamp of data corresponding to the data identifier in the library, and update the data corresponding to the data to be synchronized and the timestamp of data corresponding to the updated data identifier if the global timestamp is greater than the timestamp of data corresponding to the data identifier in the library.
In some embodiments, the first service end is further configured to determine whether a data identifier exists in the library, and if the data identifier does not exist in the library, add data corresponding to the data to be synchronized and add a timestamp of the data corresponding to the data identifier.
In a fourth aspect, an embodiment of the present disclosure provides a data synchronization apparatus, including: the data synchronization device comprises an analysis unit and a synchronization unit, wherein the analysis unit is configured to respond to a received data synchronization request sent by a client and analyze the synchronization request to acquire data to be synchronized indicated by the synchronization request; a timestamp request unit configured to send a timestamp request to the global clock server; the first storage request unit is configured to respond to a received global timestamp returned by the global clock server and send a first storage request to the first server, wherein the first storage request comprises the global timestamp and data to be synchronized, and the global timestamp is a group of numbers indicating the current time and a sequence number; the data request unit is configured to respond to the received storage success message which is returned by the first server and corresponds to the first storage request, and send a data request to the first server; the second storage request unit is configured to receive target synchronous data corresponding to the data request returned by the first server and send a second storage request to the second server, wherein the second storage request comprises a global timestamp and the target synchronous data; and the forwarding unit is configured to respond to the received data synchronization result returned by the second server and forward the data synchronization result to the client.
In some embodiments, the apparatus further comprises: the storage unit is configured to respond to the received storage failure message which is returned by the second server and corresponds to the second storage request, and store the target synchronous data and the global timestamp in the data recovery table; a recovery unit configured to perform a data recovery operation based on the data recovery table.
In some embodiments, the target synchronization data is full data corresponding to a data identifier in the first server database, where the data identifier is a unique identifier of a row of records corresponding to data to be synchronized in the first server database.
In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method as described in any implementation manner of the first aspect.
The data synchronization method, the system and the device provided by the embodiment of the disclosure adopt the steps of responding to a received data synchronization request sent by a client, analyzing the synchronization request to obtain data to be synchronized indicated by the synchronization request, sending a timestamp request to a global clock server, responding to the received global timestamp returned by the global clock server, sending a first storage request to a first server, responding to a received storage success message corresponding to the first storage request returned by the first server, sending a data request to the first server, receiving target synchronization data corresponding to the data request returned by the first server, sending a second storage request to a second server, responding to a received data synchronization result returned by the second server, and forwarding the data synchronization result to the client, so that the problem that data of the first server (distributed storage system Hbase) and data search analysis engine ES are not generated when the second server (data search analysis engine ES) is operated in a multi-thread mode is overcome The data synchronization method comprises the steps of sending a data request to a first server after receiving a storage success message returned by the first server, receiving target synchronization data corresponding to the data request returned by the first server, and sending a second storage request to a second server, so that a plurality of storage requests can be sent to a data search analysis engine ES at the same time, the problem that the whole data synchronization is blocked due to the fact that a single data fails to be synchronized in the prior art is solved, the data synchronization efficiency is improved, the influence on the performance and stability of the Hbase caused by the fact that the data synchronization is carried out by the aid of the co-processing of the distributed storage system Hbase in the prior art is avoided by using a synchronization processing server, and the performance of the Hbase is improved.
Drawings
Other features, objects and advantages of the disclosure will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which some embodiments of the present disclosure may be applied;
FIG. 2 is a flow diagram for one embodiment of a data synchronization method according to the present disclosure;
FIG. 3 is a schematic diagram of one application scenario of a data synchronization method according to the present disclosure;
FIG. 4 is a flow diagram of one embodiment of a data synchronization method applied to a second server in accordance with the present disclosure;
FIG. 5 is a timing diagram for one embodiment of a data synchronization system according to the present disclosure;
FIG. 6 is a schematic block diagram illustrating one embodiment of a data synchronization apparatus according to the present disclosure;
FIG. 7 is a schematic block diagram of a computer system suitable for use with an electronic device implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 of a data synchronization method or data synchronization apparatus to which embodiments of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, networks 104, 105, and servers 106, 107, 108, 109. Network 104 is the medium used to provide communication links between terminal devices 101, 102, 103 and server 106, and network 105 is the medium used to provide communication links between server 106 and servers 107, 108, 109. The networks 104, 105 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may use the terminal devices 101, 102, 103 to interact with the server 106 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a shopping application, a pick-up application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like. The terminal devices 101, 102, 103 may perform processing such as analysis on the transmission data to present a processing result (e.g., a data synchronization result) to the user.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and support, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, and the like.
The server 106 may be a synchronization processing server that provides various services, such as a synchronization processing server that provides support for data synchronization requests of the terminal apparatuses 101, 102, 103. The synchronization processing server may analyze and process the received data such as the synchronization request, and feed back the processing result to the terminal device.
The server 106 interacts with the servers 107, 108, 109 over the network 105 to receive or send messages and the like. For example, server 106 interacts with global clock server 107 to obtain global timestamps, and server 106 interacts with servers 108, 109 to store data sent by terminal devices 101, 102, 103 to servers 108, 109.
It should be noted that the data synchronization method provided by the embodiment of the present disclosure is generally executed by the server 106, and the corresponding data synchronization apparatus is generally disposed in the server 106.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of a data synchronization method according to the present disclosure is shown. The data synchronization method comprises the following steps:
In this embodiment, when an execution main body (for example, the synchronization processing server shown in fig. 1) of the method receives a data synchronization request sent by a client, the execution main body parses the synchronization request to obtain the data to be synchronized indicated by the synchronization request.
In this embodiment, after receiving a data synchronization request sent by a client, the execution main body sends a timestamp request to the global clock server, where sending the timestamp request to the global clock server may be performed in batch, and it is avoided that each piece of data needs to be requested once.
In this embodiment, after receiving the global timestamp returned by the global clock server, the execution main body sends a first storage request to the first service end, so that the data to be synchronized with the timestamp is sent to the first service end, where the first storage request includes the global timestamp and the data to be synchronized, and the global timestamp is a set of numbers indicating a current time and a sequence number.
In this embodiment, after receiving a storage success message corresponding to the first storage request returned by the first service, the execution main body sends a data request to the first service to obtain data information on the first service.
In this embodiment, after receiving the target synchronization data corresponding to the data request returned by the first server, the execution main body sends a second storage request to the second server, so as to synchronize the target synchronization data on the first server to the second server. Wherein the second storage request includes a global timestamp and target synchronization data.
And step 206, responding to the received data synchronization result returned by the second server and forwarding the data synchronization result to the client.
In this embodiment, when the execution body receives the data synchronization result returned by the second server, the execution body notifies the client of the synchronization result.
In some optional implementation manners of this embodiment, after receiving target synchronization data corresponding to the data request returned by the first server and sending a second storage request to the second server, the method further includes: in response to receiving a storage failure message corresponding to the second storage request returned by the second server, storing the target synchronous data and the global timestamp in a data recovery table; based on the data recovery table, a data recovery operation is performed. For example, the execution main body checks whether data to be restored exists in a data restoration table of the database at regular time, if the data to be restored exists in the data restoration table, the data to be restored is sent to the second server again so that the second server stores the data again, the data to be restored is deleted after a storage success message returned by the second server is received, and if the data to be restored does not exist in the data restoration table, the data restoration operation is not executed.
By storing the data which is not successfully stored in the data recovery table, the data loss during system failure is avoided, and the disaster tolerance capability and the safety of the system are improved.
In some optional implementation manners of this embodiment, the target synchronization data is full data corresponding to the data identifier in the first service-side database, where the data identifier is a unique identifier of a row of records corresponding to the data to be synchronized in the first service-side database. The target synchronization data acquired by the execution subject from the first server may be full data corresponding to the data identifier.
With continued reference to fig. 3, fig. 3 is a schematic diagram 300 of an application scenario of the data synchronization method according to the present embodiment. In the application scenario of fig. 3, a client 301 sends a data synchronization request 306 to the synchronization processing server 302, the synchronization processing server parses the synchronization request to obtain data to be synchronized, and sends a timestamp request 307 to the global clock server 303, after receiving a global timestamp 308 returned by the global clock server, the synchronization processing server sends a first storage request 309 to the first server 304, and when receiving a storage success message 310 returned by the first server, the synchronization processing server sends a data request 311 to the first server; and receiving target synchronization data 312 returned by the first server, sending a second storage request 313 to the second server 305, and after receiving a data synchronization result 314 returned by the second server, forwarding the data synchronization result to the client by the synchronization processing server.
The data synchronization method provided by the above embodiment of the present disclosure adopts a method that, in response to receiving a data synchronization request sent by a client, a synchronization request is analyzed to obtain data to be synchronized indicated by the synchronization request, a timestamp request is sent to a global clock server, in response to receiving a global timestamp returned by the global clock server, a first storage request is sent to a first server, in response to receiving a storage success message corresponding to the first storage request returned by the first server, a data request is sent to the first server, target synchronization data corresponding to the data request returned by the first server is received, a second storage request is sent to a second server, in response to receiving a data synchronization result returned by the second server and forwarded to the client, and a problem that data inconsistency between the first server (distributed storage system Hbase) and a data search analysis engine ES is caused when the second server (data search analysis engine ES) is operated in a multi-thread manner is overcome The data synchronization method and the data synchronization system have the advantages that the accuracy of data synchronization is improved, the data request is sent to the first server after the successful storage message returned by the first server is received, the target synchronization data corresponding to the data request returned by the first server is received, the second storage request is sent to the second server, a plurality of storage requests can be sent to the data search analysis engine ES at the same time, the problem that the whole data synchronization is blocked due to the fact that a single data fails to be synchronized in the prior art is solved, the data synchronization efficiency is improved, the influence on the performance and stability of the Hbase due to the fact that the data synchronization is conducted through the co-processing of the distributed storage system Hbase in the prior art is avoided through the synchronous processing server, and the performance of the Hbase is improved.
With further reference to fig. 4, a flow diagram of one embodiment of a data synchronization method applied to the second server is shown. The process 400 of the data synchronization method includes the following steps:
In this embodiment, the execution body of the method receives a storage request sent by the synchronization processing server, and parses the storage request to obtain target synchronization data and a global timestamp, where the global timestamp is a set of monotonically increasing numbers indicating a current time and a sequence number.
And step 402, updating the data corresponding to the target synchronous data and the version number of the data corresponding to the data ID based on the global timestamp, and returning a data synchronization result to the synchronous processing server.
In this embodiment, the execution main body updates data corresponding to the target synchronization data and updates the version number of the data corresponding to the data ID based on the global timestamp, and returns a data synchronization result to the synchronization processing server, where the data ID is a unique identifier of the data corresponding to the target synchronization data in the local database.
In some optional implementation manners of this embodiment, based on the global timestamp, updating data corresponding to the target synchronization data and updating a version number of the data corresponding to the target synchronization data, and returning a data synchronization result to the synchronization processing server, includes: judging whether the global timestamp is larger than the version number of the data corresponding to the data ID or not; if the global timestamp is greater than the version number of the data corresponding to the data ID, updating the data corresponding to the target synchronous data and updating the version number of the data corresponding to the data ID; and responding to the success of updating the data corresponding to the target synchronous data, and returning a data synchronization result to the synchronous processing server. And through judging the data version, the data corresponding to the maximum version number is always stored in the database.
In some optional implementation manners of this embodiment, after determining whether the global timestamp is greater than the version number of the data corresponding to the data ID, the method further includes: and if the global timestamp is not greater than the version number of the data corresponding to the data ID, returning a data synchronization result to the synchronization processing server. And ignoring the data synchronization request of the low version in the data synchronization process, so that the data corresponding to the maximum version number is always stored in the database.
In some optional implementation manners of this embodiment, after updating the data corresponding to the target synchronous data and updating the version number of the data corresponding to the data ID if the global timestamp is greater than the version number of the data corresponding to the data ID, the method further includes: and responding to the data failure corresponding to the updated target synchronous data, and sending a storage failure message to the synchronous processing server.
As can be seen from fig. 4, compared with the embodiment corresponding to fig. 2, the data synchronization method applied to the second server in the foregoing embodiment receives the storage request sent by the synchronization processing server, parses the storage request to obtain the target synchronization data and the global timestamp, updates the data corresponding to the target synchronization data and updates the version number of the data corresponding to the data ID based on the global timestamp, and returns the data synchronization result to the synchronization processing server, so that the problem that the data of the first server (distributed storage system Hbase) and the data search analysis engine ES are inconsistent when the second server (data search analysis engine ES) is operated in multiple threads is overcome, and the accuracy of data synchronization is improved.
With further reference to fig. 5, the present disclosure provides a data synchronization system, as shown in fig. 5, the system includes a synchronization processing server 501, a first service end 502, a second service end 503, and a global clock server 504, where the synchronization processing server is configured to execute the data synchronization method shown in fig. 2 to 3. The second server is configured to perform the data synchronization method shown in fig. 4. The first server is used for receiving a first storage request sent by the synchronous processing server, analyzing the first storage request to obtain data to be synchronized and a global timestamp, updating the timestamp of the data corresponding to the data to be synchronized and the data corresponding to the updated data identifier based on the global timestamp, wherein the data identifier is a unique identifier of a row of records corresponding to the data to be synchronized in the library, responding to the success of updating the data corresponding to the data to be synchronized, sending a storage success message corresponding to the first storage request to the synchronous processing server, receiving the data request sent by the synchronous processing server, obtaining target synchronous data corresponding to the data request in the library, and sending the target synchronous data to the synchronous processing server. And the global clock server is used for receiving the timestamp request sent by the synchronous processing server, acquiring a global timestamp corresponding to the current timestamp, and sending the global timestamp to the synchronous processing server.
In the system, the first server is further configured to determine whether the global timestamp is greater than a timestamp of data corresponding to the data identifier in the library, and update the data corresponding to the data to be synchronized and the timestamp of data corresponding to the updated data identifier if the global timestamp is greater than the timestamp of data corresponding to the data identifier in the library.
In the system, the first server is further configured to determine whether a data identifier exists in the library, and if the data identifier does not exist in the library, add data corresponding to the data to be synchronized and add a timestamp of the data corresponding to the data identifier.
With further reference to fig. 6, as an implementation of the method shown in fig. 2 to fig. 3, the present disclosure provides an embodiment of a data synchronization apparatus, which corresponds to the method embodiment shown in fig. 2, and which can be applied in various electronic devices.
As shown in fig. 6, the data synchronization apparatus 600 of the present embodiment includes: parsing unit 601, timestamp requesting unit 602, first storage requesting unit 603, data requesting unit 604, second requesting unit 605, and forwarding unit 606. The analysis unit 601 is configured to, in response to receiving a data synchronization request sent by a client, analyze the synchronization request to obtain data to be synchronized indicated by the synchronization request; a timestamp request unit 602 configured to send a timestamp request to the global clock server; a first storage request unit 603 configured to send a first storage request to the first server in response to receiving a global timestamp returned by the global clock server, where the first storage request includes the global timestamp and data to be synchronized, and the global timestamp is a set of numbers indicating a current time and a sequence number; a data request unit 604 configured to send a data request to the first server in response to receiving a storage success message corresponding to the first storage request returned by the first server; a second storage request unit 605, configured to receive target synchronous data corresponding to the data request returned by the first server, and send a second storage request to the second server, where the second storage request includes a global timestamp and the target synchronous data; and the forwarding unit 606 is configured to respond to the received data synchronization result returned by the second server and forward the data synchronization result to the client.
In this embodiment, specific processes of the parsing unit 601, the timestamp requesting unit 602, the first storage requesting unit 603, the data requesting unit 604, the second requesting unit 605 and the forwarding unit 606 of the data synchronization apparatus 600 and technical effects thereof may refer to the related descriptions of step 201 to step 206 in the embodiment corresponding to fig. 2, and are not described herein again.
In some optional implementations of this embodiment, the apparatus further includes: the storage unit is configured to respond to the received storage failure message which is returned by the second server and corresponds to the second storage request, and store the target synchronous data and the global timestamp in the data recovery table; a recovery unit configured to perform a data recovery operation based on the data recovery table.
In some optional implementation manners of this embodiment, the target synchronization data is full data corresponding to the data identifier in the first service-side database, where the data identifier is a unique identifier of a row of records corresponding to the data to be synchronized in the first service-side database.
The data synchronization device provided by the embodiment of the disclosure adopts the steps of responding to a received data synchronization request sent by a client, analyzing the synchronization request to obtain data to be synchronized indicated by the synchronization request, sending a timestamp request to a global clock server, responding to the received global timestamp returned by the global clock server, sending a first storage request to a first server, responding to a storage success message corresponding to the first storage request returned by the first server, sending a data request to the first server, receiving target synchronization data corresponding to the data request returned by the first server, sending a second storage request to a second server, responding to a data synchronization result returned by the second server, and forwarding the data synchronization result to the client, so as to overcome the problem that the data of the first server (distributed storage system Hbase) is inconsistent with the data of a data search analysis engine ES when the second server (data search analysis engine ES) is operated in a multi-thread manner, the accuracy of data synchronization is improved, because the data request is sent to the first server after the successful storage message returned by the first server is received, the target synchronization data corresponding to the data request returned by the first server is received, and the second storage request is sent to the second server, a plurality of storage requests can be sent to the data search analysis engine ES at the same time, the problem that the whole data synchronization is blocked due to the fact that a single data synchronization fails in the prior art is solved, the data synchronization efficiency is improved, the influence on the performance and stability of the Hbase caused by the fact that the data synchronization is carried out by the aid of co-processing of the distributed storage system Hbase in the prior art is avoided by the aid of the synchronization processing server, and the performance of the Hbase is improved.
Referring now to FIG. 7, a block diagram of an electronic device (e.g., the synchronization processing server of FIG. 1) 700 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The synchronization processing server shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, electronic device 700 may include a processing means (e.g., central processing unit, graphics processor, etc.) 701 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from storage 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the electronic apparatus 700 are also stored. The processing device 701, the ROM 702, and the RAM703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Generally, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 illustrates an electronic device 700 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 7 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication means 709, or may be installed from the storage means 708, or may be installed from the ROM 702. The computer program, when executed by the processing device 701, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: in response to receiving a data synchronization request sent by a client, analyzing the synchronization request to acquire data to be synchronized indicated by the synchronization request; sending a timestamp request to a global clock server; sending a first storage request to a first server in response to receiving a global timestamp returned by a global clock server, wherein the first storage request comprises the global timestamp and data to be synchronized, and the global timestamp is a group of numbers indicating the current time and a sequence number; sending a data request to the first server in response to receiving a storage success message corresponding to the first storage request returned by the first server; receiving target synchronous data corresponding to a data request returned by a first server, and sending a second storage request to a second server, wherein the second storage request comprises a global timestamp and the target synchronous data; and responding to the received data synchronization result returned by the second server and forwarding the data synchronization result to the client.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a parsing unit, a timestamp request unit, a first store request unit, a data request unit, a second request unit, and a forwarding unit. The names of these units do not in some cases constitute a limitation on the units themselves, for example, the parsing unit may also be described as "a unit that parses a synchronization request to acquire data to be synchronized indicated by the synchronization request in response to receiving a data synchronization request sent by a client".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (17)
1. A method of data synchronization, the method comprising:
in response to receiving a data synchronization request sent by a client, analyzing the synchronization request to acquire data to be synchronized indicated by the synchronization request;
sending a timestamp request to a global clock server;
sending a first storage request to a first server in response to receiving a global timestamp returned by the global clock server, wherein the first storage request comprises the global timestamp and the data to be synchronized, and the global timestamp is a group of numbers indicating the current time and a sequence number;
in response to receiving a storage success message corresponding to the first storage request returned by the first service terminal, sending a data request to the first service terminal;
receiving target synchronous data corresponding to the data request returned by the first server, and sending a second storage request to a second server, wherein the second storage request comprises the global timestamp and the target synchronous data;
and responding to the received data synchronization result returned by the second server and forwarding the data synchronization result to the client.
2. The data synchronization method according to claim 1, wherein after receiving the target synchronization data corresponding to the data request returned by the first server and sending a second storage request to a second server, the method further comprises:
in response to receiving a storage failure message corresponding to the second storage request returned by the second server, storing the target synchronization data and the global timestamp in a data recovery table;
and executing the data recovery operation based on the data recovery table.
3. The data synchronization method according to claim 1, wherein the target synchronization data is a full amount of data in the first server database corresponding to the data identifier, wherein the data identifier is a unique identifier of a row of records in the first server database corresponding to the data to be synchronized.
4. A method of data synchronization, the method comprising:
receiving a storage request sent by a synchronous processing server, and analyzing the storage request to acquire target synchronous data and a global timestamp, wherein the global timestamp is a group of monotonically increasing numbers indicating the current time and the sequence number;
and updating the data corresponding to the target synchronous data and the version number of the data corresponding to the data ID based on the global timestamp, and returning the data synchronization result to the synchronous processing server, wherein the data ID is the unique identifier of the data corresponding to the target synchronous data in the local database.
5. The data synchronization method according to claim 4, wherein the updating the data corresponding to the target synchronization data and the version number of the data corresponding to the target synchronization data based on the global timestamp, and returning the data synchronization result to the synchronization processing server includes:
judging whether the global timestamp is larger than the version number of the data corresponding to the data ID or not;
if the global timestamp is greater than the version number of the data corresponding to the data ID, updating the data corresponding to the target synchronous data and updating the version number of the data corresponding to the data ID;
and responding to the success of updating the data corresponding to the target synchronous data, and returning the data synchronization result to the synchronous processing server.
6. The data synchronization method according to claim 5, wherein after said determining whether the global timestamp is greater than a version number of data corresponding to the data ID, the method further comprises:
and if the global timestamp is not greater than the version number of the data corresponding to the data ID, returning the data synchronization result to the synchronization processing server.
7. The data synchronization method according to claim 5, wherein after updating the data corresponding to the target synchronization data and updating the version number of the data corresponding to the data ID if the global timestamp is greater than the version number of the data corresponding to the data ID, the method further comprises:
and responding to unsuccessful data updating corresponding to the target synchronous data, and sending the storage failure message to the synchronous processing server.
8. A data synchronization system, comprising: a synchronization processing server, wherein,
the synchronization processing server is used for executing the data synchronization method of any one of claims 1 to 3.
9. The data synchronization system of claim 8, wherein the system further comprises a second server;
the second server is configured to perform the data synchronization method according to any one of claims 4 to 7.
10. The data synchronization system of claim 8, wherein the system further comprises a first server and a global clock server;
the first server is configured to receive the first storage request sent by a synchronization processing server, parse the first storage request to obtain data to be synchronized and a global timestamp, update the timestamp of the data corresponding to the data to be synchronized and the timestamp of the data corresponding to an update data identifier based on the global timestamp, where the data identifier is a unique identifier of a row of records corresponding to the data to be synchronized in a library, send a storage success message corresponding to the first storage request to the synchronization processing server in response to a success in updating the data corresponding to the data to be synchronized, receive the data request sent by the synchronization processing server, obtain target synchronization data corresponding to the data request in the library, and send the target synchronization data to the synchronization processing server.
The global clock server is used for receiving a timestamp request sent by the synchronous processing server, acquiring a global timestamp corresponding to the current timestamp, and sending the global timestamp to the synchronous processing server.
11. The data synchronization system according to claim 10, wherein the first server is further configured to determine whether the global timestamp is greater than a timestamp of data corresponding to the data identifier in the repository, and update the data corresponding to the data to be synchronized and update the timestamp of the data corresponding to the data identifier if the global timestamp is greater than the timestamp of the data corresponding to the identifier in the repository.
12. The data synchronization system according to claim 10, wherein the first server is further configured to determine whether the data identifier exists in a repository, and if the data identifier does not exist in the repository, add the data corresponding to the data to be synchronized and add the timestamp of the data corresponding to the data identifier.
13. A data synchronization apparatus, the apparatus comprising:
the data synchronization device comprises an analysis unit and a synchronization unit, wherein the analysis unit is configured to respond to a data synchronization request sent by a client and analyze the synchronization request to acquire data to be synchronized indicated by the synchronization request;
a timestamp request unit configured to send a timestamp request to the global clock server;
the first storage request unit is configured to send a first storage request to a first server in response to receiving a global timestamp returned by the global clock server, wherein the first storage request comprises the global timestamp and the data to be synchronized, and the global timestamp is a group of numbers indicating the current time and a sequence number;
a data request unit configured to send a data request to the first service end in response to receiving a storage success message corresponding to the first storage request returned by the first service end;
a second storage request unit configured to receive target synchronous data corresponding to the data request returned by the first server, and send a second storage request to a second server, where the second storage request includes the global timestamp and the target synchronous data;
and the forwarding unit is configured to respond to the received data synchronization result returned by the second server and forward the data synchronization result to the client.
14. The data synchronization apparatus of claim 13, the apparatus further comprising:
the storage unit is configured to store the target synchronous data and the global timestamp in a data recovery table in response to receiving a storage failure message corresponding to the second storage request and returned by the second server;
a recovery unit configured to perform the data recovery operation based on the data recovery table.
15. The data synchronization apparatus according to claim 13, wherein the target synchronization data is a full amount of data in the first server database corresponding to the data identifier, wherein the data identifier is a unique identifier of a row of records in the first server database corresponding to the data to be synchronized.
16. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-3.
17. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010099242.6A CN111309747A (en) | 2020-02-18 | 2020-02-18 | Data synchronization method, system and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010099242.6A CN111309747A (en) | 2020-02-18 | 2020-02-18 | Data synchronization method, system and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111309747A true CN111309747A (en) | 2020-06-19 |
Family
ID=71149066
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010099242.6A Pending CN111309747A (en) | 2020-02-18 | 2020-02-18 | Data synchronization method, system and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309747A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112381348A (en) * | 2020-10-13 | 2021-02-19 | 广州赛意信息科技股份有限公司 | Production data real-time clock synchronization method and system based on API gateway |
| CN113176806A (en) * | 2021-04-19 | 2021-07-27 | 北京思特奇信息技术股份有限公司 | Real unattended data timing synchronization method |
| CN113407498A (en) * | 2021-06-18 | 2021-09-17 | 北京金山云网络技术有限公司 | Data synchronization method, device, cloud platform, system and storage medium |
| CN113742426A (en) * | 2021-09-08 | 2021-12-03 | 深圳市云鼠科技开发有限公司 | Data synchronization method, device, equipment, server and storage medium |
| CN114168306A (en) * | 2022-02-10 | 2022-03-11 | 北京奥星贝斯科技有限公司 | Scheduling method and scheduling device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101064630A (en) * | 2006-04-24 | 2007-10-31 | 华为技术有限公司 | Data synchronization method and system |
| CN105933352A (en) * | 2016-07-05 | 2016-09-07 | 广州华多网络科技有限公司 | Data synchronization method and system between servers based on client and client |
| CN107172169A (en) * | 2017-05-27 | 2017-09-15 | 广东欧珀移动通信有限公司 | Method of data synchronization, device, server and storage medium |
| CN107943594A (en) * | 2016-10-13 | 2018-04-20 | 北京京东尚科信息技术有限公司 | Data capture method and device |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN108712505A (en) * | 2018-05-31 | 2018-10-26 | 北京百度网讯科技有限公司 | Method of data synchronization, device, equipment, system and storage medium |
| CN109885581A (en) * | 2019-03-14 | 2019-06-14 | 苏州达家迎信息技术有限公司 | Synchronous method, device, equipment and the storage medium of database |
| CN109905476A (en) * | 2012-09-22 | 2019-06-18 | 谷歌有限责任公司 | The subscription informing mechanism of synchronization for distributions |
-
2020
- 2020-02-18 CN CN202010099242.6A patent/CN111309747A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101064630A (en) * | 2006-04-24 | 2007-10-31 | 华为技术有限公司 | Data synchronization method and system |
| CN109905476A (en) * | 2012-09-22 | 2019-06-18 | 谷歌有限责任公司 | The subscription informing mechanism of synchronization for distributions |
| CN105933352A (en) * | 2016-07-05 | 2016-09-07 | 广州华多网络科技有限公司 | Data synchronization method and system between servers based on client and client |
| CN107943594A (en) * | 2016-10-13 | 2018-04-20 | 北京京东尚科信息技术有限公司 | Data capture method and device |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN107172169A (en) * | 2017-05-27 | 2017-09-15 | 广东欧珀移动通信有限公司 | Method of data synchronization, device, server and storage medium |
| CN108712505A (en) * | 2018-05-31 | 2018-10-26 | 北京百度网讯科技有限公司 | Method of data synchronization, device, equipment, system and storage medium |
| CN109885581A (en) * | 2019-03-14 | 2019-06-14 | 苏州达家迎信息技术有限公司 | Synchronous method, device, equipment and the storage medium of database |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112381348A (en) * | 2020-10-13 | 2021-02-19 | 广州赛意信息科技股份有限公司 | Production data real-time clock synchronization method and system based on API gateway |
| CN113176806A (en) * | 2021-04-19 | 2021-07-27 | 北京思特奇信息技术股份有限公司 | Real unattended data timing synchronization method |
| CN113407498A (en) * | 2021-06-18 | 2021-09-17 | 北京金山云网络技术有限公司 | Data synchronization method, device, cloud platform, system and storage medium |
| CN113742426A (en) * | 2021-09-08 | 2021-12-03 | 深圳市云鼠科技开发有限公司 | Data synchronization method, device, equipment, server and storage medium |
| CN114168306A (en) * | 2022-02-10 | 2022-03-11 | 北京奥星贝斯科技有限公司 | Scheduling method and scheduling device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110609872B (en) | Method and apparatus for synchronizing node data | |
| CN111309747A (en) | Data synchronization method, system and device | |
| CN110909521B (en) | Online document information synchronous processing method and device and electronic equipment | |
| CN111432001B (en) | Method, apparatus, electronic device and computer readable medium for jumping scenes | |
| CN111163336B (en) | Video resource pushing method and device, electronic equipment and computer readable medium | |
| CN112015816A (en) | Data synchronization method, device, medium and electronic equipment | |
| CN111881216A (en) | Data acquisition method and device based on shared template | |
| CN111338834A (en) | Data storage method and device | |
| CN111694629A (en) | Information display method and device and electronic equipment | |
| CN111309366B (en) | Method, device, medium and electronic equipment for managing registration core | |
| CN111813685B (en) | Automatic test method and device | |
| CN114116247A (en) | Redis-based message processing method, device, system, server and medium | |
| CN110619101B (en) | Method and apparatus for processing information | |
| CN111596992A (en) | Navigation bar display method and device and electronic equipment | |
| CN112015746B (en) | Data real-time processing method, device, medium and electronic equipment | |
| CN115082038A (en) | System integration method and device and electronic equipment | |
| CN114115941A (en) | Resource sending method, page rendering method, device, electronic equipment and medium | |
| CN113742376A (en) | Data synchronization method, first server and data synchronization system | |
| CN112311842A (en) | Method and device for information interaction | |
| CN110750424A (en) | Resource inspection method and device | |
| CN110727694A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111294657A (en) | Information processing method and device | |
| US20240152504A1 (en) | Data interaction method, apparatus, and electronic device | |
| CN111400322B (en) | Method, apparatus, electronic device and medium for storing data | |
| CN114513548B (en) | Directional call information processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Applicant before: Jingdong Digital Technology Holding Co.,Ltd. Address after: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Applicant after: Jingdong Digital Technology Holding Co.,Ltd. Address before: Room 221, 2nd floor, Block C, 18 Kechuang 11th Street, Daxing Economic and Technological Development Zone, Beijing, 100176 Applicant before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd. |