CN112615907B

CN112615907B - Data synchronization system and method

Info

Publication number: CN112615907B
Application number: CN202011412068.2A
Authority: CN
Inventors: 周杨; 田强
Original assignee: Beijing Cheerbright Technologies Co Ltd
Current assignee: Beijing Cheerbright Technologies Co Ltd
Priority date: 2020-12-04
Filing date: 2020-12-04
Publication date: 2022-09-06
Anticipated expiration: 2040-12-04
Also published as: CN112615907A

Abstract

The invention discloses a data synchronization system, a data synchronization method and a data acquisition method, wherein the data synchronization system comprises: the system comprises a network server, a message queue server, a consumer server and a cache; the network server is suitable for saving updated data to a database and sending a data updating message to the message queue server, wherein the data updating message comprises a unique Identification (ID) of the updated data; the message queue server is suitable for receiving the data updating message sent by the network server and storing the data updating message to a message queue; and the consumer server is suitable for subscribing the message queue of the message queue server, consuming the data updating message in the message queue, searching the data corresponding to the ID from the database, and selecting part of data from the searched data corresponding to the ID to be synchronized to the cache.

Description

Data synchronization system and method

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data synchronization system, a data synchronization method, a computing device, and a storage medium.

Background

The internet system is a general term of application software and websites, the performance of the internet system will affect the experience of users, however, the efficiency of acquiring data is an important expression form of the performance of the internet system.

At present, in order to improve the efficiency of acquiring data by an internet system, hot spot data is mostly stored in a cache, the cache time of the hot spot data is set, and the hot spot data in the cache is updated after the set cache time is reached. However, since the data in the cache needs to be updated after the set cache time elapses, the hot data cannot be cached in time, and thus the performance of the internet system is affected.

For this reason, a new data synchronization scheme is needed, which can synchronize data to the cache in real time.

Disclosure of Invention

To this end, the present invention provides a data synchronization system, method and computing device in an attempt to solve or at least alleviate the problems presented above.

According to an aspect of the present invention, there is provided a data synchronization system, the system comprising: the system comprises a network server, a message queue server, a consumer server and a cache;

the network server is suitable for saving updated data to a database and sending a data updating message to the message queue server, wherein the data updating message comprises a unique Identification (ID) of the updated data;

the message queue server is suitable for receiving the data updating message sent by the network server and storing the data updating message to a message queue;

and the consumer server is suitable for subscribing the message queue of the message queue server, consuming the data updating message in the message queue, searching the data corresponding to the ID from the database, and selecting part of data from the searched data corresponding to the ID to be synchronized to the cache.

Optionally, the system further comprises a client adapted to user trigger the update data interface and the save data interface, and the web server is further adapted to:

and responding to the triggering of an update data interface and a storage data interface at the client by a user, and storing the update data in the database.

Optionally, the network server is further adapted to:

presetting dormancy time and retry times;

judging whether a data updating message is sent to the message queue server or not within preset dormancy time;

if the data updating message is not sent to the message queue server, judging whether the retry number is smaller than the preset retry number;

and if the retry times are less than the preset retry times, retransmitting the data updating message to the message queue server through a fault-tolerant mechanism, wherein the retry times are increased by 1, and if the retry times are not less than the preset retry times, stopping transmitting the data updating message to the message queue server.

Optionally, the consumer server is further adapted to:

searching data corresponding to the ID from the database;

selecting part of data and service state data from the data corresponding to the ID;

and synchronizing the selected partial data to the cache according to the service state data.

Optionally, the consumer server is further adapted to:

and synchronizing the selected part of data to the key value pair of the cached ordered set according to the service state data by taking the ID as a key of the key value pair in the ordered set and the selected part of data as key values of the key value pair in the ordered set.

Optionally, the cache includes one or more of the ordered sets.

Optionally, the client further comprises a data acquisition interface, the consumer server is further adapted to:

and splitting key values of the key value pairs in the ordered set, and recording the splitting number so that when a user triggers the data acquisition interface at the client, acquiring the key values of the key value pairs corresponding to the data ID requested by the interface in a multithreading concurrent mode, and returning the key values to the client.

Optionally, the consumer server is further adapted to:

and judging whether the selected partial data is synchronized to the cache, if not, re-synchronizing the selected partial data which fails to be synchronized to the cache by utilizing a fault-tolerant mechanism and a retry mechanism.

Optionally, the consumer server is further adapted to:

and checking whether the part of the data synchronized to the cache is consistent with the corresponding part of the updated data in the database.

Optionally, the consumer server is further adapted to:

periodically scanning a data table of the database;

determining the updating data by searching the updating data of the data table in a preset time period;

and comparing the part of the data synchronized to the cache with the corresponding part of the updated data in the data table, and if the part of the data synchronized to the cache is not consistent with the corresponding part of the selected part of the data, synchronizing the part of the updated data corresponding to the selected part of the data to the cache again.

According to another aspect of the present invention, there is provided a data synchronization method adapted to operate in the data synchronization system described above, the method comprising:

the network server saves the updated data to a database and sends a data updating message to a message queue server, wherein the data updating message comprises a unique Identification (ID) of the updated data;

the message queue server receives the data updating message sent by the network server and stores the data updating message to a message queue;

and the consumer server subscribes the message queue of the message queue server, consumes the data updating message in the message queue, searches the data corresponding to the ID from the database, and selects part of data from the searched data corresponding to the ID to be synchronized to the cache.

Optionally, the saving, by the network server, the updated data to the database comprises:

and the network server responds to the condition that the user triggers the update data interface and the storage data interface at the client and stores the update data into the database.

Optionally, the method further comprises:

the network server presets dormancy time and retry times;

in a preset sleep time, the network server judges whether a data updating message is sent to the message queue server;

if the data updating message is not sent to the message queue server, the network server judges whether the retry times are smaller than the preset retry times;

if the retry times are less than the preset retry times, the network server resends the data updating message to the message queue server through a fault-tolerant mechanism and a retry mechanism, the retry times are increased by 1, and if the retry times are not less than the preset retry times, the data updating message is stopped from being sent to the message queue server.

Optionally, the consumer server searches the database for data corresponding to the ID, and the step of selecting a part of data from the searched data corresponding to the ID to be synchronized to the cache comprises:

the consumer server searching the database for data corresponding to the ID;

Optionally, the synchronizing, by the consumer server, the selected part of the data to the cache according to the service state data includes:

and synchronizing the selected part of data to the key value pairs of the cached ordered set according to the service state data by taking the ID as a key of the key value pairs in the ordered set and the selected part of data as key values of the key value pairs in the ordered set.

Optionally, the cache includes one or more of the ordered sets.

Optionally, the client further includes a data obtaining interface, and the method further includes:

and the consumer server splits the key values of the key value pairs in the ordered set and records the split number, so that when a user triggers the data acquisition interface at the client, the key values of the key value pairs corresponding to the data ID acquired by the interface request are acquired in a multithreading concurrent mode and returned to the client.

Optionally, the method further comprises:

and the consumer server judges whether to synchronize the selected partial data to the cache, and if not, the selected partial data which fails to be synchronized is resynchronized to the cache by utilizing a fault-tolerant mechanism and a retry mechanism.

Optionally, the method further comprises:

the consumer server checks whether the portion of data synchronized to the cache is consistent with the corresponding portion of updated data in the database.

Optionally, the verifying, by the consumer server, whether the part of the data synchronized to the cache is consistent with the corresponding part of the updated data in the database comprises:

periodically scanning a data table of the database;

and comparing the part of the data synchronized to the cache with the corresponding part of the updated data in the data table, and if the part of the data synchronized to the cache is inconsistent with the corresponding part of the selected part of the data, synchronizing the part of the updated data corresponding to the selected part of the data to the cache again.

According to another aspect of the present invention, there is provided a data acquisition method, performed in a network server, the method including:

receiving an HTTP request sent by a client, and analyzing the HTTP request to obtain a data Identifier (ID) of data requested to be obtained by the client;

determining an ordered set of a cache where the client requests to acquire data and the number of key value splits of key value pairs of the data, wherein the cached data synchronize the data to the cache according to the data synchronization method;

reading a key value of a key value pair corresponding to the data requested to be obtained by the client from the cache in a multithread concurrent mode;

and sending the key value of the read key value pair to the client through the HTTP response.

According to another aspect of the invention, there is provided a computing device comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the method as described above.

According to yet another aspect of the present invention, there is provided a readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method as described above.

The invention provides a data synchronization method, which is characterized in that a network server sends a data updating message to a message queue server, the message queue server stores the message to a message queue, a consumer server consumes the data updating message in the message queue and synchronously updates data to a cache based on the data updating message, and the asynchronous operation of data updating and data caching is realized through the message queue, so that the data in the cache is updated in real time, namely, the changed data in a database can be synchronized to the cache in real time, and the normal operation of data updating caused by the cache can be effectively avoided.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.

FIG. 1 shows a schematic diagram of a data synchronization system 100 according to one embodiment of the invention;

FIG. 2 illustrates a block diagram of a computing device 200, according to one embodiment of the invention;

FIG. 3 shows a flow diagram of a data synchronization method 300 according to one embodiment of the invention;

FIG. 4 shows a flow diagram of a data acquisition method 400 according to one embodiment of the invention;

fig. 5 is a schematic diagram illustrating a public praise presented by the client 110 of the home website of the automobile according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

The internet system is a general term of application software and websites, the performance of the internet system will affect the experience of users, however, the efficiency of acquiring data is an important expression form of the performance of the internet system. At present, in order to improve the efficiency of obtaining data by an internet system, a common mode is to add a cache in a data interface, store data in the cache in a key-value pair manner, use a parameter requested by the data interface as a key of the key-value pair, use data updated and stored by a client as a key of the key-value pair, so as to synchronize the data updated at the client to the cache, and directly obtain the data from the cache when a user requests to obtain the data, thereby reducing the access amount of a database, relieving the phenomena of blocking and the like caused by large access amount of the database, improving the efficiency of obtaining data by the user, namely improving the performance of the internet system.

However, in the current cache mode pair, the time for storing data in the cache is preset, and the data in the cache can be updated again after the preset time (for example, 30 minutes) is reached, and even if updated data exists in the preset time, the corresponding data in the cache cannot be updated in time under the condition that the cache time of the corresponding data in the cache does not reach the preset time, that is, if the storage time of a certain data in the cache does not reach 30 minutes, the data cannot be updated in the cache even if the data is updated in 30 minutes, and the corresponding data is updated only after the preset time of 30 minutes is reached. Therefore, the data in the cache can be updated only after the preset time is expired, so that the cache cannot update the data in real time.

To solve the above problem, the present invention provides a data synchronization system, and fig. 1 shows a block diagram of a data synchronization system 100 according to an embodiment of the present invention. As shown in FIG. 1, the data synchronization system includes a client 110, a web server 120, a database 130, a message queue server 140, a consumer server 150, and a cache 160. Client 110 is coupled to web server 120, web server 120 is coupled to database 130, message queue server 140, and cache 160, respectively, database 130 is coupled to client 110, web server 120, and consumer server 150, respectively, and consumer server 150 is coupled to database 130, message queue server 140, and cache 160, respectively.

The client 110 is a terminal device used by a user, and may specifically be a personal computer such as a desktop computer and a notebook computer, or may also be a mobile phone, a tablet computer, a multimedia device, an intelligent wearable device, and the like, but is not limited thereto. The user interaction interface of the client 110 includes interfaces such as a data update interface, a data saving interface, and a data obtaining interface, and a user can update data on the user interaction interface of the client 110 by triggering the data update interface, and save the updated or published data to the database 130 by triggering the data saving interface, and can also obtain data from the database 130 or the cache 160 through the network server 120 by triggering the data obtaining interface.

There may be multiple web servers, with multiple web servers 120 forming a web server cluster, that collectively provide services for saving updated data to the database 130, and for sending data update messages to the message queue server 140.

The database 130 may be a relational database such as MySQL, ACCESS, etc., or a non-relational database such as NoSQL, etc.; the database may be a local database residing in the computing device, or may be a distributed database, such as HBase, etc., disposed at multiple geographic locations, in short, the database 130 is used to store data published or updated by the user at the client 110, and the present invention does not limit the specific deployment and configuration of the database 130.

The number of the message queue servers 140 may be one or more, where one or more message queue servers 140 form a message queue server cluster, and the message queue server cluster provides a service for receiving the data update message sent by the network server 120 and storing the data update message in a message queue.

Likewise, the consumer server 150 includes one or more, and one or more consumer servers 150 form a consumer server cluster, and the consumer server cluster collectively provides messages subscribing to the message queue and consuming the messages of the message queue, and services capable of accessing the database 130 and the cache 160, and the present invention is not limited to the specific deployment and configuration of the consumer server 150.

The cache 160 may be a redis cache, and the present invention does not limit the specific type, deployment and configuration of the cache.

The data synchronization system of the present invention starts with a user triggering an update data interface at a client terminal 110, when the user triggers the update data interface at a user interaction interface of the client terminal 110, a web server 120 responds to a request of the user to pop up a window, the user can make actions of publishing new data, modifying data, deleting data or not modifying data at the window, and triggers a save data interface, an operator can review the data published by the user, mark that an audit state is pass or not pass, and use the audit state as service state data, the web server 120 responds to the user to trigger the save data interface, save the updated data and the service state data to a database 130, and send a data update message to a message queue server 140.

It should be noted that only a unique Identification (ID) of the update data is included in the data update message, and the web server 120 assigns an ID to each piece of data published by the user. For example, each time a user posts a word-of-mouth (i.e., the user posts a comment) on the user interaction interface of the home website of the automobile, the web server 120 assigns an ID to the word-of-mouth, and if the same user posts three word-of-mouth, each word-of-mouth is assigned an ID.

In an embodiment of the present invention, as long as the user triggers the update data interface and the save data interface at the user interaction interface of the client 110, the web server 120 will save the data to the database 130 and send the data update message to the message queue server 140 even if the user does not actually modify the data.

The operation after the user triggers the update of the data interface comprises adding data, deleting data, modifying data, not doing any operation and the like. If the user enters new data in the window, the web server 120 assigns an ID to the new data and stores the entered data corresponding to the ID in the database 130. If the user modifies and deletes the stored data in the update window, the web server 120 does not reassign the ID to the data, stores the modified data in the database 130 by using the original data ID, or deletes the data corresponding to the ID from the database 130 by using the original data ID. If the user does not perform any operation in the update window, the web server 120 will replace the data corresponding to the original ID in the database with the current data, and the data is substantially unchanged, and only one data replacement operation is performed.

For example, a user at a home website of an automobile posts a new word-of-mouth (e.g., a comment to audi A6L) on the user interaction interface of the client 110 and triggers the save data interface, and the web server 120 assigns an ID to the new word-of-mouth posted by the user, saves the new word-of-mouth to the database 130, and sends a data update message to the message queue server 140 in response to the user triggering the save data interface. If the user modifies the published public word in the user interaction interface of the client 110, for example, adds a comment or modifies the published comment, and triggers the data storage interface, the web server 120 responds to the user's trigger operation on the data storage interface, stores the added comment or modified comment into the database 130 based on the original public word ID, and sends a data update message to the message queue server 140. If the user does not perform any action on the user interaction interface of the client terminal 110, but the data saving interface is also triggered, at this time, the web server 120 replaces the original public praise with the current public praise in the database based on the original public praise ID in response to the user's trigger operation on the data saving interface, and sends a data update message to the message queue server 140.

In the process of sending the data update message to the message queue server 140 by the network server 120, a situation that the data update message is sent unsuccessfully may occur due to a network flash, and the like, and then the network server 120 may preset a sleep time and retry times in order to ensure that the data update message can be sent to the message queue server 140. In an embodiment of the present invention, the network server 120 sets a sleep time by using a read () function of java, the sleep time is generally set to about 200ms, the retry number may be set to 10, within a preset sleep time (200ms), the network server 120 determines whether the data update message is sent to the message queue server 140, if not sent to the message queue server 140, determines whether the retry number is less than a preset retry number, if the retry number is less than the preset retry number 10, resends the data update message to the message queue server 140 through a fault tolerance mechanism, the retry number increases by 1, and if the retry number is greater than the preset retry number 10, stops sending the data update message to the message queue server 140.

In a real-time mode of the invention, the fault-tolerant mechanism is a try catch mechanism, whether the retry number is smaller than the preset retry number is judged by using the judgment result, if the retry number is smaller than the preset retry number, the data updating message is retransmitted to the message queue server through the fault-tolerant mechanism, the retry number is increased by 1, while the retry mechanism is realized based on the while loop and the set retry number (10), thereby ensuring that the network server 120 sends the data updating message to the message queue server 140 through the fault-tolerant mechanism and the retry mechanism when the network flash happens.

After the network server 120 sends the data update message to the message queue server 140, the message queue server 140 receives the data update message sent by the network server 120 and stores the data update message in a message queue.

Thereafter, the consumer server 150 subscribes to the message queue of the message queue server 140, consumes the data update message in the message queue, searches the database 130 for data corresponding to the ID, and selects a part of the data corresponding to the ID from the searched data to be synchronized to the cache 160.

Specifically, the consumer server 150 subscribes to the message queue of the message queue server 140, continuously polls the message queue, and when there is a new data update message, pulls the data update message from the message queue, acquires a data ID from the data update message, searches for data corresponding to the ID from the database 130, and selects a part of data and service status data from the data corresponding to the ID. Wherein the business status data comprises a pass or no pass status of the operator audit user public praise flag as described above.

The selected part of data is selected from each piece of data stored in the database 130, and is stored in the cache 160, for example, a public praise published by a user in a home website of an automobile is a piece of data, each public praise corresponds to an ID, each public praise includes evaluation contents of fields such as addition, most satisfaction, least satisfaction, space, power, control, oil consumption, comfort, appearance, interior, cost performance, and the like, and part of data is selected from data corresponding to the ID, that is, comment contents of part of fields are selected from a public praise, for example, comment contents of three fields such as addition, most satisfaction, and satisfaction are selected from a public praise.

In the invention, the data updating message is sent to the message queue server 140 through the network server 120, the message queue server 140 stores the message into the message queue, the consumer server 150 consumes the data updating message in the message queue and synchronously updates the data to the cache 160 based on the data updating message, thereby realizing asynchronous operation of data updating and data caching, enabling the data in the cache 160 to be updated in real time, namely, the changed data in the database 130 can be synchronized to the cache 160 in real time, and also effectively avoiding the normal operation of data updating influenced by the blockage of the cache 160.

In an embodiment of the present invention, the consumer server 150 uses the ID as a key (key) of a key-value pair in the ordered set, and uses the selected partial data as a key (value) of a key-value pair in the ordered set, and since the selected partial data may occupy more memory, the value of the key-value pair in the cache occupies too much memory to affect the efficiency of reading data from the cache 160, for example, when the value exceeds 10KB, the performance of the cache 160 is affected. In order to subsequently read data from the cache efficiently, the consumer server 150 splits value into a plurality of pieces, the key of each split key value is the same and is an ID, and records the split number. That is, the partial data selected under one ID corresponds to a plurality of key-value pairs, the key of each key-value pair is the ID, the value of each key-value pair is a part of the selected data, and the sum of the values of the plurality of key-value pairs is all the selected data.

It is noted that there is timeliness in updating data, which means that the order of sending messages is the same as the order of receiving messages, for example: if the message receiver receives the message B first and then receives the message A, the receiving sequence of the messages is disordered.

Next, in order to ensure timeliness of the updated data, the consumer server 150 synchronizes the selected partial data to the cache 160 according to the service status data, specifically: firstly, acquiring service state data, determining the state of the service state data, if the service state data is in a pass state, storing the key value pairs corresponding to the selected partial data into one or more ordered sets of the cache 160, and if the service state data is in a fail state, not storing the selected partial data into the cache 160.

For example, if the public praise ID issued by the user is 00000018, the consumer server obtains the state data of 00000018 from the database 130 according to the ID, and if the service state data passes, selects the contents of the three fields of addition, most satisfaction and least satisfaction from the evaluation contents of the fields of addition, most satisfaction, least satisfaction, space, power, control, oil consumption, comfort, appearance, interior, cost performance and the like of the public praise with the ID of 00000018, and divides the contents of the fields of addition, most satisfaction and least satisfaction into three parts, which are the contents of the field of addition, most satisfaction and least satisfaction respectively, and divides the selected data into three key values to combine into three key value pairs with 00000018 as the key of the key value pair, which are: (00000018: additional field data "), (00000018: most satisfied field data"), and (00000018: least satisfied field data "), record the value split number as 3, and save the 3 key value pairs corresponding to the selected partial data into one or more ordered sets of the cache 160. If the service status data corresponding to 00000018 does not pass, the selected part of data will not be saved in the buffer 160.

The consumer server 150 splits the values of the key-value pairs in the ordered set, and records the split number, so that when the client 110 triggers the data acquisition interface, a user acquires the values of the key-value pairs corresponding to the interface request acquisition data ID in a multithreading concurrent manner, and returns the values to the client 110, and therefore, when the cached values of the key-value pairs are large, the efficiency of reading data from the cache is not affected.

Since a data synchronization failure may occur due to a network flash during the process of synchronizing the data to the cache 160, in order to prevent the data synchronization failure, the consumer server 150 determines whether to synchronize the selected partial data to the cache 160, and if not, re-synchronizes the selected partial data that has failed to be synchronized to the cache 160 by using a fault-tolerant mechanism and a retry mechanism. In an embodiment of the present invention, the fault tolerant mechanism is a try catch mechanism, and if the retry number is smaller than the preset retry number, the retry number is determined to be smaller than the preset retry number, and if the retry number is smaller than the preset retry number, the data update message is re-sent to the message queue server through the fault tolerant mechanism, and the retry number is increased by 1 ", so as to implement the retry mechanism, and based on the while loop and the set retry number (10), thereby ensuring that the consumer server 150 synchronizes the selected data to the cache 160 through the fault tolerant mechanism and the retry mechanism when the network flash occurs.

To address this problem, in the scenario where a data update, data cache asynchronous mechanism is used, the consumer server 150 may also check whether the portion of data synchronized to the cache 160 is consistent with the corresponding portion of the updated data in the database 130, specifically: if the database 130 is a relational database, the consumer server 150 periodically scans the data table of the database 130, determines the updated data by searching the updated data of the data table within a preset time period, compares part of the data synchronized to the cache 160 with the corresponding part of the updated data in the data table, synchronizes the part of the updated data corresponding to the selected part of the data to the cache 160 again if the updated data is inconsistent with the corresponding part of the updated data in the data table, pulls the data from the cache again for comparison after pushing is completed, and if the updated data is inconsistent with the selected part of the data, a program error may occur and manual intervention is notified.

For example, the ID of the updated data is 00000018, the consumer server 150 queries the data updated in the previous day of the data table in the database 130, searches data corresponding to 00000018 (including evaluation contents of the fields of addition, most satisfactory, least satisfactory, space, power, control, fuel consumption, comfort, appearance, interior and cost performance) from the updated data, compares the contents of the three fields of addition, most satisfactory and least satisfactory in the cache with the contents of the fields of addition, most satisfactory and least satisfactory in the database 130, and completes data synchronization if the contents of the three fields of addition, most satisfactory and least satisfactory are the same as the contents of the fields of addition, most satisfactory and least satisfactory in the database 130. If the field contents are not the same, the contents of the three fields of the most satisfactory field and the least satisfactory field corresponding to 00000018 in the database 130 are synchronized to the cache 160 again, after the pushing is completed, the data corresponding to 00000018 is pulled from the cache 160 again for comparison, if the data are not consistent, the data may be a program bug, a log needs to be recorded, and manual intervention is notified.

It should be noted that the data synchronization system 100 shown in fig. 1 is only exemplary, and in a specific implementation, different numbers of the web servers 120, the message queue servers 140, and the consumer servers 150 may be deployed, and the number and the deployment area of these devices are not limited by the present invention.

The web server 120, the message queue server 140, and the consumer server 150 in the data synchronization system 100 may be implemented as a single computing device, but the invention is not limited to the specific type of device of the server, for example, the web server 120, the message queue server 140, and the consumer server 150 may be implemented as a desktop computer, a notebook computer, a processor chip, a mobile phone, a tablet computer, or other computing devices, but not limited thereto.

Fig. 2 shows a block diagram of a computing device 200 according to an embodiment of the present invention, and it should be noted that the computing device 200 shown in fig. 2 is only an example, and in practice, the computing device for implementing the data synchronization method and the data acquisition method of the present invention may be any type of device, and the hardware configuration thereof may be the same as the computing device 200 shown in fig. 2 or different from the computing device 200 shown in fig. 2. In practice, the computing device for implementing the data synchronization method and the data acquisition method of the present invention may add or delete hardware components of the computing device 200 shown in fig. 2, and the present invention does not limit the specific hardware configuration of the computing device.

As shown in FIG. 2, in a basic configuration 202, computing device 200 typically includes system memory 206 and one or more processors 204. A memory bus 208 may be used for communicating between the processor 204 and the system memory 206.

Depending on the desired configuration, the processor 204 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a digital information processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.

Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The physical memory in the computing device is usually referred to as volatile memory RAM, and data in the disk needs to be loaded into the physical memory to be read by the processor 204. System memory 206 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, the application 222 can be arranged to execute instructions on the operating system with the program data 224 by the one or more processors 204. Operating system 220 may be, for example, Linux, Windows, or the like, which includes program instructions for handling basic system services and for performing hardware-dependent tasks. The application 222 includes program instructions for implementing various user-desired functions, and the application 222 may be, for example, but not limited to, a browser, instant messenger, a software development tool (e.g., an integrated development environment IDE, a compiler, etc.), and the like. When the application 222 is installed into the computing device 200, a driver module may be added to the operating system 220.

When the computing device 200 is started, the processor 204 reads program instructions of the operating system 220 from the memory 206 and executes them. Applications 222 run on top of operating system 220, utilizing the interface provided by operating system 220 and the underlying hardware to implement various user-desired functions. When the user starts the application 222, the application 222 is loaded into the memory 206, and the processor 204 reads the program instructions of the application 222 from the memory 206 and executes the program instructions.

Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices such as a display or speakers via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.

The network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.

In a computing device 200 according to the invention, the application 222 comprises instructions for performing the

method

300 or 400 of the invention, which may instruct the processor 204 to perform the

method

300 or 400 of the invention.

FIG. 3 shows a flow diagram of a data synchronization method 300, according to one embodiment of the invention, suitable for execution in the data synchronization system shown in FIG. 1.

The method saves the updated data to the database 130 through the web server 120 and transmits a data update message to the message queue server 140 in step S310. Notably, only a unique Identification (ID) of the update data is included in the data update message, and the web server 120 assigns an ID to each piece of data published by the user.

For example, a user at a home website of an automobile posts a new word-of-mouth (e.g., a comment to audi A6L) on the user interaction interface of the client 110 and triggers the save data interface, and the web server 120 assigns an ID to the new word-of-mouth posted by the user, saves the new word-of-mouth to the database 130, and sends a data update message to the message queue server 140 in response to the user triggering the save data interface. If the user modifies the published public praise, for example, adds a comment or modifies the published comment, and triggers the save data interface on the user interaction interface of the client 110, at this time, the web server 120 responds to the user's trigger operation on the save data interface, saves the added comment or modified comment to the database 130 based on the original public praise ID, and sends a data update message to the message queue server 140. If the user does not perform any action on the window of the client 110 but the save data interface is also triggered, the web server 120 replaces the original word-of-mouth with the current word-of-mouth in the database based on the original word-of-mouth ID in response to the user's trigger operation on the save data interface, and sends a data update message to the message queue server 140.

In the process of sending the data update message to the message queue server 140 by the web server 120, a failure in sending the data update message may occur due to a network flash, and the web server 120 may preset a sleep time and retry times in order to ensure that the data update message is sent to the message queue server 140. In an embodiment of the present invention, in order to set the sleep time by using a read () function of java to the server 120, the sleep time is generally set to about 200ms, the retry number may be set to 10, within a preset sleep time (200ms), the network server 120 determines whether the data update message is sent to the message queue server 140, if not sent to the message queue server 140, determines whether the retry number is less than a preset retry number 10, if the retry number is less than the preset retry number 10, resends the data update message to the message queue server 140 through a fault tolerance mechanism, the retry number increases by 1, and if the retry number is greater than the preset retry number 10, stops sending the data update message to the message queue server 140.

The above-mentioned fault-tolerant mechanism is a try catch mechanism, and it is determined whether the retry number is less than the preset retry number by using "if the retry number is less than the preset retry number, the data update message is retransmitted to the message queue server through the fault-tolerant mechanism, the retry number is increased by 1", and the retry mechanism is implemented based on the while loop and the set retry number (10), so that when a network flash occurs, the network server 120 is guaranteed to transmit the data update message to the message queue server 140 through the fault-tolerant mechanism and the retry mechanism.

Next, in step S320, the message queue server 140 receives the data update message sent by the web server 120 and saves the data update message to the message queue.

Then, in step S330, the consumer server 150 subscribes to the message queue of the message queue server 140, consumes the data update message in the message queue, searches the database 130 for data corresponding to the ID, and selects a part of the data corresponding to the ID from the searched data to be synchronized with the cache 160.

Specifically, the consumer server 150 subscribes to the message queue of the message queue server 140, continuously polls the message queue, and when there is a new data update message, pulls the data update message from the message queue, acquires a data ID from the data update message, searches for data corresponding to the ID from the database 130, and selects a part of data and service state data from the data corresponding to the ID. Wherein the business status data comprises a pass or no pass status of the operator audit user public praise flag as described above.

The selected part of data is selected from each piece of data stored in the database 130 and stored in the cache 160, for example, a public praise published by the user in the home website of the automobile is a piece of data, each public praise corresponds to an ID, each public praise includes evaluation contents of fields such as addition, most satisfaction, least satisfaction, space, power, control, oil consumption, comfort, appearance, interior, cost performance, and the like, and part of data is selected from data corresponding to the ID, that is, the comment contents of part of fields are selected from a public praise, for example, comment contents of three fields such as addition, most satisfaction, and least satisfaction are selected from a public praise.

In the invention, the data updating message is sent to the message queue server 140 through the network server 120, the message queue server 140 stores the message to the message queue, the consumer server 150 consumes the data updating message in the message queue and synchronously updates the data to the cache 160 based on the data updating message, thereby realizing the asynchronous operation of data updating and data caching, enabling the data in the cache 160 to be updated in real time, namely, the changed data in the database 130 can be synchronized to the cache 160 in real time, and also effectively avoiding the normal operation of data updating influenced by the blocking of the cache 160.

In an embodiment of the present invention, the consumer server 150 uses the ID as a key (key) of a key-value pair in the ordered set, and uses the selected partial data as a key (value) of a key-value pair in the ordered set, and since the selected partial data may occupy more memory, the value of the key-value pair in the cache occupies too much memory to affect the efficiency of reading data from the cache 160, for example, when the value exceeds 10KB, the performance of the cache 160 is affected. In order to subsequently read data from the cache efficiently, the consumer server 150 splits value into a plurality of pieces, the key of each split key value is the same and is an ID, and records the split number. That is to say, part of data selected under one ID corresponds to multiple key-value pairs, the key of each key-value pair is the ID, the value of each key-value pair is a part of the selected data, and the sum of the values of the multiple key-value pairs is all the selected data.

It is noted that there is timeliness in updating data, which means that the order of sending messages is the same as the order of receiving messages, for example: the program sends the message a first and then sends the message B, the message receiver should receive the message a first and then the message B, and if the message receiver receives the message B first and then the message a, the receiving order of the messages is disordered.

For example, if the public praise ID issued by the user is 00000018, the consumer server obtains the state data of 00000018 from the database 130 according to the ID, and if the service state data passes, selects the contents of the three fields of addition, most satisfaction and least satisfaction from the evaluation contents of the fields of addition, most satisfaction, least satisfaction, space, power, control, oil consumption, comfort, appearance, interior, cost performance and the like of the public praise with the ID of 00000018, and divides the contents of the fields of addition, most satisfaction and least satisfaction into three parts, which are the contents of the field of addition, most satisfaction and least satisfaction respectively, and divides the selected data into three key values to combine into three key value pairs with 00000018 as the key of the key value pair, which are: (00000018: additional field data), (00000018: most satisfied field data), and (00000018: least satisfied field data), record the value split number as 3, and save the 3 key value pairs corresponding to the selected partial data into one or more ordered sets of the cache 160. If the service status data corresponding to 00000018 is failed, the selected part of data will not be saved in the buffer 160.

Since a data synchronization failure may occur due to a network flash during the process of synchronizing the data to the cache 160, in order to prevent the data synchronization failure, the consumer server 150 determines whether to synchronize the selected partial data to the cache 160, and if not, re-synchronizes the selected partial data that has failed to be synchronized to the cache 160 by using a fault-tolerant mechanism and a retry mechanism. In one embodiment of the present invention, the fault tolerant mechanism is a try catch mechanism, and if the retry number is smaller than the preset retry number, the retry number is judged to be smaller than the preset retry number, and if the retry number is smaller than the preset retry number, the data update message is re-sent to the message queue server through the fault tolerant mechanism, and the retry number is increased by 1 ", so as to implement the retry mechanism, and based on the while loop and the set retry number (10), thereby when the network flash occurs, the consumer server 150 is ensured to synchronize the selected data to the cache 160 through the fault tolerant mechanism and the retry mechanism.

To address this problem, in the scenario where a data update, data cache asynchronous mechanism is used, the consumer server 150 may also check whether a portion of the data synchronized to the cache 160 is consistent with a corresponding portion of the updated data in the database 130, specifically: if the database 130 is a relational database, the consumer server 150 periodically scans the data table of the database 130, determines the updated data by searching the updated data of the data table within a preset time period, compares a part of the data synchronized to the cache 160 with a corresponding part of the updated data in the data table, if the part of the data is not consistent with the corresponding part of the selected part of the data, synchronizes the part of the updated data with the corresponding part of the selected part of the data to the cache 160 again, after the pushing is completed, pulls the data from the cache again for comparison, and if the part of the data is not consistent with the corresponding part of the updated data, a program error may occur, and a human intervention is notified.

Fig. 4 shows a flow diagram of a data acquisition method 400, suitable for execution in the web server 120, according to one embodiment of the invention.

It should be noted that the data retrieval method 400 retrieves data from the cache 160 of the data synchronization method 300.

In step S410, the HTTP request sent by the client 110 is received, and the HTTP request is parsed to obtain a data Identifier (ID) of the data requested by the client 110 to obtain.

The user triggers a data acquisition interface on a user interaction interface of the client terminal 110, sends an HTTP request to the web server 120, where the HTTP request includes an ID of the acquired data, and the web server 120 receives and analyzes the HTTP request sent by the client terminal 110, thereby obtaining the ID of the acquired data.

Next, in step S420, the ordered set of the cache where the client requests to acquire data is located and the number of key value splits of the key value pair of the request to acquire data are determined according to the ID.

It should be noted that the cached data is obtained by synchronizing the data to the cache according to the above-mentioned data synchronization method 300, so that a part of the data corresponding to each ID stored in the cache 160 is stored, and a part of the data selected under one ID in the cache 160 is split into a plurality of key-value pairs, a key of each key-value pair is an ID, and a value of each key-value pair is a part of the selected data, that is, there may be a plurality of key-value pairs corresponding to IDs in the cache.

Therefore, the network server 120 first determines the location of the data requested to be obtained in the cache, that is, in which ordered set of the cache, searches for the number of key value pairs corresponding to the ID, and reads the key value of the key value pair corresponding to the data requested to be obtained by the client 110 from the cache 160 in a multi-thread concurrent manner, thereby obtaining the data requested to be obtained by the client 110.

Finally, in step S420, the key value of the read key value pair is sent to the client through an HTTP response and displayed. As shown in fig. 5, fig. 5 is a schematic diagram illustrating a public praise displayed by the client 110 of the home website of the automobile, and since only part of the public praise data published by the user is cached in the cache 160, it can be seen in fig. 5 that the client 110 only displays the contents of the three fields of the additional, most satisfactory and least satisfactory of the public praise, corresponding to the cached data. If the user wants to view all the contents of the word-of-mouth, the user needs to trigger a "view all the contents" interface, and send an HTTP request to the web server 120, where the HTTP request includes a word-of-mouth ID for requesting to obtain all the contents, the web server 120 receives and parses the HTTP request, thereby determining the ID corresponding to the word-of-mouth for requesting to obtain all the contents, searches all the data corresponding to the ID from the database 130, and sends the data to the client 110 through an HTTP response, and displays all the contents of the word-of-mouth on the user interaction interface of the client 110.

In the invention, the key value of the key value pair corresponding to the data requested to be obtained by the client 110 is read from the cache 160 in a multithread concurrent manner, and the data is obtained from the cache 160 in the multithread concurrent manner, so that the reading efficiency of the cache 160 can be ensured under the condition that the value occupies a large memory. And the data read from the cache is displayed on the user interface, and part of public praise content is read from the cache, so that a plurality of public praise can be displayed on one page of the display interface of the client, and the user can view the plurality of public praise at one time through the user interface.

The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.

In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the data synchronization method and the data acquisition method of the present invention according to instructions in the program code stored in the memory.

By way of example, and not limitation, readable media includes readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.

In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment, or alternatively may be located in one or more devices different from the device in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore, may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the means for performing the functions performed by the elements for the purpose of carrying out the invention.

As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense with respect to the scope of the invention, as defined in the appended claims.

Claims

1. A data synchronization system, the system comprising: the system comprises a network server, a message queue server, a consumer server and a cache;

the network server is suitable for responding to the condition that a user triggers an update data interface and a save data interface at a client, saving the updated data to a database and sending a data update message to the message queue server, wherein the data update message only comprises a unique Identification (ID) of the updated data;

2. The system of claim 1, wherein the system further comprises a client adapted for user-triggered updating of the data interface and saving of the data interface.

3. The system of claim 2, wherein the network server is further adapted to:

presetting dormancy time and retry times;

if the data updating message is not sent to the message queue server, judging whether the retry time is less than the preset retry time;

4. The system of claim 1, wherein the consumer server is further adapted to:

searching data corresponding to the ID from the database;

5. The system of claim 4, wherein the consumer server is further adapted to:

6. The system of claim 5, wherein the cache includes one or more of the ordered sets.

7. The system of claim 6, wherein the client further comprises a data acquisition interface, the consumer server further adapted to:

8. The system of any one of claims 1 to 7, wherein the consumer server is further adapted to:

9. The system of any one of claims 1-8, wherein the consumer server is further adapted to:

10. The system of claim 9, wherein the consumer server is further adapted to:

periodically scanning a data table of the database;

11. A data synchronization method adapted to operate in a data synchronization system as claimed in any one of claims 1 to 10, the method comprising:

the network server responds to a user to trigger an update data interface and a save data interface at a client, saves the updated data to a database, and sends a data update message to a message queue server, wherein the data update message only comprises a unique Identification (ID) of the update data;

12. The method of claim 11, further comprising the step of:

the network server presets dormancy time and retry times;

if the retry times are less than the preset retry times, the network server resends the data updating message to the message queue server through a fault-tolerant mechanism and a retry mechanism, the retry times are increased by 1, and if the retry times are not less than the preset retry times, the data updating message is stopped being sent to the message queue server.

13. The method of claim 11, wherein the consumer server searches the database for data corresponding to the ID, and wherein selecting a portion of the data from the searched data corresponding to the ID to be synchronized to the cache comprises:

the consumer server searching the database for data corresponding to the ID;

14. The method of claim 13, wherein the consumer server synchronizing the selected portion of data to the cache based on the traffic state data comprises:

and synchronizing the selected part of data to the key value pairs of the cached ordered set according to the service state data by taking the ID as a key of the key value pair in the ordered set and the selected part of data as the key values of the key value pair in the ordered set.

15. The method of claim 14, wherein the cache includes one or more of the ordered sets.

16. The method of claim 15, wherein the client further comprises a data acquisition interface, the method further comprising:

and the consumer server splits the key values of the key value pairs in the ordered set and records the split number, so that when a user triggers the data acquisition interface at the client, the key values of the key value pairs corresponding to the data ID acquired by the interface request are acquired in a multi-thread concurrent mode and returned to the client.

17. The method of any of claims 11 to 16, further comprising the step of:

18. The method of any one of claims 11-17, further comprising the step of:

19. The method of claim 18, wherein the consumer server verifying that the portion of data synchronized to the cache is consistent with the corresponding portion of updated data in the database comprises:

periodically scanning a data table of the database;

20. A data acquisition method, performed in a network server, the method comprising:

determining an ordered set of a cache where the client requests to obtain data and a key value split number of a key value pair requesting to obtain data according to the ID, where the cached data is synchronized to the cache according to the method according to any one of claims 11 to 19;

and sending the key value of the read key value pair to the client through an HTTP response.

21. A computing device, comprising:

at least one processor; and

a memory having stored thereon program instructions configured to be executed by the at least one processor, the program instructions comprising instructions for performing the method of any of claims 11-20.

22. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method of any of claims 11-20.