WO2021142965A1

WO2021142965A1 - Data synchronization method and apparatus, and computer device and storage medium

Info

Publication number: WO2021142965A1
Application number: PCT/CN2020/085566
Authority: WO
Inventors: 丁晶晶
Original assignee: 深圳壹账通智能科技有限公司
Priority date: 2020-01-14
Filing date: 2020-04-20
Publication date: 2021-07-22
Also published as: CN111262923B; CN111262923A

Abstract

A data synchronization method and apparatus, and a computer device and a storage medium. The method comprises: monitoring a client cluster in real time, the client cluster comprising N clients, and N being a natural number greater than 1 (S10); if detecting that any client in the cluster of the N clients sends a data request, determining the client which sends a data request firstly to be a host, and determining the remaining N-1 clients to be slaves (S20); establishing an index for data corresponding to the data request sent by the host, and constructing a blocking queue having a length of N-1 on the basis of data corresponding to data requests sent by the slaves (S30); intercepting, in a network proxy manner, an HTTP packet in the data request sent by the host, and checking whether a header message contained in the HTTP packet exists in the index (S40); if the header message contained in the HTTP packet does not exist in the index, filling response data returned by the host into a deadlock queue corresponding to the blocking queue, such that the data of the slaves is synchronized with the data of the host (S50). Therefore, the data synchronization efficiency is improved.

Description

Data synchronization method, device, computer equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on January 14, 2020, the application number is 202010037664.0, and the invention title is "data synchronization method, device, computer equipment and storage medium", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of data processing, and in particular to a data synchronization method, device, computer equipment, and storage medium.

Background technique

In software testing, it is necessary to test the compatibility of a software in different operating system versions, different models of equipment, etc. In order to prevent compatibility differences due to differences in models, software testers need to install the software on different models and operating system versions. The traversal compatibility test is performed on the above. In order to reduce the test time, a synchronization operation is required. It is necessary to synchronize the data received by a host from the server to the slave and cut off the outgoing of the slave. However, it is subject to the network of different devices during the test. The real request will be sent to the server side, causing data disorder and unsynchronized operation caused by data dependence. All server data must be synchronized in real time, that is, for n test machines, the http data of 1 host will be synchronized to n- in real time. One host, at the same time, only one service request can be sensed on the server side, which can realize the operation and control of the measured object, and simultaneously record the intermediate data and power consumption characteristics in the operation process to develop Further analysis, and transfer to the PC side for storage, processing and analysis.

Traditionally, the data in the master-slave mode is usually synchronized through network mirroring, that is, the port of the master is mirrored to the slave. The inventor realized that because the slave cannot selectively keep synchronized with the master, all masters must be accepted. Synchronize the data poured into the mirror. Therefore, there is a need for a new data synchronization technology that can maintain data synchronization between clients in the same system.

Summary of the invention

The embodiments of the present application provide a data synchronization method, device, computer equipment, and storage medium to solve the problem of low data synchronization efficiency.

A data synchronization method includes:

Real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

If it is detected that any client in the cluster of N clients sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as the slaves;

Indexing the data corresponding to the data request sent by the master, and constructing a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

Intercepting the HTTP message in the data request sent by the host by means of a network proxy, and verifying whether the header message contained in the HTTP message exists in the index;

If the header message contained in the HTTP message does not exist in the index, the response data returned by the host is filled into the deadlock queue corresponding to the blocking queue, so that the data of the slave is consistent with all The data synchronization of the host.

A data synchronization device includes:

The cluster monitoring module is used for real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

The master-slave determination module is used to determine if any client in the cluster of N clients sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined As a slave

The queue establishment module is used to index the data corresponding to the data request sent by the master, and construct a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

An index check module, configured to intercept the HTTP message in the data request sent by the host by means of a network proxy, and check whether the header message contained in the HTTP message exists in the index;

The data synchronization module is configured to, if the header message contained in the HTTP message does not exist in the index, fill the response data returned by the host into the deadlock queue corresponding to the blocking queue, so that the The data of the slave is synchronized with the data of the master.

A computer device includes a memory, a processor, and a computer program that is stored in the memory and can run on the processor, and the processor implements the above data synchronization method when the computer program is executed.

A computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above data synchronization method is realized.

In the above-mentioned data synchronization method, device, computer equipment and storage medium, firstly, the client cluster is monitored in real time, and the client cluster includes N clients, so as to subsequently determine the master-slave role of each client in the client cluster; then, if it is detected To send a data request to any client in a cluster of N clients, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as slaves, which is convenient for the client cluster Maintenance also solves the problem of the inability to perceive the host caused by the inability to detect the host label in the client cluster; then, the data corresponding to the data request sent by the host is indexed, and the data construction length corresponding to the data request sent by the slave is N -1 blocking queue, so as to ensure that the queue with the longest waiting time can access the queue first, which is beneficial to improve the efficiency of obtaining data from the slave; next, use the network proxy to intercept the HTTP in the data request sent by the host Message, check whether the header message contained in the HTTP message exists in the index, to ensure that 100% of the data sent from the slave machine is truncated, and the slave machine can selectively receive the synchronization data of the master according to the index; finally, if the HTTP message If the header message contained in the text does not exist in the index, the response data returned by the master is filled into the deadlock queue corresponding to the blocking queue, so that the data of the slave is synchronized with the data of the master, reducing system overhead, improving system performance, and then Improve the efficiency of data synchronization.

Description of the drawings

FIG. 1 is a schematic diagram of an application environment of a data synchronization method provided by an embodiment of the present application;

FIG. 2 is an example diagram of a data synchronization method provided by an embodiment of the present application;

FIG. 3 is another example diagram of a data synchronization method provided by an embodiment of the present application;

FIG. 4 is another example diagram of a data synchronization method provided by an embodiment of the present application;

FIG. 5 is another example diagram of a data synchronization method provided by an embodiment of the present application;

Fig. 6 is a functional block diagram of a data synchronization device provided by an embodiment of the present application;

FIG. 7 is another functional block diagram of a data synchronization device provided by an embodiment of the present application;

FIG. 8 is another functional block diagram of a data synchronization device provided by an embodiment of the present application;

Fig. 9 is a schematic diagram of a computer device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The data synchronization method provided by this application can be applied to the application environment as shown in Figure 1, where the client communicates with the server through the network, and the server monitors the client cluster in real time. The client cluster includes N clients, when When any client in the cluster of N clients is detected to send a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as slaves; then, the master is The data corresponding to the sent data request is indexed, and a blocking queue of length N-1 is constructed based on the data corresponding to the data request sent by the slave; then, the HTTP message in the data request sent by the host is intercepted by a network proxy. Check whether the header message contained in the HTTP message exists in the index; then when the header message included in the HTTP message does not exist in the index, the response data returned by the host is filled into the deadlock queue corresponding to the blocking queue, so that The data of the slave is synchronized with the data of the master. Among them, the client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers.

In an embodiment, as shown in FIG. 2, the method is applied to the server in FIG. 1 as an example for description, including the following steps:

S10: Real-time monitoring of the client cluster. The client cluster includes N clients, where N is a natural number greater than 1.

Specifically, the server monitors the client cluster in real time. The client cluster is composed of N clients. Taking software testing as an example, for a cluster composed of N test machines, that is, N clients, in order to prevent model differences (Differences between different clients) lead to differences in compatibility. The server-side monitoring tool can monitor the status of the client cluster in real time, such as the number of connected clients, the number of message requests, etc., by monitoring the status of the client cluster, so as to subsequently determine each client cluster. The master-slave role of each client.

S20: If it is detected that any client in the cluster of N clients sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as the slaves.

Specifically, the server performs real-time detection on a client cluster composed of N clients, and determines 1 master and N-1 slaves according to the detection results, where N is a natural number greater than 1.

When synchronizing a client cluster containing N clients, due to the performance differences of each client, there is a delay when each client sends a request to the server, and the server will intercept the earliest request , It is determined that it is the first client that sends a request to the server, that is, the master, and the remaining N-1 clients in the client cluster are determined as slaves. Understandably, using the first client to make the request as the host, instead of forcing the host to facilitate the maintenance of the client cluster, also solves the problem of the inability to perceive the host caused by the inability to detect the host label in the client cluster. Furthermore, it can also prevent the problem of slave waiting caused by the speed of the slave being better than the master when the master is forcibly designated, preventing the host from going down when the master is forcibly designated, and overcoming the failure of the entire cluster. The problem with the host.

S30: Establish an index for the data corresponding to the data request sent by the host, and construct a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave.

Among them, the index (Index) is a sorted list, a data structure used for the database to efficiently obtain data. The list of this data structure stores the value of the index and the physical address of the row where the data containing this value is located. When the data is very large, the index can greatly speed up the query. Therefore, you do not need to scan the entire table after using the index. To locate a row of data, first find the physical address corresponding to the row of data through the index table, and then access the corresponding data. Understandably, the master in step S20 is the client corresponding to the request for indexing that a request sent by a client does not exist, and the request from the slave machine is the client corresponding to the request for indexing for N-1 clients.

Among them, the blocking queue refers to a chain data structure with blocking function. When the queue is empty, the read queue request is blocked, and it can be taken out when the queue is not empty; when the queue is full, the input queue is blocked and the queue needs to be waited. It can be input (inserted) when it is not full, which is used to increase the storage space of the returned data, and it can also ensure that there is only one thread operating the queue at the same time.

Specifically, the data sending and receiving of the first host that sent the HTTP message is processed: the sent data is indexed first, and the received data is returned after the received data is returned to establish a fixed-length (equivalent to the number of slaves) blocking queue. Use the create index method in SQL to index the data requested by the host, and create an ArrayBlockingQueue object based on the array based on the data in the data request sent by the slave, and the capacity of the object is N-1, that is, a length of The blocking queue of N-1 can ensure that the queue with the longest waiting time can access the queue first, which is beneficial to improve the data acquisition efficiency of the slave.

S40: Intercept the HTTP message in the data request sent by the host by means of a network proxy, and check whether the header message contained in the HTTP message is in the index.

Among them, the HTTP message refers to all data of the HTTP request protocol, including the data request sent by the client requesting the server and the response data returned by the server. Network proxy, that is, client C needs to access server S. Through proxy, client C does not directly access server S. Client C needs to access intermediate proxy server P first. After intermediate proxy server P receives client C’s request , The data request is sent to server S, and the response from server S is obtained at the same time, and then returned to client C. Among them, the header message refers to the header of the HTTP message, that is, the message header, which is composed of key/value key-value pairs, one pair per line, and the key and value are separated by a colon ":". The request header informs the server about The request information on the client side is the information returned by the server according to the request. It can be obtained using the HttpServletRequest method. The obtained header message is compared with the index established in step S30 to verify whether the header message contained in the HTTP message is It is stored in the index so that the data of the master and the data of the slave can be further processed according to the verification result.

Understandably, intercepting the HTTP message in the request of the host to send data through a network proxy can hijack the data packet sent by the client to the server. At the same time, in a client cluster of N clients, the host's sending and response are hijacked After that, the response of the remaining N-1 (slave) clients can be directly hijacked to synchronize the response of the master, so that the subsequent synchronization effect of the master and the slave can be achieved based on the requested content as an index. Therefore, the network proxy method allows the host and the slave to directly have a time delay, and can control the synchronization information of the slave. Further, after the request from the slave reaches the proxy service, because the index header can be queried through the index header, the request information is directly thrown away, not sent to the server, and the returned content is directly taken from the blocking queue and returned to each slave. , To ensure that 100% of the data sent from the slave is truncated, and the slave can selectively receive the synchronization data of the master according to the index.

Preferably, a theoretical timeout value can also be set for the blocking queue, where the time can be 5 seconds (S).

S50: If the header message contained in the HTTP message does not exist in the index, fill the response data returned by the host into the deadlock queue corresponding to the blocking queue, so that the data of the slave and the data of the host are synchronized.

Specifically, if the header message contained in the HTTP message does not exist in the index, the data of the client cluster is empty, so the corresponding data returned by the host is copied for N-1 copies and filled into the deadlock queue corresponding to the blocking queue , Using the proxy method, intercept the request sent from the slave machine and throw it away directly, look up the corresponding queue address from the index established by the master, and take out the data from the queue and send it to the slave machine. Send the data in the queue to the slave and find the corresponding information in the index. If the information is obtained in the index when the request passes through the proxy server, a deadlock queue is returned, and the deadlock queue is not empty, then the deadlock The queue takes out an element and returns it to the slave to realize the synchronization of the data of the master and the data of the slave. Further, in this embodiment, the deadlock queue solves the problem of the time difference in request return. When the host's request arrives at the server, the queue will not be filled before the server returns, and then the deadlock from the opportunity is waiting for the host to send The return is responded, and interrupts are used in the deadlock queue mode, and the CPU hardware is used to process data, which reduces system overhead and improves system performance, thereby increasing the efficiency of data synchronization.

It should be noted that in this embodiment, the HTTP message is added to the preset index, so that which client the host is can be determined, and the accuracy of the host can be further ensured and the data of the slave can be obtained 100%.

In this embodiment, first, the client cluster is monitored in real time, and the client cluster includes N clients, so that the master-slave role of each client in the client cluster can be determined subsequently; then, if any of the N client clusters is detected When a client sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as slaves, which facilitates the maintenance of the client cluster and also solves the problem of the client cluster. Unable to detect the problem of the inability to perceive the host caused by the host tag; then, index the data corresponding to the data request sent by the host, and build a blocking queue of length N-1 based on the data corresponding to the data request sent by the slave, so that it can be subsequently It is ensured that the queue with the longest waiting time can access the queue first, which is beneficial to improve the efficiency of obtaining data from the slave; then, the HTTP message in the data request sent by the host is intercepted by the network proxy method, and the HTTP message is checked Whether the included header message exists in the index, to ensure that 100% of the data sent from the slave machine is truncated, and the slave machine can selectively receive the synchronization data of the host according to the index; finally, if the header message included in the HTTP message does not have an index In the middle, the response data returned by the master is filled into the deadlock queue corresponding to the blocking queue, so that the data of the slave is synchronized with the data of the master, reducing system overhead, improving system performance, and improving data synchronization efficiency.

In one embodiment, as shown in FIG. 3, in step S40, checking whether the header message contained in the HTTP message exists in the index, specifically includes the following steps:

S41: Extract the URL and interface parameters contained in the HTTP message.

Among them, URL (Uniform Resource Locator) is a uniform resource locator, a concise representation of the location and access method of resources available on the Internet, and is the address of a standard resource on the Internet, such as http: //abcom/ queryData.do is the URL contained in the HTTP message. The interface parameters refer to the parameters of the HTTP message request line, such as a_key=a_data. Specifically, the URL contained in the HTTP message can be extracted through request-URL, and the interface parameters can be extracted through the getParameters() method.

S42: Combine the URL and the interface parameters according to a preset combination manner to obtain a query index.

Among them, the preset combination mode refers to a preset information connection combination rule, and the combination mode in this embodiment is a sequential connection, that is, the "information A+information B" mode. Query index refers to key information used for query or search. Exemplarily, when the network proxy server intercepts a request, for example, the address of an http request is: http://a.b.com/queryData.do The parameter is a_key=a_data; b_key=b_data, then the query index is:

http: //a.b.com/queryData.do? a_key=a_data&b_key=b_data.

Understandably, the uniqueness of the query index can be ensured by combining the URL and interface parameters according to a preset combination as the query index.

S43: Determine whether the header message included in the HTTP message exists in the index based on the query index.

Specifically, take the query index in step S42 as an example, that is, http://a.b.com/queryData.do? a_key=a_data&b_key=b_data is used as the query index to determine whether the header message contained in the HTTP message exists in the index. The index method improves the query efficiency and speeds up the verification.

It should be noted that since the index is a hashmap, the key is the index header, and the data value is the deadlock queue. If the query index does not exist in the index, then the query index http://a.b.com/queryData.do? a_key=a_data&b_key=b_data is added to the hashmap, and the value corresponding to the index header is a newly created empty deadlock queue to ensure the accuracy and integrity of the index.

In this embodiment, first, the URL and interface parameters contained in the HTTP message are extracted; then, the URL and interface parameters are combined according to a preset combination to obtain a query index to ensure the uniqueness of the query index; finally, based on the query index It is judged whether the header message contained in the HTTP message exists in the index, and the query efficiency is improved by the index method, and the speed of the verification is accelerated.

In an embodiment, as shown in FIG. 4, after step S40, that is, after checking whether the header message contained in the HTTP message exists in the index, the data synchronization method further includes:

S60: If the header message included in the HTTP message is stored in the index, then the data is retrieved from the index.

Specifically, the network proxy server intercepts a request, for example, the address of an http request is: http://abcom/insertData. The parameter of do is c_key=c_data; d_key=d_data, then the header message contained in the HTTP message is http : //Abcom/insertData.do? c_key=c_data&d_key=d_data, query whether the request is in the index, when the header message exists in the index, the index header exists, that is, it is determined that the client corresponding to the request is a slave. Since the index has been hit, the index header is used from the index If the returned data is retrieved from the index, the index header is used to retrieve the returned data from the index. If the queue is not blocked at this time, the data can be retrieved. If the queue is blocked at this time, the execution will continue after waiting for the queue to be filled, and the data will be directly returned to the slave , That is, take out the value part in the index hashmap in step S43, that is, the data in this embodiment. Further, if the header message contained in the HTTP message is stored in the index, the server will discard the request from the slave to realize the interception of the data of the slave.

S70: Return the data to the slave, so that the data of the slave is synchronized with the data of the master.

Specifically, after intercepting the data from the slave, the server directly returns the data retrieved from the index to the slave, which accurately and quickly realizes the synchronization of the data of the master and the data of the slave.

In this embodiment, if the header message contained in the HTTP message exists in the index, the data is taken out from the index to realize the interception of the data of the slave; the data is returned to the slave, so that the data of the slave and the master The data synchronization can accurately and quickly realize the synchronization of the data of the master and the data of the slave.

In an embodiment, after step S50, that is, after synchronizing the data of the slave with the data of the master, the data synchronization method further includes:

Release data from the deadlock queue to cut off the data outgoing from the slave.

Specifically, the deadlock queue is released to cut off the data outgoing from the slave. The fixed length is N-1 and the blocking queue is released immediately after being fetched, or the blocking queue is released immediately after the timeout, so as to prevent the same interface from the second time The request for data access to the first cache cuts off the outgoing of data from the slave, and further improves the efficiency of synchronization between the data of the slave and the data of the master.

In this embodiment, data is released on the deadlock queue to cut off data outgoing from the slave, which further improves the efficiency of synchronization between the data of the slave and the data of the master.

In one embodiment, as shown in FIG. 5, releasing data from the deadlock queue to cut off data outgoing from the slave machine specifically includes the following steps:

S81: If the blocking queue data is read N-1 times, delete the blocking queue.

Specifically, take the N-1 times of the blocking queue and release it actively. When the queue is fetched, the counter will increase by 1. When the N-1th time is fetched, the blocking queue will be deleted from the index hashmap first, and then the blocking queue will be released. , Thereby cutting off the data from the slave machine, preventing the cache of the slave machine from being accessed, and improving the accuracy of data synchronization.

S82: If it is detected that the time that the blocking queue is filled is greater than the preset timeout threshold, delete the blocking queue.

Specifically, there is a thread globally, and the existence time of the blocking queue is monitored from the thread loop. If it is monitored that a certain blocking queue is filled and exceeds a preset timeout threshold, the preset timeout threshold can be set to 5S (set in step S40 The timeout theoretical value), the blocking queue is deleted and released, so that the data from the slave is no longer sent to the server, but directly fetched from the blocking queue cached by the slave, which ensures the accuracy and efficiency of data synchronization.

In this embodiment, if the blocking queue data is read N-1 times, the blocking queue is deleted, thereby intercepting the outgoing data of the slave machine, preventing the cache of the slave machine from being accessed, and improving the accuracy of data synchronization; If it is detected that the time that the blocking queue is filled is greater than the preset timeout threshold, the blocking queue is deleted so that the data from the slave is no longer sent to the server, but directly fetched from the blocking queue cached by the slave. Improve the accuracy and efficiency of data synchronization.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

In one embodiment, a data synchronization device is provided, and the data synchronization device corresponds to the data synchronization method in the above-mentioned embodiment one-to-one. As shown in FIG. 6, the data synchronization device includes a cluster monitoring module 10, a master-slave machine determination module 20, a queue establishment module 30, an index verification module 40 and a data synchronization module 50. The detailed description of each functional module is as follows:

The cluster monitoring module 10 is used for real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

The master-slave determination module 20 is used to determine the client that sends the data request first as the master when any client in the cluster of N clients is detected to send a data request, and the remaining N-1 clients Determined as a slave;

The queue establishment module 30 is configured to index the data corresponding to the data request sent by the master, and construct a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

The index check module 40 is configured to intercept the HTTP message in the data request sent by the host by means of a network proxy, and check whether the header message contained in the HTTP message exists in the index;

The data synchronization module 50 is configured to fill the response data returned by the host into the deadlock queue corresponding to the blocking queue when the header message contained in the HTTP message does not exist in the index, so that all The data of the slave is synchronized with the data of the master.

Preferably, as shown in FIG. 7, the index verification module 40 includes a message extraction unit 41, an index acquisition unit 42 and a verification unit 43.

The message extraction unit 41 is configured to extract the URL and interface parameters contained in the HTTP message;

The index obtaining unit 42 is configured to combine the URL and the interface parameters according to a preset combination manner to obtain a query index;

The checking unit 43 is configured to judge whether the header message contained in the HTTP message exists in the set index based on the query index.

Preferably, as shown in FIG. 8, the data synchronization device further includes a data acquisition module 60 and a data return module 70.

The data acquisition module 60 is configured to retrieve data from the index if the header message included in the HTTP message is stored in the index;

The data return module 70 is configured to return the data to the slave, so that the data of the slave is synchronized with the data of the host.

Preferably, the data synchronization device further includes a data release module, configured to release data from the deadlock queue to intercept data outgoing from the slave.

Preferably, the data release module includes a queue deletion unit and a blocked queue deletion unit.

A queue deletion unit, configured to delete the blocked queue if the blocked queue data is read N-1 times;

The blocking queue deleting unit is configured to delete the blocking queue if it is detected that the time for which the blocking queue is filled is greater than the preset timeout threshold.

For the specific limitation of the data synchronization device, please refer to the above limitation of the data synchronization method, which will not be repeated here. Each module in the above-mentioned data synchronization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 9. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used to store the data used in the data method. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to realize a data synchronization method.

In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor implements the data synchronization method in the foregoing embodiment when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided. The storage medium is a volatile storage medium or a non-volatile storage medium, and a computer program is stored thereon. When the computer program is executed by a processor, the foregoing The data synchronization method in the embodiment.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of protection of this application.

Summary of the invention

technical problem

The solution to the problem

The beneficial effects of the invention

Claims

A data synchronization method, the data synchronization method includes:

Real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

If it is detected that any client in the cluster of N clients sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as the slaves;

Indexing the data corresponding to the data request sent by the master, and constructing a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

Intercepting the HTTP message in the data request sent by the host by means of a network proxy, and verifying whether the header message contained in the HTTP message exists in the index;

If the header message contained in the HTTP message does not exist in the index, the response data returned by the host is filled into the deadlock queue corresponding to the blocking queue, so that the data of the slave is consistent with all The data synchronization of the host.
The data synchronization method according to claim 1, wherein the checking whether the header message contained in the HTTP message exists in the index comprises:

Extracting the URL and interface parameters contained in the HTTP message;

Combine the URL and the interface parameters according to a preset combination manner to obtain a query index;

Determine whether the header message included in the HTTP message exists in the index based on the query index.
5. The data synchronization method according to claim 1, after the checking whether the header message contained in the HTTP message exists in the index, the data synchronization method further comprises:

If the header message included in the HTTP message is stored in the index, then data is retrieved from the index;

The data is returned to the slave, so that the data of the slave is synchronized with the data of the host.
8. The data synchronization method according to claim 1, after said synchronizing the data of the slave machine with the data of the master, the data synchronization method further comprises:

Release data from the deadlock queue to cut off data outgoing from the slave.
8. The data synchronization method according to claim 4, said releasing data from the deadlock queue to cut off data outgoing from the slave machine, comprising:

If the blocking queue data is read N-1 times, delete the blocking queue;

If it is detected that the time that the blocking queue is filled is greater than the preset timeout threshold, the blocking queue is deleted.
A data synchronization device, the data synchronization device includes:

The cluster monitoring module is used for real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

The master-slave determination module is used to determine the client that sends the data request first as the master when any client in the cluster of N clients is detected to send a data request, and the remaining N-1 clients are determined As a slave

The queue establishment module is used to index the data corresponding to the data request sent by the master, and construct a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

An index check module, configured to intercept the HTTP message in the data request sent by the host by means of a network proxy, and check whether the header message contained in the HTTP message exists in the index;

The data synchronization module is used to fill the response data returned by the host into the deadlock queue corresponding to the blocking queue when the header message contained in the HTTP message does not exist in the index, so that the The data of the slave is synchronized with the data of the master.
7. The data synchronization device according to claim 6, wherein the index check module comprises:

The message extraction unit is used to extract the URL and interface parameters contained in the HTTP message;

An index obtaining unit, configured to combine the URL and the interface parameters according to a preset combination manner to obtain a query index;

The checking unit is configured to judge whether the header message contained in the HTTP message exists in the set index based on the query index.
7. The data synchronization device according to claim 6, the data synchronization device further comprising:

A data acquisition module, configured to retrieve data from the index when the header message contained in the HTTP message is stored in the index;

The data return module is used to return the data to the slave, so that the data of the slave is synchronized with the data of the host.
7. The data synchronization device according to claim 6, the data synchronization device further comprising:

The data release module is used to release data from the deadlock queue to cut off the data outgoing from the slave.
9. The data synchronization device according to claim 9, wherein the data release module comprises:

A queue deleting unit, configured to delete the blocking queue when the blocking queue data is read N-1 times;

The blocking queue deleting unit is configured to delete the blocking queue when it is monitored that the time for which the blocking queue is filled is greater than a preset timeout threshold.
A computer device includes a memory, a processor, and a computer program that is stored in the memory and can run on the processor, and when the processor executes the computer program, a data synchronization method is implemented:

Wherein, the data synchronization method includes:

Real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

If it is detected that any client in the cluster of N clients sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as the slaves;

Indexing the data corresponding to the data request sent by the master, and constructing a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

Intercepting the HTTP message in the data request sent by the host by means of a network proxy, and verifying whether the header message contained in the HTTP message exists in the index;

If the header message contained in the HTTP message does not exist in the index, the response data returned by the host is filled into the deadlock queue corresponding to the blocking queue, so that the data of the slave is consistent with all The data synchronization of the host.
11. The computer device according to claim 11, wherein the step of verifying whether the header message contained in the HTTP message exists in the index comprises:

Extracting the URL and interface parameters contained in the HTTP message;

Combine the URL and the interface parameters according to a preset combination manner to obtain a query index;

Determine whether the header message included in the HTTP message exists in the index based on the query index.
11. The computer device according to claim 11, after the step of verifying whether the header message contained in the HTTP message exists in the index, further comprising:

If the header message included in the HTTP message is stored in the index, then data is retrieved from the index;

The data is returned to the slave, so that the data of the slave is synchronized with the data of the host.
11. The computer device according to claim 11, after the step of synchronizing the data of the slave with the data of the master, further comprising:

Release data from the deadlock queue to cut off data outgoing from the slave.
The computer device according to claim 14, wherein the step of releasing data from the deadlock queue to intercept data outgoing from the slave machine comprises:

If the blocking queue data is read N-1 times, delete the blocking queue;

If it is detected that the time that the blocking queue is filled is greater than the preset timeout threshold, the blocking queue is deleted.
A computer-readable storage medium, the computer-readable storage medium stores a computer program, and a method for data synchronization when the computer program is executed by a processor:

Wherein, the data synchronization method includes the following steps:

Real-time monitoring of a client cluster, the client cluster includes N clients, and N is a natural number greater than 1;

If it is detected that any client in the cluster of N clients sends a data request, the client that sends the data request first is determined as the master, and the remaining N-1 clients are determined as the slaves;

Indexing the data corresponding to the data request sent by the master, and constructing a blocking queue with a length of N-1 based on the data corresponding to the data request sent by the slave;

Intercepting the HTTP message in the data request sent by the host by means of a network proxy, and verifying whether the header message contained in the HTTP message exists in the index;

If the header message contained in the HTTP message does not exist in the index, the response data returned by the host is filled into the deadlock queue corresponding to the blocking queue, so that the data of the slave is consistent with all The data synchronization of the host.
16. The computer-readable storage medium according to claim 16, wherein the step of verifying whether the header message contained in the HTTP message exists in the index comprises:

Extracting the URL and interface parameters contained in the HTTP message;

Combine the URL and the interface parameters according to a preset combination manner to obtain a query index;

Determine whether the header message included in the HTTP message exists in the index based on the query index.
16. The computer-readable storage medium according to claim 16, after the step of verifying whether the header message contained in the HTTP message exists in the index, further comprising:

If the header message included in the HTTP message is stored in the index, then data is retrieved from the index;

The data is returned to the slave, so that the data of the slave is synchronized with the data of the host.
15. The computer-readable storage medium according to claim 16, after the step of synchronizing the data of the slave with the data of the master, further comprising:

Release data from the deadlock queue to cut off data outgoing from the slave.
The computer-readable storage medium according to claim 19, wherein the step of releasing data from the deadlock queue to intercept data outgoing from the slave machine comprises:

If the blocking queue data is read N-1 times, the blocking queue is deleted; if it is detected that the time for the blocking queue to be filled is greater than the preset timeout threshold, the blocking queue is deleted.