WO2018049983A1 - 数据同步方法、系统、同步获取方法及装置 - Google Patents

数据同步方法、系统、同步获取方法及装置 Download PDF

Info

Publication number
WO2018049983A1
WO2018049983A1 PCT/CN2017/099687 CN2017099687W WO2018049983A1 WO 2018049983 A1 WO2018049983 A1 WO 2018049983A1 CN 2017099687 W CN2017099687 W CN 2017099687W WO 2018049983 A1 WO2018049983 A1 WO 2018049983A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
server
type
information
synchronization
Prior art date
Application number
PCT/CN2017/099687
Other languages
English (en)
French (fr)
Inventor
周龄
陈峥
严俊明
吴承
姜凤波
张力
陈方舟
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2018049983A1 publication Critical patent/WO2018049983A1/zh
Priority to US16/040,692 priority Critical patent/US10895996B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2097—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements maintaining the standby controller/processing unit updated
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023—Failover techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2038—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant with a single idle spare processing component
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2048—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant where the redundant components share neither address space nor persistent storage
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/81—Threshold
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/82—Solving problems relating to consistency

Definitions

  • the embodiments of the present invention relate to the field of communications technologies, and in particular, to a data synchronization method, a system, a synchronization acquisition method, and related devices.
  • the active and standby servers are deployed in the existing cloud storage system.
  • the standby server is used to store data uploaded by the user through the client. This requires ensuring that the corresponding data stored in the primary server and the standby server are the same, so that after the primary and secondary servers are switched, it is guaranteed that the data previously uploaded by the client can also be obtained from the standby server.
  • the embodiment of the invention provides a data synchronization method, a system, a synchronization acquisition method and a related device, which realize that asynchronously, data that is not larger than a preset value is asynchronously transmitted to a standby server for synchronization after processing the upload request.
  • the embodiment of the invention provides a data synchronization method, which is applied to a primary server, and the method includes:
  • a second type of work log is generated and stored, and the second type of work log includes: indication information for indicating data that is not synchronized to the standby server.
  • the embodiment of the invention further provides a data acquisition method, which is applied to an alternate server, and the method includes:
  • the third verification code is inconsistent with the fourth verification code, or the third verification code is empty and the fourth verification code is not empty, the information corresponding to the to-be-obtained data is obtained from the primary server. And responding to the client with data corresponding to the information of the data to be acquired.
  • the embodiment of the invention further provides a data synchronization device, comprising:
  • the uploading and receiving unit is configured to receive an uploading request sent by the client, where the uploading request includes data to be uploaded;
  • a request processing unit configured to store the acquired data in response to the upload request of the client after successfully acquiring the data
  • a synchronization unit configured to initiate a synchronization request to the standby server when the data is not greater than a preset value, to request synchronization of the data to the standby server, so that the standby server stores the data;
  • a first recording unit configured to generate and store a second type of work log when the size of the data is greater than the preset value, where the second type of work log includes: indication information, indicating that the data is not synchronized The data of the standby server.
  • the embodiment of the invention further provides a data synchronization acquiring device, which comprises:
  • the acquiring unit is configured to receive a data acquisition request sent by the client, where the data acquisition request includes information about the data to be acquired;
  • a verification unit configured to acquire a third verification code that stores data corresponding to the information of the data to be acquired, where the locally stored data is stored according to a synchronization request sent by the primary server; and the primary usage is obtained.
  • a fourth verification code of the data corresponding to the information of the data to be acquired stored by the server if the third verification code is inconsistent with the fourth verification code, or the third verification code is empty and the fourth verification code is Not empty, the notification verification processing unit responds to the customer order;
  • the verification processing unit is configured to acquire data corresponding to the information of the data to be acquired from the primary server, and respond to the client with data corresponding to the information of the data to be acquired.
  • An embodiment of the present invention further provides a data synchronization system, including a primary server and a backup server, where The primary server includes:
  • the uploading and receiving unit is configured to receive an uploading request sent by the client, where the uploading request includes data to be uploaded;
  • a request processing unit configured to store the acquired data in response to the upload request of the client after successfully acquiring the data
  • a synchronization unit configured to initiate a synchronization request to the standby server when the data is not greater than a preset value, to request synchronization of the data to the standby server, so that the standby server stores the data;
  • a first recording unit configured to generate and store a second type of work log when the size of the data is greater than the preset value, where the second type of work log includes: indication information, indicating that the data is not synchronized The data of the standby server;
  • the standby server is configured to receive a synchronization request for the data, and store the data.
  • the primary server receives the upload request sent by the client, and after processing the upload request, the data is successfully obtained, and immediately responds to the upload request of the client, and the size is asynchronously greater than the preset.
  • the value of the data is synchronized to the alternate server. Since the processing procedure of the primary server for uploading the request is asynchronous with the step of synchronizing with the data, the primary server does not need to wait for the primary server and the standby server to store the data, and then responds to the client, and implements Synchronization of data does not affect the uploading of business processes.
  • the primary server temporarily does not perform the synchronization operation, only records the second type of work log, and the second type of work log includes the information of the data that fails to synchronize with the standby server, so that the primary server Subsequent synchronization of the data can be completed according to the second type of work log, which can prevent the synchronization of the larger data from affecting the synchronization of other data.
  • FIG. 1 is a schematic structural diagram of a data synchronization system according to an embodiment of the present invention.
  • FIG. 2 is a flowchart of a data synchronization method according to Embodiment 1 of the present invention.
  • Embodiment 3 is a flowchart of a reconciliation process initiated by a primary server in Embodiment 1 of the method of the present invention
  • Embodiment 4 is a flowchart of a data synchronization acquisition method provided by Embodiment 2 of the method of the present invention.
  • FIG. 5 is a schematic structural diagram of a data synchronization system according to Embodiment 3 of the method of the present invention.
  • FIG. 6 is a flowchart of a data synchronization method according to Embodiment 3 of the method of the present invention.
  • FIG. 7 is a flowchart of a method for data consistency performed by a primary server in Embodiment 3 of the method of the present invention.
  • FIG. 8 is a flowchart of a method for data consistency performed by a standby server in Embodiment 3 of the method of the present invention.
  • FIG. 9 is a schematic structural diagram of a data synchronization apparatus according to Embodiment 1 of the apparatus of the present invention.
  • FIG. 10 is a schematic structural diagram of another data synchronization apparatus according to Embodiment 1 of the apparatus of the present invention.
  • FIG. 11 is a schematic structural diagram of a data synchronization acquiring apparatus according to Embodiment 2 of the device of the present invention.
  • FIG. 12 is a schematic structural diagram of another data synchronization acquiring apparatus according to Embodiment 2 of the apparatus of the present invention.
  • FIG. 13 is a schematic structural diagram of a server according to Embodiment 3 of the device of the present invention.
  • the primary server when data synchronization is performed on the primary and secondary servers, when the primary server receives the upload request from the client, the primary server sends the upload request to the public gateway interface on the primary server and the standby server (Common).
  • the Gateway Interface (CGI) module processes the upload request.
  • the CGI module of the primary server and the standby server respectively processes the upload request, the primary server responds to the upload request. If the primary and secondary servers are geographically and cross-operating, they will be built. It has a long response time and low reliability. Moreover, the controllability of this scheme is relatively low, and it is unable to cope with various emergencies.
  • the embodiments of the present invention provide the following technical solutions.
  • a data synchronization method is provided, which is mainly applied to a data synchronization system, such as a cloud storage system, as shown in FIG. 1, and includes a client 110, a primary server 120, and a standby server 130, wherein the client 110 can
  • the primary server 120 stores the data uploaded by the client by the primary server 120 and synchronizes the data to the standby server 130.
  • the client 110 and the backup server 130 do not communicate with each other, but communicate with the primary server 120.
  • the client 110 can communicate with the backup server 130. Whether the corresponding data stored by the server 120 and the standby server 130 are consistent.
  • the method in this embodiment is a method performed by a primary server in the system, and the flowchart is as shown in FIG. 2, including:
  • Step 101 Receive an upload request sent by the client, and include data to be uploaded in the upload request.
  • Step 102 After successfully obtaining the data, store the acquired data in response to the upload request of the client.
  • Step 103 If the size of the data stored in step 102 is not greater than the preset value, initiate a synchronization request for the data to the standby server to request synchronization of the data to the standby server, so that the standby server stores the data.
  • Step 104 If the size of the data is greater than a preset value, generating and storing a second type of work log, where the second type of work log includes: information for synchronizing the failed data to the standby server, that is, indication information, indicating that the data has not been synchronized. Data to the alternate server.
  • the indication information includes: identification information of the data, information indicating that the data synchronization fails and/or is not synchronized, information indicating the size of the data, and the like.
  • the primary server does not synchronize the data to the standby server according to the processing of the upload request, but first records the second type of work log. In addition, if you are to the same server When the step data fails, the primary server can also record the second type of work log. Failure to synchronize data with the standby server can mean that the primary server failed to initiate a synchronization request to the standby server, causing the data to not be successfully synchronized to the standby server.
  • the primary server may also record the first type of work log, and each of the first type of work logs may include: information of the successfully synchronized data. For example, the identification information of the data, the storage path, such as the name and data size of the bucket for storing the data, and the information such as the client identifier and the upload path.
  • an upload request corresponds to a first type of work log or a second type of work log.
  • the primary server receives the upload request sent by the client, and processes the upload request, that is, after successfully obtaining the data, the active server immediately responds to the upload request of the client, and asynchronously sets the size to be not greater than the pre-
  • the data of the set value is synchronized to the standby server. Since the response process of the primary server to the upload request is synchronized with the data of the primary server, that is, the synchronization to the standby server is asynchronous, the primary server does not need to wait for the primary server and the backup server to store the data. Respond to the client's upload request. Because of the quick response to the client's upload request, the data synchronization process does not affect the process of uploading the business.
  • the primary server temporarily does not perform the synchronization operation, but records the data that is not synchronized to the standby server in the second type of work log. In this way, the primary server can subsequently synchronize data that has not been synchronized to the standby server to the standby server according to the second type of work log, so that synchronization of relatively large data can be avoided to affect synchronization of other data.
  • the primary server may start multiple threads in sequence, and perform the above processing asynchronously. For example, after confirming successful data acquisition, a thread is generated to respond to the client's upload request. Another thread is generated to request that data of a size no larger than the preset value be synchronized to the alternate server.
  • the above threads can be executed asynchronously or synchronously.
  • the primary server may also initiate a reconciliation process periodically, as shown in FIG. 3, including:
  • Step 201 Read a plurality of first-class work logs stored locally, and perform the following steps 202 to 205 for each of the first-type work logs.
  • Step 202 Acquire a first verification code of the first data corresponding to the first type of work log stored by the primary server.
  • the primary server can read information of the data from the first type of work log.
  • the data may include: identification information, storage path, and the like.
  • the first data is read according to the information of the data, and the first data is hashed to obtain a first verification code.
  • Step 203 Acquire a second verification code of the second data corresponding to the first type of work log stored by the standby server.
  • the primary server may initiate a request to the standby server, where the request may include information in the first type of work log, such as identification information of the data, and the standby server obtains the corresponding second data according to the request, and the second data is used.
  • a hash calculation is performed to obtain a second verification code and returned to the primary server.
  • Step 204 Determine whether the first verification code is consistent with the second verification code. If they are consistent, return to step 202 for another first type of work log; if not, the data stored on the primary server and the standby server are inconsistent. If the synchronization needs to be performed again, step 205 is performed.
  • Step 205 Send the first data to the standby server, so that the standby server replaces the second data with the first data.
  • FIG. 3 is only a specific embodiment.
  • the synchronization operation can be re-executed for the inconsistent synchronization data stored on the primary server and the standby server, thereby ensuring data consistency.
  • the active server can continuously initiate a synchronous redo process.
  • the primary server continuously reads the locally stored second type of work logs one by one, and for each read.
  • the second type of work log obtains the third data corresponding to the second type of work log stored by the primary server, and sends the third data to the standby server, so that the standby server stores the third data.
  • the primary server may store data corresponding to each upload request to another storage server deployed outside the primary server, and the primary server still has the first type of work log and the second type.
  • the work log is stored on the primary server, and the stored first type of work log and the second type of work log are sent to the log backup machine, so that the log backup machine stores, and when the primary server fails, the log is used.
  • the backup machine initiates the above reconciliation process, that is, according to the first type of work log, when the data stored by the primary server and the standby server are inconsistent, the data stored on the storage server is sent to the standby server for storage; or according to the second type of work.
  • the log sends the failed data to the standby server for storage.
  • a data synchronization acquisition method is provided, which can be mainly applied to the data synchronization system shown in FIG. 1.
  • the method in this embodiment is a method for acquiring data that has been synchronized by the standby server in the system, and the flow is
  • the process diagram is shown in Figure 4, including:
  • Step 301 Receive a data acquisition request sent by the client, and include information about the data to be acquired in the data acquisition request.
  • the standby server stores the corresponding data according to the synchronization request.
  • the user can initiate the acquisition of the specific data that has been uploaded by the client directly to the standby server to verify the consistency of the specific data stored on the standby server and the primary server, and the standby server initiates the process of this embodiment.
  • Step 302 Acquire a third verification code that locally stores data corresponding to the information of the data to be acquired, where the locally stored data is stored according to the synchronization request sent by the primary server.
  • the standby server may read the corresponding data according to the information of the data to be acquired, such as the identification information of the data to be acquired, and then obtain a third verification code by performing hash calculation on the read data.
  • Step 303 Acquire a fourth verification code of data corresponding to the information of the data to be acquired stored by the primary server.
  • the standby server may initiate a request to the primary server, where the request may include identification information of the data to be acquired, and the primary server obtains corresponding data according to the request, and then may perform hash calculation on the data to obtain a fourth verification.
  • the code is returned to the alternate server.
  • Step 304 Determine whether the third verification code is consistent with the fourth verification code. If the third verification code is inconsistent, or the third verification code is empty and the fourth verification code is not empty, the data stored on the primary server and the standby server are inconsistent, and then executed. Step 305: If the fourth verification code is empty and the third verification code is not empty, indicating that the corresponding data has been deleted on the primary server, step 306 is performed; if yes, step 307 is performed.
  • Step 305 Acquire data corresponding to the information of the data to be acquired from the primary server, and respond to the client with data corresponding to the information of the data to be acquired.
  • the standby server may record the second type of work log for the data, and the second type of work log includes information of the data synchronization failure corresponding to the information of the data to be acquired.
  • the identification information of the data, and the information indicating that the data synchronization fails may also include the size of the data.
  • the standby server can continuously initiate the redo process, that is, read the second type of work log stored by the standby server, obtain the data corresponding to the second type of work log from the primary server for storage, and immediately ensure the primary server and The consistency of the data stored on the alternate server.
  • Step 306 deleting locally stored data corresponding to the information to be acquired, and responding to the client The information corresponding to the information corresponding to the information to be acquired does not exist.
  • Step 307 Respond to the client for data corresponding to the locally stored information of the data to be acquired.
  • FIG. 4 is only a specific embodiment.
  • the client can actively obtain data that has been synchronized on the standby server to the standby server, and the standby server verifies, according to the information of the data to be acquired, whether the data stored by the standby server is consistent with the data stored by the active server, if If they are inconsistent, the data is retrieved from the primary server and returned to the client.
  • the standby server can verify whether the data stored on the standby server and the primary server are consistent according to the request of the client, and then can take certain measures to ensure the consistency of the data stored on the standby server and the primary server.
  • the data synchronization method of the present embodiment is described in a specific embodiment, and is mainly applied to the data synchronization system shown in FIG. 5, and the data in this embodiment is an upload file.
  • the system of this embodiment :
  • the main server includes: a CGI module, a sync (Syncer) module, a storage module, a reconciliation module, and a redoer module.
  • the standby server includes a CGI module, a storage module, and a redo module. And in the specific synchronization process of the data in this embodiment, the flowchart is as shown in FIG. 6, and includes:
  • step 401 the client sends an upload request, and the upload file includes an upload file.
  • Step 402 The CGI module of the primary server receives the upload request, and processes the upload request, that is, parses the uploaded file in the upload request. If the size of the uploaded file is not greater than the preset value, the uploaded file is stored in the storage. In the module. The CGI module then sends the uploaded file to the synchronization module of the primary server.
  • Step 403 After receiving the uploaded file, the synchronization module immediately responds to the CGI module of the active server, so that the CGI module records the first type of work log and stores it in the storage module.
  • the full-flow Binlog is used. Including the meta information of the file, that is, the file name, the upload path, the bucket name, the client identifier, and the file size; if the synchronization module returns a failure message to the CGI module, or the communication between the CGI module and the synchronization module appears
  • the CGI module records the second type of work log and stores it in the storage module.
  • the error is Binlog.
  • the error Binlog is the same as the full-flow Binlog format. The difference is that the error is also included in the error Binlog. Information about file synchronization failure, etc.
  • the uploaded file does not pass through the synchronization module, and the CGI module directly records the error Binlog.
  • Binlog a full-flow Binlog or error Binlog is stored for an upload request.
  • Step 404 After receiving the feedback of the synchronization module, the CGI module of the primary server responds to the upload request of the client.
  • the response of the synchronization module to the CGI module is used to indicate whether the synchronization module successfully receives the uploaded file. If the uploaded file is successfully received, the synchronization module sends a successful response to the CGI module. If the uploaded file is not successfully received, The synchronization module will send a failure response to the CGI module.
  • the response of the CGI module to the client's upload request is used to indicate whether the primary server can successfully process the upload request, that is, whether the uploaded file has been successfully received and the uploaded file is stored, and the CGI module only receives the response of the synchronization module. Will respond to the client.
  • Step 405 The synchronization module of the primary server further initiates a synchronization request to the CGI module of the standby server, and includes the uploaded file acquired by the synchronization module in the synchronization request.
  • the CGI module of the standby server After receiving the synchronization request, stores the uploaded file included therein into the storage module (this step is not shown in FIG. 6).
  • the synchronization module is a multi-threaded, fully asynchronous module that can immediately respond to the CGI module and asynchronously initiate a synchronization request to the CGI module of the standby server, thereby synchronizing the uploaded file from the primary server to the standby server as soon as possible without affecting the primary
  • the business process is the upload process between the primary server and the client. For example, after confirming that the uploaded file is successfully received, the primary server generates a first thread, and uses the first thread to invoke an upload request of the CGI module corresponding to the client, and generates a second thread to invoke the synchronization module to The CGI module of the standby server initiates a synchronization request and performs subsequent synchronization operations.
  • Step 406 If the synchronization request initiated by the synchronization module to the CGI module of the standby server fails, the synchronization module may record the error Binlog and store it in the storage module.
  • the primary server may perform the following steps.
  • the flowchart is as shown in FIG. 7, and includes:
  • Step 501 The reconciliation module of the primary server can periodically initiate a reconciliation process, and read the full amount of the running water Binlog stored in the storage module for a period of time.
  • the reconciliation module can process the full-flow Binlog stored for four hours before the storage module at the whole point.
  • Step 502 The reconciliation module obtains the verification code of the uploaded file stored in the storage module for each full-flow Binlog, in this embodiment, the message digest algorithm 5 (MD5) -1;
  • the reconciliation module also requests the CGI module of the standby server to obtain the MD5-2 value of the corresponding uploaded file stored in the storage module of the standby server.
  • the reconciliation module can simultaneously send a HEAD request to the CGI module of the primary server and the CGI module of the standby server to obtain MD5-1 and MD5-2 values, respectively.
  • step 503 if MD5-1 is the same as MD5-2, indicating that the uploaded files stored in the active and standby servers are the same, the other full-flow Binlog is processed according to the step of step 502; if not, the reconciliation module may notify the redo module. Resynchronize.
  • the reconciliation module also has a part of the function of the redo module.
  • the MD5-1 is inconsistent with the MD5-2, the corresponding upload file is sent to the CGI module of the standby server.
  • Step 504 The redo module sends the corresponding upload file stored in the storage module of the primary server to the CGI module of the standby server, and the CGI module of the standby server stores the file to the storage module.
  • Step 505 The redo module of the primary server continuously reads the error Binlog stored in the storage module, and performs a redo process, that is, sends the uploaded file that is not synchronized to the backup server identified in the error Binlog to the CGI of the standby server. Module. If the redo module does not read the error Binlog, it will wait for 30s and then continue to try to read the error Binlog and perform the redo process.
  • the redo module fails to perform a redo process on a certain error Binlog, it can perform a retry of up to three redo processes for the error Binlog. If all fails, the upload file is discarded. Generally, all the failures occur when the network between the active and standby servers fails. In this case, another way may be adopted, that is, after the network is restored, the reconciliation module of the primary server can read the specified time. The error Binlog, and the corresponding upload file is sent by the reconciliation module to the CGI module of the standby server.
  • the reconciliation module can set the specified one-hour task as uniformly as possible at the current time. One hour of execution.
  • the CGI module of the primary server can store the uploaded file to another deployment outside the primary server when processing the upload request.
  • a storage server (the storage server dedicated storage upload file), and the full stream Binlog and error Binlog are stored in the storage module of the primary server.
  • the synchronization module of the primary server can back up the full-flow Binlog and error Binlog stored in the storage module to the additionally deployed log backup machine.
  • the log backup machine also has the function of initiating the reconciliation process, specifically after backing up each Binlog.
  • the processing of the full-flow Binlog stored in the one hour before the backup operation is initiated. How to deal with the reconciliation module of the active server in the above steps 502 to 504 is similar to the processing method of the full-flow Binlog, except that the log backup is performed.
  • the machine will get the MD5 value of the uploaded file from the storage server that is deployed separately.
  • the standby server may perform the following steps.
  • the flowchart is as shown in FIG. 8, and includes:
  • Step 601 The client sends a Get (GET) request for an uploaded file to the standby server, where the information of a certain uploaded file is included.
  • GET Get
  • Step 602 The CGI module of the standby server acquires the MD5 value of the corresponding uploaded file stored in the storage module according to the obtaining request, and requests the CGI module of the primary server to obtain the MD5 value of the uploaded file corresponding to the obtaining request.
  • the CGI module of the standby server may obtain the MD5 value of the corresponding uploaded file stored by the storage module of the standby server through the back interface, and send a HEAD request to the CGI module of the primary server, where the HEAD request is used to obtain the primary server.
  • the MD5 value of the corresponding uploaded file stored on the storage module.
  • Step 603 The CGI module of the standby server compares the obtained two MD5 values. If the two MD5 values are consistent, the corresponding upload file stored in the storage module of the standby server is returned to the client.
  • Step 604 if the two MD5 values are inconsistent, or the MD5 value obtained from the local is null, and the MD5 value obtained from the CGI module of the active server is not empty, the CGI module of the standby server will be from the CGI module of the primary server. Obtain the corresponding upload file and return it to the client. The CGI module can record the error Binlog corresponding to the uploaded file and store it in the storage module.
  • Step 605 If the MD5 value obtained from the local is not empty, and the MD5 value obtained from the CGI module of the active server is null, the information indicating that the uploaded file does not exist may be directly returned to the client. And the CGI module of the standby server deletes the uploaded file stored in the storage module.
  • the uploaded file obtained from the primary server may be sent to the client; if the CGI module of the standby server obtains the MD5 value of the uploaded file from the standby server, If successful, you can directly send the locally stored upload file to the client.
  • step 606 the redo module of the standby server continuously reads the error Binlog from the storage module, and performs a redo process, that is, the upload file corresponding to the error Binlog is obtained from the CGI module of the primary server and stored in the storage module. If the redo module does not read the error Binlog, it will wait for 30s and continue to try to read the error. Binlog, and the redo process.
  • the redo module fails to perform a redo process on a certain error Binlog, it can perform a retry of up to three redo processes for the error Binlog.
  • a data synchronization device such as the above-mentioned primary server, is shown in FIG.
  • the uploading and receiving unit 10 is configured to receive an uploading request sent by the client, where the uploading request includes data to be uploaded;
  • the request processing unit 11 is configured to: after successfully acquiring the data, store the acquired data in response to the upload request of the client;
  • the synchronization unit 12 is configured to initiate a synchronization request for the data to the standby server if the data stored by the request processing unit 11 is not greater than a preset value, so that the standby server stores the data.
  • the first recording unit 13 is configured to record a second type of work log if the data stored by the request processing unit 11 is greater than a preset value, where the second type of work log includes: indication information, indicating that the data is not synchronized The data of the standby server.
  • the uploading and receiving unit 10 receives the uploading request sent by the client, and the request processing unit 11 processes the uploading request, that is, after successfully acquiring the data, it immediately responds to the upload request of the client, and the synchronization unit 12 asynchronously synchronizes data of a size no larger than the preset value to the standby server. Since the data synchronization device performs the asynchronous processing step of the upload request and the synchronization step for the data, the data synchronization device does not need to wait for the data synchronization device and the backup server to store the data, and then responds to the client, and implements Synchronization of data does not affect the uploading of business processes.
  • the data synchronization device temporarily does not perform the synchronization operation, except that the first recording unit 13 records the second type of work log, and the second type of work log includes the information of the data that fails to synchronize with the standby server. In this way, the data synchronization device can subsequently complete the synchronization of the data according to the second type of work log, and can avoid synchronization of the larger data to affect the synchronization of other data.
  • the data synchronization device may include, in addition to the structure shown in FIG. 9, a first recording unit 13, a reconciliation unit 14, a first redo unit 15 and Log backup unit 16, where:
  • the first recording unit 13 is further configured to: if the data is successfully synchronized to the standby server, such as The synchronization request initiated by the synchronization unit 12 to the standby server is successful, and the first type of work log is recorded, and the information of the data is included in the first type of work log.
  • the first recording unit 13 is further configured to: if the data is failed to be synchronized to the standby server, for example, the synchronization request initiated by the synchronization unit 12 to the standby server fails, and the second type of work log is recorded, in the second type of work log. Information including the data that failed to synchronize with the alternate server.
  • the reconciliation unit 14 is configured to periodically initiate a reconciliation process, that is, read a plurality of first type of work logs recorded by the first recording unit 13; for each of the first type of work logs, from the first Reading the information of the data in the class work log, acquiring the first data locally according to the information of the data, and the first verification code of the first data, acquiring the first request from the standby server according to the information of the data.
  • the second data, and the second verification code of the second data notify the first redo unit 15 to send the first data;
  • the first redo unit 15 is configured to send the first data to the standby server, so that the standby server replaces the second data with the first data.
  • the first redo unit 15 is further configured to continuously initiate a redo process, that is, read a second type of work log recorded by the first recording unit 13; and acquire the second data stored by the data synchronization device
  • the third data corresponding to the class work log is sent to the standby server, so that the standby server stores the third data.
  • the log backup unit 16 is configured to send the first type of work log and the second type of work log recorded by the first record unit 13 to the log backup machine, so that the log backup machine determines according to the first type of work log.
  • the data stored by the primary server and the standby server are inconsistent, the inconsistent data stored by the data synchronization device is sent to the standby server for storage; or the second type of work log is not synchronized to the standby.
  • the server's data is sent to the alternate server for storage.
  • a data synchronization acquisition device such as the above-mentioned backup server, is shown in FIG.
  • the receiving unit 20 is configured to receive a data acquisition request sent by the client, where the data acquisition request includes information about the data to be acquired;
  • the verification unit 21 is configured to acquire a third verification code that locally stores data corresponding to the information of the data to be acquired included in the data acquisition request received by the acquisition receiving unit 20, where the locally stored data is Obtaining, according to the synchronization request sent by the primary server, obtaining a fourth verification code of the data corresponding to the information about the data to be acquired stored by the primary server; if the third verification code is inconsistent with the fourth verification code, Or the third verification code is empty and the fourth verification code is not empty, and the notification verification processing unit 22 responds to the customer order;
  • the verification processing unit 22 is configured to acquire data corresponding to the information of the data to be acquired from the primary server, and respond to the client with data corresponding to the information of the data to be acquired.
  • the verification processing unit 22 is further configured to: if the verification order 21 determines that the fourth verification code is empty, and the third verification code is not empty, delete the locally stored data corresponding to the information of the to-be-acquired data. And responding to the client with information that the data corresponding to the information of the data to be acquired does not exist.
  • the client can actively obtain the data that has been synchronized locally to the data synchronization acquiring device of the embodiment, so that the verification unit 21 of the data synchronization acquiring device first verifies the data stored by the standby server and the active server. Whether the stored data is consistent. If they are inconsistent, the data obtained from the primary server is returned to the client. In this way, the data synchronization obtaining device can verify whether the data stored on the primary server is consistent according to the request of the client, and thus can adopt a certain means to ensure the consistency of the data stored on the data synchronization acquiring device and the primary server.
  • the data synchronization device may include, in addition to the structure shown in FIG. 11, a second recording unit 23 and a second redo unit 24, wherein:
  • a second recording unit 23 configured to record the second category if the verification unit 21 verifies that the third verification code is inconsistent with the fourth verification code, or the third verification code is empty and the fourth verification code is not empty a work log, where the second type of work log includes information that the data synchronization fails corresponding to the information of the data to be acquired.
  • the second redo unit 24 is configured to read the second recording unit 23 to record the second type of work log, and obtain data corresponding to the second type of work log from the primary server for storage. Thereby ensuring the consistency of the data stored on the data synchronization acquisition device and the primary server.
  • a server is also provided.
  • the schematic diagram of the structure is shown in FIG. 13.
  • the server may have a large difference due to different configurations or performances, and may include one or more central processing units (CPUs) 30 (for example, One or more processors) and memory 31, one or more storage media 32 storing application 321 or data 322 (eg, one or one storage device in Shanghai).
  • the memory 31 and the storage medium 32 may be short-term storage or permanent storage.
  • the program stored on the storage medium 32 can be To include one or more modules (not shown), each module can include a series of instruction operations in a data synchronization system.
  • central processor 30 may be arranged to communicate with storage medium 32 to perform a series of instruction operations in storage medium 32 on the data synchronization system.
  • the server may also include one or more power sources 33, one or more wired or wireless network interfaces 34, one or more input and output interfaces 35, and/or one or more operating systems 323, such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and more.
  • the steps performed by the primary server or the standby server described in the above method embodiments may be based on the structure of the server shown in FIG.
  • the memory 31 or the storage medium 32 stores machine readable storage execution
  • the central processor 30 executes the machine readable storage to execute the method shown in FIG. 2, FIG. 3, FIG. 4, FIG. 6 to FIG. And the operation of the apparatus shown in Figs. 9 to 12 .
  • the embodiment of the present invention further provides a data synchronization system, including a primary server and a backup server, wherein the configuration of the primary server may be the data synchronization device according to the first embodiment of the device, or the device according to the third embodiment.
  • the structure of the server; the standby server is configured to receive a synchronization request for the data, and store the data.
  • the structure of the backup server may be the same as the structure of the server as described in the foregoing device embodiment 2, or the structure of the server as described in the third embodiment of the device, and details are not described herein.
  • Embodiments of the present application provide a non-volatile storage medium for storing machine readable instructions that are executed when the machine readable instructions are executed.
  • the program may be stored in a computer readable storage medium, and the storage medium may include: Read only memory (ROM), random access memory (RAM), magnetic or optical disk, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Hardware Redundancy (AREA)
  • Information Transfer Between Computers (AREA)
  • Computer And Data Communications (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本发明实施例公开了数据同步方法、系统、同步获取方法及相关装置，应用于通信技术领域。在本实施例的方法中，主用服务器接收到客户端发送的上传请求，在成功获取所述数据后，响应所述客户端的上传请求，存储获取的数据，且异步地将大小不大于预置的值的数据同步给备用服务器，而对于大小大于预置的值的数据，主用服务器不会进行同步操作，只是记录第二类工作日志，第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的数据。

Description

数据同步方法、系统、同步获取方法及装置
本申请要求于2016年9月13日提交中国专利局、申请号为201610821980.0、发明名称为“一种数据同步方法、系统、同步获取方法及相关装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明实施例涉及通信技术领域，特别涉及一种数据同步方法、系统、同步获取方法及相关装置。
背景技术
现有的云储存系统中部署有主备服务器和备用服务器。当主用服务器故障时，利用备用服务器储存用户通过客户端上传的数据。这就需要确保在主用服务器和备用服务器中存储的对应数据是一样的，这样在主备服务器切换后，能够保证同样可以从备用服务器获取客户端之前上传的数据。
发明内容
本发明实施例提供一种数据同步方法、系统、同步获取方法及相关装置，实现了在对上传请求进行处理后，异步地将大小不大于预置的值的数据传送给备用服务器进行同步。
本发明实施例提供一种数据同步方法，应用于主用服务器，所述方法包括：
接收客户端发送的上传请求，所述上传请求中包括待上传的数据；
在成功获取所述数据后，响应所述客户端的上传请求；
存储所述获取的数据；
当所述数据不大于预置的值时，向备用服务器发起同步请求，以请求将所述数据同步到所述备用服务器，以使所述备用服务器储存所述数据；
当所述数据大于所述预置的值时，生成并存储第二类工作日志，所述第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的数据。
本发明实施例还提供一种数据获取方法，应用于备用服务器，所述方法包括：
接收客户端发送的数据获取请求，所述数据获取请求中包括待获取数据的信息；
获取本地储存与所述待获取数据的信息对应的数据的第三验证码；其中，所述本地储存的数据是根据主用服务器发送的同步请求储存的；
获取所述主用服务器储存的与所述待获取数据的信息对应的数据的第四验证码；
如果所述第三验证码与第四验证码不一致，或所述第三验证码为空且所述第四验证码不为空，从所述主用服务器获取与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据。
本发明实施例还提供一种数据同步装置，包括：
上传接收单元，用于接收客户端发送的上传请求，所述上传请求中包括待上传的数据；
请求处理单元，用于在成功获取所述数据后，响应所述客户端的上传请求，存储所述获取的数据；
同步单元，用于当所述数据的不大于预置的值时，向备用服务器发起同步请求，以请求将所述数据同步到所述备用服务器，以使所述备用服务器储存所述数据；
第一记录单元，用于当所述数据的大小大于所述预置的值时，生成并存储第二类工作日志，所述第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的所述数据。
本发明实施例还提供一种数据同步获取装置，其特征在于，包括：
获取接收单元，用于接收客户端发送的数据获取请求，所述数据获取请求中包括待获取数据的信息；
验证单元，用于获取本地储存与所述待获取数据的信息对应的数据的第三验证码，其中，所述本地储存的数据是根据主用服务器发送的同步请求储存的；获取所述主用服务器储存的与所述待获取数据的信息对应的数据的第四验证码；如果所述第三验证码与第四验证码不一致，或所述第三验证码为空且所述第四验证码不为空，通知验证处理单元响应所述客户单；
验证处理单元，用于从所述主用服务器获取与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据。
本发明实施例还提供一种数据同步系统，包括主用服务器和备用服务器，其中， 所述主用服务器包括：
上传接收单元，用于接收客户端发送的上传请求，所述上传请求中包括待上传的数据；
请求处理单元，用于在成功获取所述数据后，响应所述客户端的上传请求，存储所述获取的数据；
同步单元，用于当所述数据的不大于预置的值时，向备用服务器发起同步请求，以请求将所述数据同步到所述备用服务器，以使所述备用服务器储存所述数据；
第一记录单元，用于当所述数据的大小大于所述预置的值时，生成并存储第二类工作日志，所述第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的所述数据；
所述备用服务器，用于接收到对所述数据的同步请求，储存所述数据。
可见，在本实施例的方法中，主用服务器接收到客户端发送的上传请求，对上传请求进行处理即成功获取数据后，会立即响应客户端的上传请求，且异步地将大小不大于预置的值的数据同步给备用服务器。由于主用服务器对于上传请求的处理步骤与对于数据的同步步骤之间是异步进行的，主用服务器并不需要等待主用服务器和备用服务器都储存了数据后，才会响应客户端，实现了对于数据的同步不会影响上传业务流程。且对于大小大于预置的值的数据，主用服务器暂时不会进行同步操作，只是记录第二类工作日志，第二类工作日志中包括向备用服务器同步失败的数据的信息，这样主用服务器后续可以根据第二类工作日志完成该数据的同步，可以避免比较大的数据的同步影响其它数据的同步。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例提供的一种数据同步系统的结构示意图；
图2是本发明方法实施例一提供的一种数据同步方法的流程图；
图3是本发明方法实施例一中主用服务器发起的对账流程的流程图；
图4是本发明方法实施例二提供的一种数据同步获取方法的流程图；
图5是本发明方法实施例三提供的一种数据同步系统的结构示意图；
图6是本发明方法实施例三提供的一种数据同步方法的流程图；
图7是本发明方法实施例三中主用服务器执行的数据一致性的方法流程图；
图8是本发明方法实施例三中备用服务器执行的数据一致性的方法流程图；
图9是本发明设备实施例一提供的一种数据同步装置的结构示意图；
图10是本发明设备实施例一提供的另一种数据同步装置的结构示意图；
图11是本发明设备实施例二提供的一种数据同步获取装置的结构示意图；
图12是本发明设备实施例二提供的另一种数据同步获取装置的结构示意图；
图13是本发明设备实施例三提供的一种服务器的结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”“第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以使这里描述的本发明的实施例例如能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排它的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本申请一实施例中，在主备服务器上进行数据同步时，当主用服务器接收到客户端的上传请求后，会将给上传请求同时传送给主用服务器和备用服务器上的公共网关接口(Common Gateway Interface，CGI)模块以对上传请求进行处理，当主用服务器和备用服务器的CGI模块分别对上传请求都处理完后，主用服务器才会响应该上传请求。如果在主备服务器之间跨地域、跨运营商的情况下，会造 成响应时间长，可靠性低的问题。并且这种方案的可控性比较低，无法应对各种突发情况，例如流量陡升出现大量主备服务器之间的同步错误，以及主备服务器之间断网，断网期间的数据完全无法同步到备用服务器的情况。为了解决上述问题，本发明实施例提供如下技术方案。
本发明方法实施例一
提供一种数据同步方法，主要应用于如图1所示的数据同步系统，比如云储存系统等，在系统中包括客户端110，主用服务器120和备用服务器130，其中，客户端110可以向主用服务器120数据，由主用服务器120存储客户端上传的数据，并将该数据同步至备用服务器130。一般情况下，客户端110与备用服务器130之间不进行通信，而是与主用服务器120之间进行通信，在本实施例中，客户端110可以与备用服务器130之间进行通信，主备服务器120和备用服务器130存储的对应数据是否一致。
本实施例的方法是系统中的主用服务器所执行的方法，流程图如图2所示，包括：
步骤101，接收客户端发送的上传请求，在上传请求中包括待上传的数据。
步骤102，在成功获取该数据后，响应客户端的上传请求，储存获取的数据。
步骤103，如果步骤102储存的数据的大小不大于预置的值，向备用服务器发起对该数据的同步请求，以请求将该数据同步到该备用服务器，以使该备用服务器储存该数据。
可以理解，上述步骤102中响应客户端的上传请求，与本步骤103的同步步骤之间并没有顺序关系，可以同时执行，也可以顺序执行，图2中所示的只是一个具体的实施例。
步骤104，如果数据的大小大于预置的值，生成并存储第二类工作日志，第二类工作日志中包括：向备用服务器同步失败的数据的信息，即指示信息，用于指示还未同步到该备用服务器的数据。例如，在本发明一实施例中，该指示信息包括：数据的标识信息，及标识该数据同步失败和/或未进行同步的信息，以及标识数据的大小的信息等。
本实施例中，对于比较大的数据，主用服务器不会根据对上传请求的处理来同步该数据给备用服务器，而是先记录第二类工作日志。此外，如果向备用服务器同 步数据失败时，主用服务器也可以记录该第二类工作日志。向备用服务器同步数据失败可以指主用服务器向备用服务器发起同步请求失败，导致数据未成功同步至备用服务器。
如果向备用服务器同步数据成功，比如主用服务器向备用服务器发起的同步请求成功，主用服务器还可以记录第一类工作日志，每一第一类工作日志中可以包括：成功同步的数据的信息，比如数据的标识信息，存储路径比如储存数据的桶(Bucket)的名称及数据大小，且还可以包括客户端标识及上传路径等信息。
需要说明的是，一个上传请求对应一条第一类工作日志或对应一条第二类工作日志。
可见，在本实施例的方法中，主用服务器接收到客户端发送的上传请求，对上传请求进行处理，即成功获取数据后，会立即响应客户端的上传请求，且异步地将大小不大于预置的值的数据同步给备用服务器。由于主用服务器对上传请求的响应处理与主用服务器对数据的同步处理，即同步到备用服务器是异步进行的，主用服务器并不需要等主用服务器和备用服务器都储存了数据后，才响应客户端的上传请求。因为快速响应客户端的上传请求，数据的同步过程不会影响上传业务的流程。且对于大小大于预置的值的数据，主用服务器暂时不会进行同步操作，而是在第二类工作日志记录未同步到备用服务器的数据。这样，主用服务器后续可以根据该第二类工作日志将还未同步到备用服务器的数据同步至该备用服务器，可以避免比较大的数据的同步影响其它数据的同步。
在本发明实施例中，主用服务器可以先后启动多个线程，异步执行上述处理。例如，在确认成功获取数据后，生成一线程用于响应客户端的上传请求。生成另外一个线程，用于请求将大小不大于预置的值的数据同步给备用服务器。上述线程可以异步或同步执行。
在一个具体的实施例中，如果在主用服务器上储存了上述第一类工作日志，主用服务器还可以定时地发起对账流程，具体可以如图3所示，包括：
步骤201，读取本地储存的多条第一类工作日志，分别针对每一条第一类工作日志，执行如下步骤202到205。
步骤202，获取主用服务器储存的与第一类工作日志对应的第一数据的第一验证码。具体地，主用服务器可以从该第一类型工作日志中读取数据的信息。该数据 的信息可以包括：标识信息，存储路径等。根据该数据的信息读取第一数据，对第一数据进行哈希计算得到第一验证码。
步骤203，获取备用服务器储存的与该第一类型工作日志对应的第二数据的第二验证码。具体地，主用服务器可以向备用服务器发起请求，在请求中可以包括第一类型工作日志中的信息比如数据的标识信息等，备用服务器根据该请求获取到对应的第二数据，对第二数据进行哈希计算得到第二验证码，并返回给主用服务器。
步骤204，判断第一验证码与第二验证码是否一致，如果一致，则针对另一条第一类工作日志，返回执行步骤202；如果不一致，说明主用服务器与备用服务器上储存的数据不一致，需要重新进行同步，则执行步骤205。
步骤205，将第一数据发送给备用服务器，以使备用服务器用第一数据替换第二数据。
需要说明的是，上述步骤202和203之间没有顺序关系，可以同时执行，也可以顺序执行，图3所示的只是一种具体的实施例。
通过上述步骤201到205，可以针对主用服务器和备用服务器上储存的不一致的同步数据，重新执行同步操作，保证了数据的一致性。
在另一个具体的实施例中，主用服务器可以不断地发起同步的重做流程，具体地：主用服务器会不断地逐条地读取本地储存的第二类工作日志，并针对读取的每条第二类工作日志，获取主用服务器储存的与该第二类工作日志对应的第三数据，将第三数据发送给备用服务器，以使备用服务器储存第三数据。
在其它具体的实施例中，主用服务器可以将各个上传请求对应的数据储存到该主用服务器之外部署的另一个存储服务器上，而主用服务器还是将第一类工作日志和第二类工作日志储存到该主用服务器上，且会将储存的第一类工作日志和第二类工作日志发送给日志备份机器，以使日志备份机器进行储存，且在主用服务器故障时，由日志备份机器发起如上的对账流程，即根据第一类工作日志确定主用服务器和备用服务器储存的数据不一致时，将存储在存储服务器上的数据发送给备用服务器进行储存；或根据第二类工作日志将同步失败的数据发送给备用服务器进行储存。
本发明方法实施例二
提供一种数据同步获取方法，主要可以应用于如图1所示的数据同步系统，本实施例的方法是系统中的备用服务器所执行的对于已经同步的数据的获取方法，流 程图如图4所示，包括：
步骤301，接收客户端发送的数据获取请求，在数据获取请求中包括待获取数据的信息。
可以理解，主用服务器根据客户端发送的上传请求储存数据后，发送对该数据的同步请求给备用服务器后，备用服务器根据同步请求储存对应的数据。而用户可以通过客户端直接向备用服务器发起对已经上传的特定数据的获取，以验证备用服务器与主用服务器上储存的该特定数据的一致性，则备用服务器会发起本实施例的流程。
步骤302，获取本地储存与待获取数据的信息对应的数据的第三验证码，其中，本地储存的数据是根据主用服务器发送的同步请求储存的。具体地，备用服务器可以根据待获取数据的信息比如待获取数据的标识信息等读取到对应的数据，然后可以通过对读取的数据进行哈希计算得到第三验证码。
步骤303，获取主用服务器储存的与待获取数据的信息对应的数据的第四验证码。具体地，备用服务器可以向主用服务器发起请求，在请求中可以包括待获取数据的标识信息等，主用服务器根据该请求获取到对应的数据，然后可以对数据进行哈希计算得到第四验证码，并返回给备用服务器。
步骤304，判断第三验证码与第四验证码是否一致，如果不一致，或第三验证码为空且第四验证码不为空，说明主用服务器和备用服务器上储存的数据不一致，则执行步骤305；如果第四验证码为空且第三验证码不为空，说明在主用服务器上已经删除了对应的数据，则执行步骤306；如果一致，则执行步骤307。
步骤305，从主用服务器获取与待获取数据的信息对应的数据，并向客户端响应与待获取数据的信息对应的数据。
在这种情况下，备用服务器可以记录针对该数据的第二类工作日志，在第二类工作日志中包括与该待获取数据的信息对应的数据同步失败的信息。比如数据的标识信息，及标识该数据同步失败的信息，且还可以包括数据的大小等。
这样，备用服务器还可以不断地发起重做流程，即读取备用服务器储存的第二类工作日志，从主用服务器获取与第二类工作日志对应的数据进行储存，即时地保证主用服务器和备用服务器上储存的数据的一致性。
步骤306，删除本地储存的与待获取数据的信息对应的数据，并向客户端响应 与待获取数据的信息对应的数据不存在的信息。
步骤307，向客户端响应本地储存的与待获取数据的信息对应的数据。
需要说明的是，上述步骤302和303之间没有绝对的顺序关系，可以同时执行，也可以顺序执行，图4所示的只是一种具体的实施例。
可见，本实施例中，客户端可以主动向备用服务器获取已经同步在备用服务器上的数据，备用服务器根据该待获取数据的信息验证备用服务器储存的数据与主用服务器储存的数据是否一致，如果不一致，则从主用服务器获取数据，返回给客户端。这样备用服务器可以根据客户端的请求验证备用服务器和主用服务器上储存的数据是否一致，进而可以采取一定的手段保证备用服务器和主用服务器上储存的数据的一致性。
本发明方法实施例三
以一个具体的实施例来说明本实施例的数据同步方法，主要应用于如图5所示的数据同步系统，且本实施例的数据为上传文件，在本实施例的系统中：
主用服务器包括：CGI模块，同步(Syncer)模块，存储模块，对账模块和重做(Redoer)模块。备用服务器包括CGI模块，存储模块和重做模块。且在本实施例的数据在具体的同步过程中，流程图如图6所示，包括：
步骤401，客户端发送上传请求，在上传请求中包括上传文件。
步骤402，主用服务器的CGI模块接收到上传请求，对上传请求进行处理，即解析得到上传请求中的上传文件，如果上传文件的大小不大于预置的值，则将该上传文件储存到存储模块中。然后CGI模块将上传文件发送给主用服务器的同步模块。
步骤403，同步模块在接收到上传文件后，立即响应主用服务器的CGI模块，这样CGI模块会记录第一类工作日志并储存到存储模块中，在本实施例中为全量流水Binlog，具体可以包括文件的元(meta)信息，即文件名、上传路径、Bucket名称，客户端标识和文件大小等内容；如果同步模块向CGI模块返回了失败信息，或CGI模块与同步模块之间的通信出现错误时，CGI模块会记录第二类工作日志并储存到存储模块中，在本实施例中为错误Binlog，错误Binlog与全量流水Binlog的格式一样，不同的是，在错误Binlog中还包括标识上传文件同步失败的信息等。
其中，对于如果上传文件的大小比较大，比如大于预置的值，这种上传文件不经过同步模块，CGI模块会直接记录错误Binlog。
需要说明的是，针对一个上传请求会储存一条全量流水Binlog或错误Binlog。
步骤404，当主用服务器的CGI模块接收到同步模块的反馈后，会响应客户端的上传请求。
需要说明的是，同步模块给CGI模块的响应用来表示同步模块是否成功接收到上传文件，如果成功接收到上传文件，则同步模块就会发送成功响应给CGI模块，如果未成功接收上传文件，则同步模块会发送失败响应给CGI模块。而CGI模块对客户端的上传请求的响应是用来表示主用服务器是否能成功对上传请求进行处理，即是否已成功接收该上传文件，并储存上传文件，CGI模块只要接收到同步模块的响应，就会响应客户端。
步骤405，主用服务器的同步模块还会向备用服务器的CGI模块发起同步请求，在同步请求中包括同步模块获取的上传文件。
备用服务器的CGI模块在接收到同步请求后，会将其中包括的上传文件储存到存储模块中(图6中未示出这个步骤)。
同步模块是多线程、全异步执行的模块，可以立即响应CGI模块，且异步地向备用服务器的CGI模块发起同步请求，从而尽快地将上传文件从主用服务器同步到备用服务器，并且不影响主业务流程即主用服务器与客户端之间的上传流程。例如，该主用服务器在确认已成功接收到该上传文件后，生成第一线程，利用该第一线程调用其CGI模块相应该客户端的上传请求，生成第二线程调用其同步模块，以向该备用服务器的CGI模块发起同步请求，以及执行后续同步操作。
步骤406，如果同步模块向备用服务器的CGI模块发起的同步请求失败，则同步模块可以记录错误Binlog，并储存到存储模块中。
在本实施例中，为了保障主用服务器和备用服务器上储存数据的一致性，主用服务器可以按照如下步骤来进行，流程图如图7所示，包括：
步骤501，主用服务器的对账模块可以定时地发起对账流程，从存储模块中读取一段时间内储存的全量流水Binlog。
在具体实现时，对账模块可以在整点，拉起对存储模块四个小时之前储存的全量流水Binlog进行处理。
步骤502，对账模块针对每条全量流水Binlog，获取存储模块中储存的上传文件的验证码，本实施例中为消息摘要算法第五版(Message Digest Algorithm 5，MD5) -1；同时对账模块还会向备用服务器的CGI模块请求获取备用服务器的存储模块中储存的对应的上传文件的MD5-2值。
具体地，对账模块可以同时向主用服务器的CGI模块及备用服务器的CGI模块发送头部(HEAD)请求，以分别获取MD5-1和MD5-2值。
步骤503，如果MD5-1与MD5-2一致，说明主备服务器中储存的上传文件一致，则按照上述步骤502的步骤处理另一条全量流水Binlog；如果不一致，则对账模块可以通知重做模块重新同步。
在另一种情况下，对账模块中也具有重做模块的部分功能，即当MD5-1与MD5-2不一致时，将对应的上传文件发送给备用服务器的CGI模块。
步骤504，重做模块会将主用服务器的存储模块中储存的对应的上传文件发送给备用服务器的CGI模块，由备用服务器的CGI模块储存到存储模块中。
步骤505，主用服务器的重做模块会不断地读取存储模块中储存的错误Binlog，并执行重做流程，即将该错误Binlog中标识的未同步到备用服务器的上传文件发送给备用服务器的CGI模块。如果重做模块读取不到错误Binlog，会等待30s后继续尝试读取错误Binlog，并进行重做流程。
如果重做模块对某一条错误Binlog执行的重做流程失败，可以针对该条错误Binlog进行最多三次的重做流程的重试，如果全部失败，就放弃发送上传文件。一般主备服务器之间网络出故障时会出现这种全部失败的情况，对于这种情况，可以采用另外一种途径，即在网络恢复后，可以由主用服务器的对账模块读取指定时间的错误Binlog，并由对账模块将对应的上传文件发送给备用服务器的CGI模块。
为了避免短时间内主用服务器大量获取上传文件的验证码(即上述发起的大量HEAD请求)对客户端的上传业务产生影响，对账模块可以把指定的一小时的任务尽量均匀地设置在当前这一小时执行。
在另一个具体的实施例中，针对主用服务器故障导致对账无法进行的问题，主用服务器的CGI模块在处理上传请求时，可以将上传文件储存到在该主用服务器之外部署的另一存储服务器(该存储服务器专用储存上传文件)中，而全量流水Binlog和错误Binlog还是储存在主用服务器的存储模块中。主用服务器的同步模块可以将存储模块中储存的全量流水Binlog和错误Binlog备份到另外部署的日志备份机器上。且日志备份机器也具有发起对账流程的功能，具体可以在备份了各条Binlog后， 发起对备份操作之前的一个小时内储存的全量流水Binlog的处理，具体如何处理可以见上述步骤502到504中主用服务器的对账模块对全量流水Binlog的处理方法类似，不同的是，日志备份机器会从另外部署的存储服务器获取上传文件的MD5值。
在本实施例中，为了保障主用服务器和备用服务器上储存数据的一致性，备用服务器可以按照如下步骤来进行，流程图如图8所示，包括：
步骤601，客户端发送某个上传文件的获取(GET)请求给备用服务器，其中包括某个上传文件的信息。
步骤602，备用服务器的CGI模块根据该获取请求获取存储模块中储存的对应的上传文件的MD5值，并向主用服务器的CGI模块请求获取该获取请求对应的上传文件的MD5值。
具体地，备用服务器的CGI模块可以通过后端接口获取备用服务器的存储模块储存的对应的上传文件的MD5值，并向主用服务器的CGI模块发送HEAD请求，该HEAD请求用于获取主用服务器的存储模块上储存的对应的上传文件的MD5值。
步骤603，备用服务器的CGI模块会将获取的两个MD5值进行比较，如果两个MD5值一致，则向客户端返回备用服务器的存储模块中储存的对应的上传文件。
步骤604，如果两个MD5值不一致，或者从本地获取的MD5值为空，且从主用服务器的CGI模块获取的MD5值不为空，则备用服务器的CGI模块会从主用服务器的CGI模块获取对应的上传文件，并返回给客户端。且CGI模块可以记录该上传文件对应的错误Binlog储存到存储模块中。
步骤605，如果从本地获取的MD5值不为空，且从主用服务器的CGI模块获取的MD5值为空，则可以直接向客户端返回用于指示上传文件不存在的信息。且备用服务器的CGI模块会删除存储模块中储存的该上传文件。
另外，如果备用服务器的CGI模块从本地获取上传文件的MD5值未成功，可以将从主用服务器获取的上传文件发送给客户端；如果备用服务器的CGI模块从备用服务器获取上传文件的MD5值未成功，可以直接将本地储存的上传文件发送给客户端。
步骤606，备用服务器的重做模块会不断地从存储模块中读取错误Binlog，执行重做流程，即向主用服务器的CGI模块获取错误Binlog对应的上传文件并储存到存储模块中。如果重做模块读取不到错误Binlog，会等待30s后继续尝试读取错误 Binlog，并进行重做流程。
如果重做模块对某一条错误Binlog执行的重做流程失败，可以针对该条错误Binlog进行最多三次的重做流程的重试。
本发明设备实施例一
还提供一种数据同步装置，比如上述的主用服务器，其结构示意图如图9所示，具体可以包括：
上传接收单元10，用于接收客户端发送的上传请求，所述上传请求中包括待上传的数据；
请求处理单元11，用于在成功获取所述数据后，响应所述客户端的上传请求，存储所述获取的数据；
同步单元12，用于如果请求处理单元11储存的所述数据的不大于预置的值，向备用服务器发起对所述数据的同步请求，以使所述备用服务器储存所述数据。
第一记录单元13，用于如果请求处理单元11储存的所述数据大于预置的值，记录第二类工作日志，所述第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的所述数据。
可见，在本实施例的数据同步装置中，上传接收单元10接收到客户端发送的上传请求，请求处理单元11对上传请求进行处理即成功获取数据后，会立即响应客户端的上传请求，同步单元12会异步地将大小不大于预置的值数据同步给备用服务器。由于数据同步装置对于上传请求的处理步骤与对于数据的同步步骤之间是异步进行的，数据同步装置并不需要等待数据同步装置和备用服务器都储存了数据后，才会响应客户端，实现了对于数据的同步不会影响上传业务流程。且对于大小大于预置的值的数据，数据同步装置暂时不会进行同步操作，只是第一记录单元13记录第二类工作日志，第二类工作日志中包括向备用服务器同步失败的数据的信息，这样数据同步装置后续可以根据第二类工作日志完成该数据的同步，可以避免比较大的数据的同步影响其它数据的同步。
参考图10所示，在一个具体的实施例中，数据同步装置除了可以包括如图9所示的结构外，还可以包括第一记录单元13，对账单元14，第一重做单元15和日志备份单元16，其中：
第一记录单元13，用于还用于如果向所述备用服务器同步所述数据成功，比如 同步单元12向备用服务器发起的同步请求成功，记录第一类工作日志，所述第一类工作日志中包括所述数据的信息。该第一记录单元13，还用于如果向所述备用服务器同步所述数据失败，比如同步单元12向备用服务器发起的同步请求失败，记录第二类工作日志，所述第二类工作日志中包括向所述备用服务器同步失败的所述数据的信息。
对账单元14，用于定时地发起对账流程，即读取多条所述第一记录单元13记录的第一类工作日志；针对每条所述第一类工作日志，从所述第一类工作日志中读取所述数据的信息，根据所述数据的信息从本地获取第一数据，及所述第一数据的第一验证码，根据所述数据的信息从所述备用服务器获取第二数据，及所述第二数据的第二验证码，如果所述第一验证码与第二验证码不一致，通知第一重做单元15发送所述第一数据；
第一重做单元15，用于将所述第一数据发送给所述备用服务器，以使所述备用服务器用所述第一数据替换所述第二数据。
所述第一重做单元15，还用于不断地发起重做流程，即读取所述第一记录单元13记录的第二类工作日志；获取所述数据同步装置储存的与所述第二类工作日志对应的第三数据，将所述第三数据发送给所述备用服务器，以使所述备用服务器储存所述第三数据。
日志备份单元16，用于将所述第一记录单元13记录的第一类工作日志和第二类工作日志发送给日志备份机器，以使所述日志备份机器根据所述第一类工作日志确定主用服务器和备用服务器储存的数据不一致时，将所述数据同步装置储存的所述不一致的数据发送给所述备用服务器进行储存；或根据所述第二类工作日志将未同步到所述备用服务器的数据发送给所述备用服务器进行储存。
本发明设备实施例二
还提供一种数据同步获取装置，比如上述的备用服务器，其结构示意图如图11所示，具体可以包括：
获取接收单元20，用于接收所述客户端发送的数据获取请求，所述数据获取请求中包括待获取数据的信息；
验证单元21，用于获取本地储存与所述获取接收单元20接收的数据获取请求包括的待获取数据的信息对应的数据的第三验证码，其中，所述本地储存的数据是 根据主用服务器发送的同步请求储存的；获取所述主用服务器储存的与所述待获取数据的信息对应的数据的第四验证码；如果所述第三验证码与第四验证码不一致，或所述第三验证码为空且所述第四验证码不为空，通知验证处理单元22响应所述客户单；
验证处理单元22，用于从所述主用服务器获取与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据。
所述验证处理单元22，还用于如果所述验证单21确定第四验证码为空，且所述第三验证码不为空，删除本地储存的与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据不存在的信息。
可见，本实施例中，客户端可以主动向本实施例的数据同步获取装置获取已经同步在本地的数据，这样数据同步获取装置的验证单元21就会先验证备用服务器储存的数据与主用服务器储存的数据是否一致，如果不一致，则从主用服务器获取了数据返回给客户端。这样数据同步获取装置可以根据客户端的请求验证与主用服务器上储存的数据是否一致，进而可以采取一定的手段保证数据同步获取装置和主用服务器上储存的数据的一致性。
参考图12所示，在一个具体的实施例中，数据同步装置除了可以包括如图11所示的结构外，还可以包括第二记录单元23和第二重做单元24，其中：
第二记录单元23，用于如果所述验证单元21验证第三验证码与第四验证码不一致，或所述第三验证码为空且所述第四验证码不为空，记录第二类工作日志，所述第二类工作日志中包括与所述待获取数据的信息对应的数据同步失败的信息。
第二重做单元24，用于读取所述第二记录单元23记录第二类工作日志，从所述主用服务器获取与所述第二类工作日志对应的数据进行储存。从而保证数据同步获取装置和主用服务器上储存的数据的一致性。
本发明设备实施例三
还提供一种服务器，其结构示意图如图13所示，该服务器可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(central processing units，CPU)30(例如，一个或一个以上处理器)和存储器31，一个或一个以上存储应用程序321或数据322的存储介质32(例如一个或一个以上海量存储设备)。其中，存储器31和存储介质32可以是短暂存储或永久存储。存储在存储介质32的程序可 以包括一个或一个以上模块(图示没标出)，每个模块可以包括对数据同步系统中的一系列指令操作。更进一步地，中央处理器30可以设置为与存储介质32通信，在数据同步系统上执行存储介质32中的一系列指令操作。
服务器还可以包括一个或一个以上电源33，一个或一个以上有线或无线网络接口34，一个或一个以上输入输出接口35，和/或，一个或一个以上操作系统323，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
上述方法实施例中所述的由主用服务器或备用服务器所执行的步骤可以基于该图13所示的服务器的结构。例如，该存储器31或存储介质32中存储机器可读存储执行，该中央处理器30执行该机器可读存储执行用于执行图2、图3、图4、图6～图8所示的方法以及图9～图12所示的装置的操作。
本发明实施例还提供一种数据同步系统，包括主用服务器和备用服务器，其中主用服务器的结构可以如上述设备实施例一所述的数据同步装置，或如上述设备实施例三所述的服务器的结构；所述备用服务器，用于接收到对所述数据的同步请求，储存所述数据。
进一步地，备用服务器的结构可以如上述设备实施例二所述的数据同步获取装置，或如上述设备实施例三所述的服务器的结构，在此不进行赘述。
本申请实施例提供一种非易失性存储介质，用于存储机器可读指令，当所述机器可读指令被执行时，执行上述方法。
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序可以存储于一计算机可读存储介质中，存储介质可以包括：只读存储器(ROM)、随机存取存储器RAM)、磁盘或光盘等。
以上对本发明实施例所提供的数据同步方法、系统、同步获取方法及相关装置进行了详细介绍，本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的一般技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本发明的限制。

Claims (23)

  1. 一种数据同步方法，其特征在于，应用于主用服务器，所述方法包括：
    接收客户端发送的上传请求，所述上传请求中包括待上传的数据；
    在成功获取所述数据后，响应所述客户端的上传请求；
    存储所述获取的数据；
    当所述数据不大于预置的值时，向备用服务器发起同步请求，以请求将所述数据同步到所述备用服务器，以使所述备用服务器储存所述数据；
    当所述数据大于所述预置的值时，生成并存储第二类工作日志，所述第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的数据。
  2. 如权利要求1所述的方法，其特征在于，所述方法还包括：
    如果未将所述数据成功同步到所述备用服务器，生成并存储所述第二类工作日志。
  3. 如权利要求1或2所述的方法，其特征在于，所述方法还包括：
    从所述第二类工作日志中读取所述指示信息；
    根据所述指示信息从本地读取所述未同步到所述备用服务器的所述数据，将所述数据发送给所述备用服务器，以使所述备用服务器储存所述数据。
  4. 如权利要求1所述的方法，其特征在于，所述方法还包括：
    在将所述数据成功同步到所述备用服务器后，生成并存储第一类工作日志，所述第一类工作日志中包括：所述数据的信息。
  5. 如权利要求4所述的方法，其特征在于，所述方法还包括：
    读取多条所述第一类工作日志；
    针对每条所述第一类工作日志，从所述第一类工作日志中读取所述数据的信息，根据所述数据的信息从本地获取第一数据，及所述第一数据的第一验证码；
    根据所述数据的信息从所述备用服务器获取第二数据，及所述第二数据的第二验证码；
    如果所述第一验证码与第二验证码不一致，将所述第一数据发送给所述备用服务器，以使所述备用服务器用所述第一数据替换所述第二数据。
  6. 如权利要求4所述的方法，其特征在于，所述方法还包括：
    将所述第一类工作日志和第二类工作日志发送给日志备份机器，以使所述日志备份机器根据所述第一类工作日志确定主用服务器和备用服务器储存的数据不一致时，将本地储存的所述不一致的数据发送给所述备用服务器进行储存；或根据所述第二类工作日志将未同步到所述备用服务器的所述数据发送给所述备用服务器进行储存。
  7. 一种数据同步获取方法，其特征在于，应用于备用服务器，所述方法包括：
    接收客户端发送的数据获取请求，所述数据获取请求中包括待获取数据的信息；
    获取本地储存与所述待获取数据的信息对应的数据的第三验证码；其中，所述本地储存的数据是根据主用服务器发送的同步请求储存的；
    获取所述主用服务器储存的与所述待获取数据的信息对应的数据的第四验证码；
    如果所述第三验证码与第四验证码不一致，或所述第三验证码为空且所述第四验证码不为空，从所述主用服务器获取与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据。
  8. 如权利要求7所述的方法，其特征在于，所述方法还包括：
    记录第二类工作日志，所述第二类工作日志包括与所述待获取数据的信息对应的数据同步失败的信息。
  9. 如权利要求8所述的方法，其特征在于，所述方法还包括：
    读取所述第二类工作日志，从所述主用服务器获取与所述第二类工作日志对应的数据进行储存。
  10. 如权利要求7至9任一项所述的方法，其特征在于，所述方法还包括：如果所述第四验证码为空，且所述第三验证码不为空，删除本地储存的与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据不存在的信息。
  11. 一种数据同步装置，其特征在于，包括：
    上传接收单元，用于接收客户端发送的上传请求，所述上传请求中包括待上传的数据；
    请求处理单元，用于在成功获取所述数据后，响应所述客户端的上传请求，存储所述获取的数据；
    同步单元，用于当所述数据的不大于预置的值时，向备用服务器发起同步请求，以请求将所述数据同步到所述备用服务器，以使所述备用服务器储存所述数据；
    第一记录单元，用于当所述数据的大小大于所述预置的值时，生成并存储第二类工作日志，所述第二类工作日志中包括：指示信息，用于指示未同步到所述备用服务器的所述数据。
  12. 如权利要求11所述的装置，其特征在于，
    所述第一记录单元，还用于如果未将所述数据成功同步到所述备用服务器，生成并存储所述第二类工作日志。
  13. 如权利要求11或12所述的装置，其特征在于，进一步包括：
    第一重做单元，用于从所述第二类工作日志中读取所述指示信息；根据所述指示信息从本地读取所述未同步到所述备用服务器的所述数据，将所述数据发送给所述备用服务器，以使所述备用服务器储存所述数据。
  14. 如权利要求13所述的装置，其特征在于，还包括：
    第一记录单元，用于在将所述数据成功同步到所述备用服务器后，生成并存储第一类工作日志，所述第一类工作日志中包括：所述数据的信息。
  15. 如权利要求14所述的装置，其特征在于，还包括：对账单元和第一重做单元；
    所述对账单元，用于读取多条所述第一类工作日志，针对每条所述第一类工作日志，从所述第一类工作日志中读取所述数据的信息，根据所述数据的信息从本地获取第一数据，及所述第一数据的第一验证码，根据所述数据的信息从所述备用服务器获取第二数据，及所述第二数据的第二验证码，如果所述第一验证码与第二验证码不一致，指令所述第一重做单元将所述第一数据发送给所述备用服务器；
    所述第一重做单元，用于将所述第一数据发送给所述备用服务器，以使所述备用服务器用所述第一数据替换所述第二数据。
  16. 如权利要求14所述的装置，其特征在于，还包括：
    日志备份单元，用于将所述第一类工作日志和第二类工作日志发送给日志备份机器，以使所述日志备份机器根据所述第一类工作日志确定主用服务器和备用服务器储存的数据不一致时，将本地储存的所述不一致的数据发送给所述备用服务器进行储存；或根据所述第二类工作日志将未同步到所述备用服务器的数据发送给所述 备用服务器进行储存。
  17. 一种数据同步获取装置，其特征在于，包括：
    获取接收单元，用于接收客户端发送的数据获取请求，所述数据获取请求中包括待获取数据的信息；
    验证单元，用于获取本地储存与所述待获取数据的信息对应的数据的第三验证码，其中，所述本地储存的数据是根据主用服务器发送的同步请求储存的；获取所述主用服务器储存的与所述待获取数据的信息对应的数据的第四验证码；如果所述第三验证码与第四验证码不一致，或所述第三验证码为空且所述第四验证码不为空，通知验证处理单元响应所述客户单；
    验证处理单元，用于从所述主用服务器获取与所述待获取数据的信息对应的数据，并向所述客户端响应与所述待获取数据的信息对应的数据。
  18. 如权利要求17所述的装置，其特征在于，还包括：
    第二记录单元，用于记录第二类工作日志，所述第二类工作日志中包括与所述待获取数据的信息对应的数据同步失败的信息。
  19. 如权利要求18所述的装置，其特征在于，还包括：
    第二重做单元，用于读取所述第二类工作日志，从所述主用服务器获取与所述第二类工作日志对应的数据进行储存。
  20. 如权利要求17至19任一项所述的装置，其特征在于，
    所述验证处理单元，还用于如果所述第四验证码为空，且所述第三验证码不为空，删除本地储存的与所述待获取数据的信息对应的上传数据，并向所述客户端响应与所述待获取数据的信息对应的上传数据不存在的信息。
  21. 一种数据同步系统，其特征在于，包括主用服务器和备用服务器，其中，所述主用服务器是如权利要求11至16任一项所述的数据同步装置；
    所述备用服务器，用于接收到对所述同步请求，储存所述数据。
  22. 如权利要求21所述的系统，其特征在于，所述备用服务器包括权利要求17至20任一项所述的数据同步获取装置。
  23. 一种非易失性存储介质，用于存储机器可读指令，当所述机器可读指令被执行时，执行所述权利要求1至10任一项所述的方法。
PCT/CN2017/099687 2016-09-13 2017-08-30 数据同步方法、系统、同步获取方法及装置 WO2018049983A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/040,692 US10895996B2 (en) 2016-09-13 2018-07-20 Data synchronization method, system, and apparatus using a work log for synchronizing data greater than a threshold value

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610821980.0 2016-09-13
CN201610821980.0A CN106302806B (zh) 2016-09-13 2016-09-13 一种数据同步方法、系统、同步获取方法及相关装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/040,692 Continuation US10895996B2 (en) 2016-09-13 2018-07-20 Data synchronization method, system, and apparatus using a work log for synchronizing data greater than a threshold value

Publications (1)

Publication Number Publication Date
WO2018049983A1 true WO2018049983A1 (zh) 2018-03-22

Family

ID=57710597

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/099687 WO2018049983A1 (zh) 2016-09-13 2017-08-30 数据同步方法、系统、同步获取方法及装置

Country Status (3)

Country Link
US (1) US10895996B2 (zh)
CN (1) CN106302806B (zh)
WO (1) WO2018049983A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112380292A (zh) * 2020-12-07 2021-02-19 厦门熵基科技有限公司 一种生物模板数据同步方法、装置、设备和存储介质

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106302806B (zh) 2016-09-13 2019-10-25 腾讯科技（深圳）有限公司 一种数据同步方法、系统、同步获取方法及相关装置
CN107679058B (zh) * 2017-06-26 2021-04-06 平安科技（深圳）有限公司 数据同步方法、装置及计算机可读存储介质
CN107453953B (zh) * 2017-08-30 2020-09-15 苏州朗动网络科技有限公司 一种基于httpclient的数据更新系统及方法
CN107864209B (zh) * 2017-11-17 2021-05-18 北京联想超融合科技有限公司 数据写入的方法、装置及服务器
CN107703901B (zh) * 2017-11-21 2023-12-19 丹东华通测控有限公司 一种旁路工控信息安全工业控制系统
CN111858744B (zh) * 2019-04-29 2024-07-09 北京嘀嘀无限科技发展有限公司 数据库数据同步方法、服务器及系统
CN110287273A (zh) * 2019-07-03 2019-09-27 成都路行通信息技术有限公司 基于监听MySQL数据库binlog变化实现数据同步的方法和系统
CN110442643A (zh) * 2019-07-05 2019-11-12 苏州浪潮智能科技有限公司 一种用于双机热备的文件同步方法及装置
CN110572445B (zh) * 2019-08-23 2022-05-17 深圳华工能源技术有限公司 配用电数据采集存储空间管理方法、终端及存储介质
CN111078418B (zh) * 2019-12-23 2024-04-16 北京奇艺世纪科技有限公司 操作同步方法、装置、电子设备及计算机可读存储介质
CN111949625B (zh) * 2020-08-13 2022-03-04 广东省林业科学研究院 一种快检手机端用并行数据同步上传系统
US11409865B1 (en) * 2021-08-16 2022-08-09 Cyberark Software Ltd. Verification code injection at build time
CN114979181B (zh) * 2022-05-25 2023-12-26 南京国电南自维美德自动化有限公司 一种计划值曲线同步方法、装置及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120066293A1 (en) * 2010-09-09 2012-03-15 Fuji Xerox Co., Ltd. Data relay system, relay device, and computer readable medium
CN102801699A (zh) * 2011-12-28 2012-11-28 北京安天电子设备有限公司 防止服务器数据篡改的系统、方法及设备
CN105407117A (zh) * 2014-09-10 2016-03-16 腾讯科技（深圳）有限公司 分布式备份数据的方法、装置和系统
CN105610953A (zh) * 2015-12-31 2016-05-25 深圳市优户科技有限公司 一种分布式数据同步系统及方法
CN106302806A (zh) * 2016-09-13 2017-01-04 腾讯科技（深圳）有限公司 一种数据同步方法、系统、同步获取方法及相关装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150199414A1 (en) * 2005-12-30 2015-07-16 David E. Braginsky Locally cached file system
US10911380B2 (en) * 2014-11-07 2021-02-02 Quest Software Inc. Automated large attachment processing during migration

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120066293A1 (en) * 2010-09-09 2012-03-15 Fuji Xerox Co., Ltd. Data relay system, relay device, and computer readable medium
CN102801699A (zh) * 2011-12-28 2012-11-28 北京安天电子设备有限公司 防止服务器数据篡改的系统、方法及设备
CN105407117A (zh) * 2014-09-10 2016-03-16 腾讯科技（深圳）有限公司 分布式备份数据的方法、装置和系统
CN105610953A (zh) * 2015-12-31 2016-05-25 深圳市优户科技有限公司 一种分布式数据同步系统及方法
CN106302806A (zh) * 2016-09-13 2017-01-04 腾讯科技（深圳）有限公司 一种数据同步方法、系统、同步获取方法及相关装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112380292A (zh) * 2020-12-07 2021-02-19 厦门熵基科技有限公司 一种生物模板数据同步方法、装置、设备和存储介质
CN112380292B (zh) * 2020-12-07 2023-11-17 厦门熵基科技有限公司 一种生物模板数据同步方法、装置、设备和存储介质

Also Published As

Publication number Publication date
CN106302806B (zh) 2019-10-25
US20180329630A1 (en) 2018-11-15
CN106302806A (zh) 2017-01-04
US10895996B2 (en) 2021-01-19

Similar Documents

Publication Publication Date Title
WO2018049983A1 (zh) 数据同步方法、系统、同步获取方法及装置
WO2019154394A1 (zh) 分布式数据库集群系统、数据同步方法及存储介质
RU2714098C1 (ru) Способ и устройство обработки данных
US9934242B2 (en) Replication of data between mirrored data sites
US20190129976A1 (en) Apparatus for controlling synchronization of metadata on network and method for the same
CN106776130B (zh) 一种日志恢复方法、存储装置和存储节点
WO2014059804A1 (zh) 数据同步方法及系统
US11709743B2 (en) Methods and systems for a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system
WO2017152736A1 (zh) 一种分布式文件系统hdfs的管理方法、装置及系统
US20200278911A1 (en) Synchronization storage solution after an offline event
TW201741901A (zh) 資料遷移方法和裝置
US20120278429A1 (en) Cluster system, synchronization controlling method, server, and synchronization controlling program
WO2019057081A1 (zh) 数据存储方法、数据查询方法、计算机设备及存储介质
JP4801196B2 (ja) ウェブファームへのデータ配布における二相コミットのための方法および装置
CN108228581B (zh) Zookeeper兼容通信方法、服务器及系统
CN109672752A (zh) 数据同步的方法及节点
US11372727B2 (en) Method, device, and computer-readable storage medium for managing storage system
CN108762982B (zh) 一种数据库恢复方法、装置及系统
US20240054054A1 (en) Data Backup Method and System, and Related Device
US8977897B2 (en) Computer-readable recording medium, data management method, and storage device
CN116389233B (zh) 容器云管理平台主备切换系统、方法、装置和计算机设备
CN115114112A (zh) 数据同步方法及装置
CN109542519A (zh) 一种自动分布式系统重建方法
WO2016000137A1 (zh) 一种多站点自动更新方法、客户端和系统
CN107239505B (zh) 集群镜像同步方法和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17850185

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 06/08/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 17850185

Country of ref document: EP

Kind code of ref document: A1