WO2020238748A1 - 数据同步的处理方法、装置、电子设备及计算机存储介质 - Google Patents

数据同步的处理方法、装置、电子设备及计算机存储介质 Download PDF

Info

Publication number
WO2020238748A1
WO2020238748A1 PCT/CN2020/091523 CN2020091523W WO2020238748A1 WO 2020238748 A1 WO2020238748 A1 WO 2020238748A1 CN 2020091523 W CN2020091523 W CN 2020091523W WO 2020238748 A1 WO2020238748 A1 WO 2020238748A1
Authority
WO
WIPO (PCT)
Prior art keywords
dirty page
node
dirty
information
read
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/091523
Other languages
English (en)
French (fr)
Inventor
贺广南
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2020238748A1 publication Critical patent/WO2020238748A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Definitions

  • the present invention relates to the field of logistics technology, in particular to a data synchronization processing method, device, electronic equipment and computer storage medium.
  • MVCC Multi-Version Concurrency Control
  • FIG. 1 shows a schematic structural diagram of a distributed system.
  • This distributed system includes a read-write node (RW NODE), a read-only node (RO NODE), and a storage node (STORAGE NODE).
  • RW NODE read-write node
  • RO NODE read-only node
  • STORAGE NODE storage node
  • embodiments of the present invention provide a dirty page processing solution to solve some or all of the above-mentioned problems.
  • a data synchronization processing method which includes: acquiring dirty page refresh information from a first node, wherein the dirty page refresh information is executed at least by the first node A dirty page is generated after storage; according to the dirty page refresh information, the dirty page indicated by the dirty page refresh information in the cache pool is determined; the determined dirty page in the cache pool is cleaned up.
  • a data synchronization processing method which includes: storing at least one dirty page, and generating dirty page refresh information; sending the dirty page refresh information to a second node , So that the second node cleans up dirty pages in the second node according to the dirty page refresh information.
  • a processing device for data synchronization which includes: a first acquisition module configured to acquire dirty page refresh information from a first node, wherein the dirty page refresh information passes all The first node is generated after performing the storage of at least one dirty page; the first determining module is configured to determine the dirty page indicated by the dirty page refresh information in the cache pool according to the dirty page refresh information; the cleaning module uses To perform a cleanup operation on the determined dirty pages in the buffer pool.
  • a data synchronization processing device which includes: a storage and generation module, configured to store at least one dirty page in a first node and generate dirty page refresh information;
  • the information sending module is configured to send the dirty page refresh information to the second node, so that the second node performs a cleanup operation on the dirty pages in the second node according to the dirty page refresh information.
  • an electronic device including: a processor, a memory, a communication interface, and a communication bus.
  • the processor, the memory, and the communication interface complete each other through the communication bus.
  • the memory is used to store at least one executable instruction that causes the processor to perform operations corresponding to the data synchronization processing method described in the first aspect or the second aspect.
  • a computer storage medium having a computer program stored thereon, and when the program is executed by a processor, the data synchronization processing method as described in the first or second aspect is implemented.
  • the dirty page refresh information is obtained from the first node, and the dirty page indicated by the dirty page refresh information in the buffer pool is determined according to the dirty page refresh information.
  • the dirty pages in the buffer pool are cleaned up. Since the dirty page refresh information is generated after the first node performs storage on at least one dirty page, the dirty page refresh information mainly contains the information of the dirty pages that are stored.
  • other nodes other than the first node can be timely , Quickly determine the dirty pages that have been stored, and can perform centralized processing on these dirty pages, for example, clean up operations, improve processing efficiency; on the other hand, make other nodes clean up dirty pages based on dirty page refresh information to ensure
  • the dirty pages that are cleaned up are dirty pages that have been stored by the first node, which avoids that other nodes in the prior art can only synchronize the dirty pages of the first node through the WAL log, and it is impossible to determine whether the dirty pages are stored in time, which leads to cleanup.
  • the dirty page is dirty, the dirty page data is directly discarded, resulting in poor data consistency between other nodes and the first node, and the problem of MVCC in the distributed system cannot be ensured.
  • Figure 1 is a schematic diagram of the structure of a distributed system in the prior art
  • FIG. 2 is a flowchart of steps of a data synchronization processing method according to Embodiment 1 of the present invention
  • Fig. 3 is a flow chart of the steps of a data synchronization processing method according to the second embodiment of the present invention.
  • FIG. 4 is a flowchart of the steps of a data synchronization processing method according to the third embodiment of the present invention.
  • FIG. 5 is a flowchart of the steps of a data synchronization processing method according to the fourth embodiment of the present invention.
  • FIG. 6 is a structural block diagram of a data synchronization processing device according to the fifth embodiment of the present invention.
  • FIG. 7 is a structural block diagram of a data synchronization processing device according to the sixth embodiment of the present invention.
  • FIG. 8 is a structural block diagram of a data synchronization processing device according to the seventh embodiment of the present invention.
  • FIG. 9 is a structural block diagram of a data synchronization processing device according to the eighth embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of an electronic device according to the ninth embodiment of the present invention.
  • Figure 11 is a structural block diagram of one-write-multiple-read analytical database
  • Fig. 12 is a schematic diagram of steps in which a read-write node actively synchronizes dirty page refresh information
  • FIG. 13 is a schematic diagram of steps for passively synchronizing dirty page refresh information between a read and write node.
  • Step S102 Obtain dirty page refresh information from the first node.
  • the first node in the distributed system of this embodiment is a node with data reading and data writing authority.
  • the nodes other than the first node are recorded as the second node, and the second node only has the permission to read data.
  • the second node of the distributed system is used as the execution subject to describe the data synchronization processing method.
  • the second node in the distributed system only has the authority to read data in the distributed system, and the data can be written by the first node.
  • the first node in the database has the authority to read data in the distributed system and write data to the distributed system.
  • the first node and the second node can read the required data into a buffer pool and store it in a buffer pool.
  • dirty page If the data in the cache page is updated, the cache page is called a dirty page. Before the dirty page is stored, the data in the dirty page will be different from the original data.
  • dirty pages are persistently stored in external storage, that is, persistent storage of dirty pages refers to writing the data in dirty pages to the external storage of the first node. This persistent storage The operation is also called the dirty page refresh operation, and is usually performed by the first node in the distributed system.
  • dirty pages in this embodiment is described by taking persistent storage as an example.
  • the dirty pages can also be stored in the shared internal memory, and the dirty page data in the shared internal memory can be written to the external memory when needed, or the dirty pages can also be used.
  • the data is always stored in the shared internal memory.
  • the first node may generate corresponding dirty page refresh information, where the dirty page refresh information is used to indicate at least one dirty page that is persistently stored by the first node.
  • the main content of the dirty page refresh information in the embodiment of the present invention is the information of the dirty pages that are persistently stored, and does not include the first node Information about other operations performed (such as information about interactive operations of the first node, information about reading data, etc.).
  • the dirty page refresh information can be generated after one or more (two or more) dirty pages are persistently stored. Therefore, the second node can be notified of the dirty page refresh status in time. Through the dirty page refresh information, the second node can promptly and quickly determine the dirty pages to be persistently stored, and then can clean up dirty pages according to the dirty page refresh information, ensuring the timeliness and safety of dirty page cleaning.
  • the dirty page information in the dirty page refresh information can be carried in any appropriate manner.
  • the dirty page refresh information may be generated according to the identifier of the dirty page, and the identifier of the dirty page is used to indicate the dirty page that is persistently stored.
  • the dirty page refresh information can be generated according to the address of the data in the dirty page in the external memory, and the address is used to indicate the persistently stored dirty page.
  • the dirty page information is not limited to the above-listed content, and those skilled in the art can configure the dirty page information as needed, and generate adaptive dirty page refresh information based on the dirty page information.
  • Step S104 Determine the dirty page indicated by the dirty page refresh information in the second node cache pool according to the dirty page refresh information.
  • the second node can determine the dirty page in its own cache pool that corresponds to the dirty page that is persistently stored by the first node.
  • the second node can find the dirty page A2 corresponding to A1 from its cache pool based on the dirty page refresh information , Dirty page B2 corresponding to B1 and dirty page C2 corresponding to C1.
  • the same identifier can also be used to mark the corresponding dirty pages in the first node and the second node, such as dirty pages A, B, and C in the first node, and dirty pages A, B, and C in the second node. B, C.
  • Step S106 Perform a cleanup operation on the determined dirty pages in the buffer pool.
  • the second node can clean up the determined dirty page, such as discarding dirty pages
  • the data in the second node and the first node will not be inconsistent.
  • other cache pages in the cache pool of the second node and dirty pages that have not been stored may not be processed.
  • dirty page data method can be applied to the scenario of a one-write-multiple-read distributed system, but is not limited to this, and can also be applied to other scenarios of a distributed system including multiple nodes.
  • the dirty page refresh information is obtained from the first node, the dirty page indicated by the dirty page refresh information in the buffer pool is determined according to the dirty page refresh information, and the determined dirty page in the buffer pool is determined Perform cleaning operations. Since the dirty page refresh information is generated after the first node performs storage on at least one dirty page, the dirty page refresh information mainly contains the information of the dirty pages that are stored.
  • other nodes other than the first node can be timely , Quickly determine the dirty pages that have been stored, and can perform centralized processing on these dirty pages, for example, clean up operations, improve processing efficiency; on the other hand, make other nodes clean up dirty pages based on dirty page refresh information to ensure
  • the dirty pages that are cleaned up are dirty pages that have been stored by the first node, which avoids that other nodes in the prior art can only synchronize the dirty pages of the first node through the WAL log, and it is impossible to determine whether the dirty pages are stored in time, which leads to cleanup.
  • the dirty page is dirty, the dirty page data is directly discarded, resulting in poor data consistency between other nodes and the first node, and the problem of MVCC in the distributed system cannot be ensured.
  • the data synchronization processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.), and PCs.
  • FIG. 3 there is shown a flow chart of a method for processing data synchronization according to the second embodiment of the present invention.
  • the second node is a read-only node, that is, it only has the permission to read data
  • the first node is a read-write node, that is, it has the permission to read and write data.
  • Step S202 When it is determined to clean up dirty pages, generate and send a dirty page cleanup request message to the first node according to the dirty page information in the buffer pool to request dirty page refresh information.
  • available cache pages refer to cache pages that are not dirty pages in the cache pool.
  • Those skilled in the art can set appropriate judgment conditions for determining whether the cached page is insufficient according to business requirements. For example, setting the available cache page threshold, when the number of available cache pages is less than the available cache page threshold, it is determined that the available cache pages are insufficient.
  • the step S202 includes the following sub-steps:
  • Sub-step S2021 when it is determined to perform dirty page cleaning, scan dirty pages in the buffer pool.
  • the read-only node may scan dirty pages in its own buffer pool through multiple processes.
  • the scanned dirty pages record them in the flush request list (flush request list) in the shared memory of the read-only node (that is, the shared buffer, which can be shared by multiple processes of the read-only node), for subsequent follow-up
  • the information of these dirty pages is sent to the read-write node, so that some or all of the scanned dirty pages are stored persistently, so that the read-only node can safely clean up the dirty pages.
  • the use of shared memory enables convenient data interaction between multiple processes of read-only nodes, reducing communication costs and resource overhead.
  • Sub-step S2022 generate and send a dirty page cleanup request message to the first node according to the scanned dirty page information and the preset message type.
  • the receiver process of the read-only node reads the scanned dirty page information (such as the identifier of the dirty page, etc.) from the shared memory, and generates the dirty page according to it and the preset message type Clean up the request message and send the message to the first node (that is, the read-write node), for example, to the sender process of the read-write node.
  • the existing request message format is used by specializing the message type to form a message dedicated to dirty pages. Clean up the news. It not only realizes the generation and processing of proprietary messages, but also effectively utilizes existing messages, which reduces the implementation cost of dirty page cleanup request messages.
  • the dirty page cleanup request message is used to request the read-write node to perform storage (for example, persistent storage) on the dirty page indicated by the request message.
  • storage for example, persistent storage
  • other processes of the read-only node may generate and send the dirty page cleanup request message, which is not limited in this embodiment.
  • the dirty page cleanup request message includes a message header and a message body, where the message header contains information indicating the message type of the message, which is generated according to a preset message type, for example, a dirty page cleanup request message
  • the message type is'f', which is used to indicate that the message is related to dirty page cleaning.
  • the message body contains the message content, which is generated according to the information of the scanned dirty page. For example, the message content in the dirty page cleanup request message is the identification of the scanned dirty page.
  • the read-only node sends the dirty page cleanup request information so that the read-write node can refresh the dirty page in a passive manner.
  • this step is an optional step. In another feasible way, this step may not be executed.
  • the read-only node directly executes step S204 to realize that after the read-write node refreshes dirty pages in an active manner, only The read node obtains information from the read and write node and cleans up dirty pages.
  • Step S204 Obtain dirty page refresh information from the first node.
  • the step S204 includes: receiving a synchronization message from the first node, and if the message type of the synchronization message is the message type corresponding to the dirty page refresh information, obtaining from the synchronization message The dirty page refresh information.
  • the identifier'f' is used to indicate the message type corresponding to the dirty page refresh information. If the identifier of the message type of the synchronization message is'f', a corresponding analysis or extraction method is adopted according to the message type to obtain the dirty page refresh information from the synchronization message.
  • the dirty page refresh message is used to transfer the stored dirty page information. It sets a new message type and uses the existing message format to form a synchronization message dedicated to transmitting dirty page refresh information. It not only realizes the generation and processing of proprietary messages, but also effectively utilizes existing messages, reducing the transmission cost of dirty page refresh information.
  • the dirty page flushing information can be that the read-write node responds to the dirty page cleanup request message of the read-only node, and the dirty page cleanup request message indicates After one or more dirty pages are stored (for example, persistent storage), information is generated based on the stored dirty page information. For example, according to the identifier of the stored dirty page, the dirty page refresh information is generated.
  • the dirty page refresh information can be the dirty page information generated by the read-write node after the dirty page is actively stored (for example, persistent storage).
  • the page refresh information is actively sent to the read-only node.
  • Step S206 Determine the dirty page indicated by the dirty page refresh information in the buffer pool according to the dirty page refresh information.
  • step S104 in the first embodiment may be used to determine the dirty page indicated by the dirty page refresh information in the buffer pool, so the details are not described herein again.
  • Step S208 Perform a cleanup operation on the determined dirty pages in the buffer pool.
  • step S106 in the first embodiment the implementation described in step S106 in the first embodiment can be adopted; or, by clearing the dirty page mark of the determined dirty page in the buffer pool to indicate that the page is in a reusable state To implement the cleaning operation on the determined dirty pages in the buffer pool; or implement it in any other appropriate manner.
  • Step S210 Obtain the indication information of the log file from the first node.
  • this step is an optional step, which can be executed at any time during the execution of steps S202-S208.
  • the log file may be any appropriate type of file, as long as it can record the historical operation information of the read-write node within a set time period.
  • the log file may be a WAL (Write Ahead Log) log file.
  • historical operations of read and write nodes may also include interactive operations of read and write nodes, and so on.
  • the read-only node can obtain the log file of the read-write node, and perform replay operations according to the log file, thereby obtaining the specific content information of the dirty page synchronization of the read-write node, and update the specific dirty page in the cache of the read-only node based on this information data.
  • the log file indication information is sent to the read-only node, such as the address information, pointer information, and identification information of the log file.
  • the read-only node can obtain the log file of the read-write node according to the instruction information of the log file. For example, if the read-write node persistently stores the log file in a shared storage (such as a shared storage disk), it can obtain the complete content of the log file according to the indication information of the log file, such as the storage address of the log file in the shared storage.
  • the data transmitted between the read-only node and the read-write node is small, and the read-only node can quickly obtain the log file according to the instruction information, which not only realizes the persistent storage of the log file, but also realizes the read-only node and read-write Synchronization of log files between nodes.
  • Step S212 Obtain the log file according to the instruction information.
  • step S212 includes: obtaining from the shared storage of the read-write node and the read-only node the information in the storage address indicated by the pointer information The log file.
  • the read-write node and the read-only node use shared storage, the read-write node and the read-only node use the same file system and share the storage disk, which makes the communication between the read-write node and the read-only node more efficient, and solves the problem in the prior art ,
  • the additional storage node not only increases hardware resources, but also makes the communication network speed a bottleneck of communication efficiency.
  • the dirty page refresh information is obtained from the first node, the dirty page indicated by the dirty page refresh information in the buffer pool is determined according to the dirty page refresh information, and the determined dirty page in the buffer pool is determined Perform cleaning operations. Since the dirty page refresh information is generated after the first node performs storage on at least one dirty page, the dirty page refresh information mainly contains the information of the dirty pages that are stored.
  • other nodes other than the first node can be timely , Quickly determine the dirty pages that have been stored, and can perform centralized processing on these dirty pages, for example, clean up operations, improve processing efficiency; on the other hand, make other nodes clean up dirty pages based on dirty page refresh information to ensure
  • the dirty pages that are cleaned up are dirty pages that have been stored by the first node, which avoids that other nodes in the prior art can only synchronize the dirty pages of the first node through the WAL log, and it is impossible to determine whether the dirty pages are stored in time, which leads to cleanup.
  • the dirty page is dirty, the dirty page data is directly discarded, resulting in poor data consistency between other nodes and the first node, and the problem of MVCC in the distributed system cannot be ensured.
  • the read-write node When making the read-write node perform dirty page refresh (that is, persistent storage of dirty pages), the read-write node can refresh the dirty pages in an active way, that is, the read-write node actively scans dirty pages and persists the scanned dirty pages Dirty pages can also be refreshed in a passive manner, that is, read-write nodes perform persistent storage on dirty pages according to the dirty page cleanup request message sent by read-only nodes.
  • Users can configure and use a certain dirty page refreshing method or two dirty page refreshing methods according to their needs, making dirty page refreshing more flexible and diverse, refreshing opportunities are more abundant, and can guarantee read-only nodes and read-write nodes in various situations Data consistency between.
  • Dirty page synchronization is performed between read-only nodes and read-write nodes through dirty page refresh information, so that read-only nodes can clean up dirty pages based on dirty page refresh information in a timely manner, which improves the security and timeliness of dirty page cleanup for read-only nodes.
  • the read-write node and read-only node use shared storage.
  • the read-write node sends log files to the read-only node, it only needs to send the log file instructions to the read-only node, and the read-only node can get the log from the shared storage.
  • File, and realize dirty page data synchronization improve communication efficiency, reduce data transmission, and do not need to increase additional hardware resources, do not need to make in-depth transformation of the distributed system kernel, and have better applicability.
  • the data synchronization processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.), and PCs.
  • FIG. 4 there is shown a flow chart of a method for processing data synchronization according to the third embodiment of the present invention.
  • Step S302 Store at least one dirty page, and generate dirty page refresh information.
  • the first node of the distributed system is used as the execution subject to describe the data synchronization processing method.
  • the first node has the right to read data and the right to write data
  • the distributed system also includes a second node, and the second node only has the right to read data.
  • the first node can actively store at least one dirty page in its cache pool (such as persistently stored in an external memory, or stored in an internal memory).
  • the data of the dirty page is written to the first node's Take the external memory as an example.
  • the first node may also passively perform persistent storage on at least one dirty page in its buffer pool according to the dirty page cleanup request message sent by the second node.
  • the first node can generate dirty page refresh information according to the information of the persistently stored dirty pages. For example, according to the dirty page identifier (key value) in the dirty page information, dirty page refresh information is generated.
  • the dirty page refresh information is different from the existing log file.
  • the main content is the persistently stored dirty page information, and does not include information about other operations performed by the first node (such as the interaction of the first node). Information about operations, information about reading data, etc.).
  • the dirty page refresh information can be generated after one or more (two or more) dirty pages are persistently stored.
  • the second node can be informed of the dirty page refresh status in time, so that the second node can timely and quickly determine the dirty page that is executed persistent storage, and then can perform the dirty page according to the dirty page refresh information Clean up, to ensure the timeliness and safety of dirty page cleaning.
  • Step S304 Send the dirty page refresh information to the second node, so that the second node cleans up the dirty pages in the second node according to the dirty page refresh information.
  • the first node sends the dirty page refresh information to the second node, so that the second node can determine the persistently stored dirty pages, and then can clean up the persistently stored dirty pages during the dirty page cleaning operation.
  • the second node can clean up dirty pages, improve the reuse rate of cached pages, and can fully ensure the data consistency between the second node and the first node.
  • the first node after the first node stores at least one of the dirty pages, it generates dirty page refresh information, and sends the dirty page refresh information to the second node, so that the second node will The page refresh information cleans up the dirty pages in the second node.
  • the dirty page refresh information mainly indicates the dirty pages to be stored, so that the second node can complete the dirty page storage after the first node Determine the stored dirty pages in a timely and fast manner, and can process dirty pages in a centralized manner based on the dirty page refresh information, such as clean up dirty pages, and ensure the safety of dirty page cleanup, and ensure that the second node and the first node Data consistency.
  • the data synchronization processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.), and PCs.
  • FIG. 5 there is shown a flow chart of a method for processing data synchronization according to the fourth embodiment of the present invention.
  • the data synchronization processing method is still described from the perspective of the first node of the distributed system.
  • the first node is a read-write node in a distributed system
  • the second node is a read-only node in a distributed system.
  • Step S402 Store at least one dirty page, and generate dirty page refresh information.
  • step S402 when the read-write node actively stores the dirty page (eg, persistent storage), step S402 includes: scanning the buffer pool to determine at least one dirty page; and determining the determined at least one dirty page The data in is stored in the shared storage; the dirty page refresh information is generated according to the stored dirty page information.
  • the information of the dirty page includes but is not limited to: the identifier of the dirty page, etc.
  • the writer process in the read-write node periodically scans the buffer pool of the read-write node to determine at least one dirty page in the read-write node, and the determined at least one dirty page
  • the data in is stored in shared storage.
  • the default number of dirty pages scanned by the writing process at one time is 100 (that is, when 100 dirty pages are scanned from the buffer pool, it is determined to complete a scan), and the maximum number of scans in a single scan is 1 of the total number of cached pages in the buffer pool /4 (a quarter).
  • those skilled in the art can configure other default dirty page data for one scan and the maximum number of scans for a single scan as needed.
  • the write process After the write process stores the data in the dirty page in the shared storage shared by the read-write node and the read-only node, the information of the stored dirty page can be written into the shared memory of the read-write node itself, and the sending process is waked up (sender process), the sending process obtains the information of the written dirty page from the shared memory, and generates the dirty page refresh information.
  • the dirty page refresh information is generated, and so on.
  • the read-write node can use multiple process modes to store dirty pages, which can improve work efficiency, make full use of hardware resources, and use shared memory to implement data transmission between multiple processes, ensuring communication efficiency.
  • the step S402 includes: receiving a dirty page cleanup request message from the second node, and clean up according to the dirty page
  • the request message determines at least one dirty page indicated by the dirty page cleanup request message in the buffer pool; stores the determined at least one dirty page, and generates the dirty page refresh information.
  • the dirty page cleanup request message is as described in step S202 in the second embodiment, and will not be repeated here.
  • the read-write node receives the dirty page cleanup request message from the read-only node through the sending process, and determines the dirty page cleanup request in the cache pool of the read-write node according to the dirty page identifier contained in the dirty page cleanup request message At least one dirty page indicated by the message. After the read-write node determines at least one dirty page, it can directly store it, and generate dirty page refresh information according to the stored dirty page information.
  • the method of generating the dirty page refresh information can be the same as the method in the first case described above, and will not be repeated here.
  • the dirty page cleanup request message when determining at least one dirty page indicated by the dirty page cleanup request message in the cache pool of the read-write node according to the dirty page cleanup request message, if the dirty page cleanup request message indicates that there is in the cache pool of the read-write node If the dirty page is not included, the dirty page that is not included can be directly determined as the stored dirty page.
  • the dirty page cleanup request message includes dirty pages A, B, and C
  • the buffer pool of the read-write node contains the corresponding dirty pages A and B
  • at least one dirty page indicated by the determined dirty page cleanup request message is a dirty page A and B
  • the dirty page C is directly determined as the stored dirty page.
  • dirty page refresh information it is generated based on the information of dirty pages A, B, and C.
  • Step S404 Send the dirty page refresh information to the second node, so that the second node cleans up the dirty pages in the second node according to the dirty page refresh information.
  • the sending the dirty page refresh information to the read-only node includes the following sub-steps:
  • Sub-step S4041 Generate a synchronization message according to the dirty page refresh information and the preset message type.
  • the read-only node can clean up dirty pages according to the dirty page refresh information.
  • the synchronization message adopts the message type (such as'f') and uses the existing message format to form a synchronization message dedicated to the transmission of dirty page refresh information, which not only realizes the generation and processing of proprietary messages, but also effectively utilizes existing messages , Reduce the transmission cost of dirty page refresh information.
  • "f" is only an exemplary description. In practical applications, those skilled in the art can use other appropriate specific forms to indicate the message type, which is different from the existing message type.
  • the synchronization message includes a message header and a message body.
  • the message header includes the message type of the synchronization message, and the message body includes dirty page refresh information.
  • those skilled in the art can adopt messages with other structures as needed, and configure the message type and dirty page refresh information to appropriate positions, which is not limited in this embodiment.
  • Sub-step S4042 Send the synchronization message to the second node.
  • the synchronization message can be generated by the sending process in the read-write node and sent to the read-only node. For example, to the receiving process in the read-only node. It should be noted that each sending process in the read-write node corresponds to a read-only node. When the distributed system contains multiple read-only nodes, the read-write node contains multiple sending processes.
  • the read-only node can obtain the dirty page refresh information from it.
  • the dirty page refresh information is used to transfer the dirty page information of the executed storage (such as persistent storage).
  • the log file transfers the information of the dirty pages that are executed and stored.
  • the dirty page refresh information does not include information about other operations performed by the read and write nodes. Therefore, it can be generated and sent after the dirty page is stored, so it can be timely Inform the read-only node of the refreshing status of dirty pages, so that the read-only node can quickly and quickly determine the dirty pages that are executed and stored, and then clean up the dirty pages according to the dirty page refresh information, ensuring the timeliness and safety of dirty page cleaning .
  • Step S406 Generate a log file according to the historical operation information in the set time period, and store the log file in the shared storage.
  • this step can be executed at any time during the execution of steps S402-S404.
  • the read-write node uses log files to record the historical operation information of the read-write node within a set period of time, so that it can be based on the log file in the event of an unexpected power failure in the distributed system. Restore data in distributed systems.
  • the log file may be any appropriate type of file.
  • the log file may be a WAL (Write Ahead Log) log file.
  • the historical operations recorded in the log file include but are not limited to the update operation of dirty page data. For example, modification and deletion of dirty page data.
  • the read-write node can store the generated log file (for example, persistent storage) in the shared storage. This way, on the one hand, the storage of log files can be realized and data reliability can be improved; on the other hand, it is convenient for subsequent read-only nodes to obtain the log file, and read and write the content data of dirty pages in the node according to the log file.
  • the generated log file for example, persistent storage
  • Step S408 Obtain the storage address of the log file in the shared storage, and generate indication information corresponding to the log file according to the storage address.
  • the storage address in the shared storage of the log file can be obtained, and corresponding instruction information can be generated, so that the instruction information is subsequently sent to the read-only node, so that the log file can be obtained from the shared storage according to the instruction information.
  • the indication information is pointer information indicating the storage address of the log file. Because read-only nodes and read-write nodes use shared storage, read-only nodes can obtain log files from shared storage based on pointer information, so that read-write nodes only need to send pointer information to read-only nodes to achieve log files. Sharing, compared to the existing technology that requires sending the entire log file, can reduce the amount of transmitted data and improve communication efficiency and communication reliability.
  • Step S410 Send the indication information of the log file to the second node.
  • the sending process of the read-write node can send the instruction information to the read-only node, so that the read-only node can obtain the log file from the shared storage according to the instruction information, and perform the replay operation according to the log file. Synchronize dirty page data.
  • the first node after the first node stores at least one of the dirty pages, it generates dirty page refresh information, and sends the dirty page refresh information to the second node, so that the second node will The page refresh information cleans up the dirty pages in the second node.
  • the dirty page refresh information mainly indicates the dirty pages to be stored, so that the second node can complete the dirty page storage after the first node Determine the stored dirty pages in a timely and fast manner, and can process dirty pages in a centralized manner based on the dirty page refresh information, such as clean up dirty pages, and ensure the safety of dirty page cleanup, and ensure that the second node and the first node Data consistency.
  • the data synchronization processing method is applied to a distributed system based on shared storage. Since the shared storage uses a distributed file system, it can well support the elastic expansion of the centralized storage resources of the distributed system, and can achieve efficient reading and writing. Dirty page synchronization between nodes and read-only nodes. Dirty page synchronization is more direct and efficient, does not require large-scale transformation of the database kernel, does not rely on other components and hardware resources, and ensures data consistency; it has good universality and does not need Complex network and hardware environment, without adding new components, can adapt to any distributed system with one write and many read.
  • Analytical database is a distributed system with real-time high-concurrency online analysis (Realtime OLAP, online analysis technology) cloud computing services for massive data. It allows users to conduct real-time multi-dimensional analysis perspective and business exploration for hundreds of billions of data in milliseconds.
  • the analytical database's ability to freely calculate and respond to massive amounts of data allows users to conduct flexible data exploration in an instant, quickly discover the value of data, and can be directly embedded in business systems to provide analysis services for end customers.
  • the analytical database uses shared storage and uses a one-write-multiple-read architecture.
  • shared storage means that multiple instances use the same file system and share storage disks.
  • the read-write node and the read-only node share the log file of the read-write node and the data file of the database through shared storage.
  • the read-write node is responsible for writing and updating data, and the read-only node can read the data.
  • read-only nodes need to efficiently synchronize the dirty page information of read-write nodes. Otherwise, it may happen that the data read by read-only nodes from the database is out of date data, resulting in data inconsistency and MVCC is destroyed and other issues.
  • FIG. 11 A structure of a write-multiple-read analytical database based on shared storage is shown in Figure 11.
  • read-write nodes RW nodes
  • read-only nodes RO nodes
  • DBFS distributed File system shared storage internally uses an appropriate storage system to complete data placement (that is, persistent storage).
  • the API of the distributed file system is adapted to the database kernel.
  • the entire database cluster uses the same file system to read and write the same data directory.
  • the read-write node completes WAL (Write Ahead Log), which is used in the database to ensure data consistency and failure recovery
  • WAL Write Ahead Log
  • the way) log is written to the disk, that is, it is written to the distributed file system, and the WAL log is sent to the read-only node. Only the pointer of the WAL log in the shared storage is sent.
  • the read-only node reads the corresponding pointer according to the pointer. WAL log and replay operation, so that the read-only node synchronously reads and writes the content data of at least one dirty page in the node.
  • the read-write node will also write the data in the dirty page to the shared storage according to a certain strategy to realize the placement of the dirty page data.
  • the process in which the read-write node places dirty pages into the disk can be called dirty page refresh.
  • the read-write node places the dirty page data on the disk it needs to synchronize the dirty page refresh information to the read-only node in time, so that the read-only node releases the dirty pages in its cache pool as soon as possible, so that the dirty pages become reusable cache pages.
  • FIG. 12 shows a schematic diagram of steps in which a read-write node actively synchronizes dirty page refresh information.
  • the read-write node actively drops the dirty page data to the disk, and synchronizes the dirty page refresh information with the read-only node so that it can clean up the dirty page.
  • the specific steps are:
  • Step A1 The writer process of the reading and writing node periodically scans the buffer pool of the reading and writing node. If a dirty page is scanned, it will be placed on the disk, and the dirty page that has been placed on the disk (persistent storage) The information is written to the shared memory of the read and write nodes. If no dirty pages are scanned, the writing process will take a short break. Among them, the default number of dirty pages scanned by the write process at one time is 100, and the maximum number of scanned pages is 1/4 of the total number of cached pages.
  • Step B1 When a dirty page is scanned, a sender process is awakened for processing and a synchronization message is generated according to the processing result.
  • the sending process generates dirty page refresh information according to the information of the dirty pages that have been placed in the shared memory, and generates a synchronization message according to the dirty page refresh information, and sends it to the receiver process of the corresponding read-only node.
  • each sending process corresponds to a read-only node, so when there are multiple read-only nodes, there will be multiple sending processes.
  • a new message type'f' is added to the communication mechanism, which represents a synchronization message corresponding to dirty page refresh information.
  • Step C1 After the receiving process of the read-only node receives the synchronization message, it obtains and processes the dirty page refresh information from it, and according to the instructions of the dirty page refresh information, cleans up the dirty pages in the buffer pool of the read-only node. The mark is cleared, and the mark is reusable.
  • the read-only node can safely clean up dirty pages, avoiding the problem of inconsistent data between the read-only node and the read-write node in the prior art that the read-only node clears the dirty pages that are not placed on the disk. Ensure that the data read by the read-only node from the shared storage is the latest data.
  • a read-only node when a user's data query request is received, the cache page where the data requested by the data query request is located is obtained from the cache pool of the read-only node. If the cache page required by the data query request is not in the cache pool, You need to read data from the shared storage to the cache page. In this process, there is a situation where the cache pages of read-only nodes are less available. For example, read-only nodes receive a large number of data query requests, and at the same time, read-write nodes synchronize a large amount of dirty page data with read-only nodes. There will be fewer cache pages available in the node's cache pool, which will trigger dirty page cleaning.
  • FIG. 13 shows a schematic diagram of steps in which a read-write node passively synchronizes dirty page refresh information.
  • the read-write node passively drops the dirty page data to the disk, and synchronizes the dirty page refresh information to the read-only node so that it can clean up the dirty page.
  • the specific steps are:
  • Step A2 When the client query process of the read-only node requests available cache pages, if it finds that there are no available cache pages, it needs to use the clock sweep algorithm to clean up dirty pages, and the read-only ones will be found during the cleaning process.
  • the dirty pages in the cache pool of the node are recorded in the flush request list (flush request list) in the shared memory of the read-only node.
  • Step B2 When the user query process determines that there is dirty page information in the refresh request list, the receiving process is awakened, and the receiving process generates a dirty page cleanup request message according to the dirty page information, and sends it to the sending process of the read-write node.
  • the message type of the dirty page cleanup request message is the added new message type'f' to identify that the dirty page cleanup request message is related to dirty page cleanup.
  • Step C2 After receiving the dirty page cleaning request message, the sending process of the read-write node will directly perform the dirty page placement operation.
  • dirty pages between read-only nodes and read-write nodes may not be equal.
  • the dirty page in the dirty page cleanup request message is not in the buffer pool of the read-write node. At this time, it can be assumed that the dirty page has been Place an order.
  • Step D2 The sending process of the read-write node generates a synchronization message according to the dirty page refresh information, and sends it to the receiving process of the corresponding read-only node.
  • Step E2 The receiving process of the read-only node receives the synchronization message, and cleans up dirty pages according to the dirty page refresh information therein.
  • the dirty page processing process in this usage scenario is based on physical replication technology (physical replication refers to a mechanism for data synchronization between multiple instances of a database to ensure data synchronization on physical disks), combined with a distributed file system, to achieve an efficient
  • physical replication refers to a mechanism for data synchronization between multiple instances of a database to ensure data synchronization on physical disks
  • the dirty page synchronization between the read-write node and the read-only node also supports the active synchronization dirty page mechanism of the read-write node and the passive synchronization dirty page mechanism triggered by the read-only node, which is stable and efficient.
  • the data synchronization processing method in this use scenario can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as tablet computers, mobile phones, etc.), and PCs.
  • FIG. 6 there is shown a structural block diagram of a data synchronization processing device according to the fifth embodiment of the present invention.
  • the processing device for data synchronization in this embodiment includes: a first obtaining module 502, configured to obtain dirty page refresh information from a first node, where the dirty page refresh information is executed by the first node when performing at least one dirty page Generated after storage; the first determining module 504 is configured to determine the dirty page indicated by the dirty page refresh information in the cache pool according to the dirty page refresh information; the cleaning module 506 is configured to determine the dirty page in the cache pool The dirty pages are cleaned up.
  • the dirty page refresh information is obtained from the first node, and the dirty page indicated by the dirty page refresh information in the cache pool of the second node is determined according to the dirty page refresh information, and the determined cache pool The dirty pages in the cleanup operation. Since the first node performs storage on at least one dirty page, it will generate dirty page refresh information and send it to the second node.
  • the dirty page refresh information is mainly information about the dirty pages that are stored.
  • the second node can The dirty pages that have been stored are determined in a timely and fast manner, and these dirty pages can be centrally processed, for example, cleaning operations are performed, which improves processing efficiency; on the other hand, the second node cleans up dirty pages according to the dirty page refresh information , To ensure that the cleaned dirty pages are dirty pages that have been stored by the first node, avoiding that the second node in the prior art can only synchronize the dirty pages of the first node through the WAL log, and cannot determine whether the dirty pages are stored in time, As a result, the dirty page data is directly discarded when the dirty page is cleaned up, resulting in poor data consistency between the second node and the first node, and the MVCC of the distributed system cannot be guaranteed.
  • FIG. 7 there is shown a structural block diagram of a data synchronization processing device according to the sixth embodiment of the present invention.
  • the processing device for data synchronization in this embodiment includes: a first obtaining module 602, configured to obtain dirty page refresh information from a first node, where the dirty page refresh information is executed by the first node when at least one dirty page is executed. Generated after storage; the first determining module 604 is configured to determine the dirty page indicated by the dirty page refresh information in the cache pool according to the dirty page refresh information; the cleaning module 606 is configured to determine the dirty page in the cache pool The dirty pages are cleaned up.
  • the first node is a read-write node
  • the first acquisition module 602 is configured to receive a synchronization message from the read-write node, if the message type of the synchronization message is the message type corresponding to the dirty page refresh information , The dirty page refresh information is obtained from the synchronization message.
  • the device further includes: a request sending module 608, which is configured to generate, according to the dirty page information in the buffer pool, when it is determined to perform dirty page cleaning before obtaining dirty page refresh information from the first node And send a dirty page cleanup request message to the first node to request the dirty page refresh information.
  • a request sending module 608 which is configured to generate, according to the dirty page information in the buffer pool, when it is determined to perform dirty page cleaning before obtaining dirty page refresh information from the first node And send a dirty page cleanup request message to the first node to request the dirty page refresh information.
  • the request sending module 608 includes: a scanning module 6081, which is used to scan dirty pages in the buffer pool when dirty page cleaning is determined; a request generation module 6082, which is used to scan dirty pages according to the information and The preset message type generates and sends a dirty page cleanup request message to the first node.
  • the device further includes: a second obtaining module 610, configured to obtain indication information of a log file from the first node, wherein the log file is used to record that the first node is in a set time period
  • the third acquisition module 612 is configured to acquire the log file according to the instruction information.
  • the indication information is pointer information of the log file; the third obtaining module 612 is configured to obtain the storage indicated by the pointer information from the shared storage of the first node and the second node The log file described in the address.
  • the data synchronization processing device in this embodiment is used to implement the corresponding data synchronization processing methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, and will not be repeated here.
  • FIG. 8 there is shown a structural block diagram of a data synchronization processing device according to the seventh embodiment of the present invention.
  • the data synchronization processing device of this embodiment includes: a storage and generation module 702, configured to store at least one dirty page, and generate dirty page refresh information; and an information sending module 704, configured to send the dirty page refresh information to A second node, so that the second node performs a cleaning operation on dirty pages in the second node according to the dirty page refresh information.
  • dirty page refresh information is generated, and the dirty page refresh information is sent to the second node, so that the second node can check all the dirty pages according to the dirty page refresh information.
  • the dirty pages in the second node are cleaned up. Since the first node generates the corresponding dirty page refresh information for its storage operation on dirty pages, the dirty page refresh information mainly indicates the dirty pages to be stored, so that the second node can complete the dirty page storage after the first node Determine the stored dirty pages in a timely and fast manner, and can process dirty pages in a centralized manner based on the dirty page refresh information, such as clean up dirty pages, and ensure the safety of dirty page cleanup, and ensure that the second node and the first node Data consistency.
  • FIG. 9 there is shown a structural block diagram of a data synchronization processing device according to the eighth embodiment of the present invention.
  • the data synchronization processing device of this embodiment includes: a storage and generation module 802, configured to store at least one dirty page and generate dirty page refresh information; and an information sending module 804, configured to send the dirty page refresh information to A second node, so that the second node performs a cleaning operation on dirty pages in the second node according to the dirty page refresh information.
  • the storage and generation module 802 includes: a second determination module 8021, configured to scan the buffer pool to determine at least one dirty page; a first storage module 8022, configured to determine the at least one dirty page The data in the page is stored in the shared storage; the information generation module 8023 is configured to generate the dirty page refresh information according to the stored dirty page information.
  • the second node is a read-only node
  • the information sending module 804 includes: a message generating module 8041, configured to generate a synchronization message according to the dirty page refresh information and a preset message type; a message sending module 8042, configured to send the synchronization message to the second node.
  • the storage and generation module 802 includes: a receiving module 8024, configured to receive a dirty page cleanup request message from the second node, and determine the dirty page in the buffer pool according to the dirty page cleanup request message At least one dirty page indicated by the cleaning request message; the second storage module 8025 is configured to store the determined at least one dirty page and generate the dirty page refresh information.
  • the device further includes: a log generation module 806, configured to generate a log file according to the historical operation information of the first node within a set time period, and store the log file in the shared storage .
  • a log generation module 806, configured to generate a log file according to the historical operation information of the first node within a set time period, and store the log file in the shared storage .
  • the device further includes: an instruction generating module 808, configured to obtain the storage address of the log file in the shared storage after the log file is stored in the shared storage, and according to the The storage address generates instruction information corresponding to the log file; the instruction sending module 810 is configured to send the instruction information of the log file to the second node.
  • an instruction generating module 808 configured to obtain the storage address of the log file in the shared storage after the log file is stored in the shared storage, and according to the The storage address generates instruction information corresponding to the log file
  • the instruction sending module 810 is configured to send the instruction information of the log file to the second node.
  • the indication information is pointer information indicating the storage address of the log file.
  • the data synchronization processing device in this embodiment is used to implement the corresponding data synchronization processing methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, and will not be repeated here.
  • the electronic device may include: a processor (processor) 902, a communication interface (Communications Interface) 904, a memory (memory) 906, and a communication bus 908.
  • processor processor
  • Communication interface Communication Interface
  • memory memory
  • the processor 902, the communication interface 904, and the memory 906 communicate with each other through the communication bus 908.
  • the communication interface 904 is used to communicate with other electronic devices such as terminal devices or servers.
  • the processor 902 is configured to execute the program 910, and specifically can execute the relevant steps in the above-mentioned data synchronization processing method embodiment.
  • the program 910 may include program code, and the program code includes computer operation instructions.
  • the processor 902 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.
  • the one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
  • the memory 906 is used to store the program 910.
  • the memory 906 may include a high-speed RAM memory, or may also include a non-volatile memory (non-volatile memory), for example, at least one disk memory.
  • the program 910 may specifically be used to cause the processor 902 to perform the following operations: obtain dirty page refresh information from a first node, where the dirty page refresh information is generated by the first node after storing at least one dirty page; The dirty page refresh information determines the dirty page indicated by the dirty page refresh information in the cache pool; and performs a cleaning operation on the determined dirty page in the cache pool.
  • the first node is a read-write node
  • the program 910 is further configured to enable the processor 902 to receive a synchronization message from the read-write node when acquiring dirty page refresh information from the first node. If the message type of the synchronization message is the message type corresponding to the dirty page refresh information, the dirty page refresh information is acquired from the synchronization message.
  • the program 910 is further configured to cause the processor 902 to perform dirty page cleaning according to the dirty page information in the buffer pool before obtaining dirty page refresh information from the first node. , Generating and sending a dirty page cleanup request message to the first node to request the dirty page refresh information.
  • the program 910 is further configured to cause the processor 902 to generate and send the dirty page cleaning to the first node according to the dirty pages in the buffer pool when it is determined to perform dirty page cleaning.
  • the processor 902 When requesting a message, when it is determined to clean up dirty pages, scan the dirty pages in the buffer pool; generate and send a dirty page cleanup request message to the first node according to the information of the scanned dirty pages and the preset message type.
  • the program 910 is further configured to cause the processor 902 to obtain the indication information of the log file from the first node, where the log file is used to record that the first node is at a set time. Information about historical operations in the segment; obtaining the log file according to the instruction information.
  • the program 910 is further configured to enable the processor 902 to obtain the log file according to the instruction information, if the instruction information is pointer information of the log file; In the shared storage of the first node and the second node, acquiring the log file in the storage address indicated by the pointer information.
  • the program 910 may be specifically used to cause the processor 902 to perform the following operations: store at least one dirty page, and generate dirty page refresh information; send the dirty page refresh information to the second node, so that the second node The node performs a cleanup operation on the dirty pages in the second node according to the dirty page refresh information.
  • the program 910 is further configured to cause the processor 902 to scan the buffer pool when storing at least one dirty page and generating dirty page refresh information to determine whether the read/write node is Store the determined data in the at least one dirty page in shared storage; generate the dirty page refresh information according to the information of the stored dirty page.
  • the program 910 is further configured to cause the processor 902 to generate synchronization according to the dirty page refresh information and a preset message type when sending the dirty page refresh information to the second node. Message; sending the synchronization message to the second node.
  • the program 910 is further configured to cause the processor 902 to receive a dirty page cleanup request message from the second node when storing at least one dirty page and generating dirty page refresh information, and According to the dirty page cleanup request message, determine at least one dirty page indicated by the dirty page cleanup request message in the buffer pool; store the determined at least one dirty page, and generate the dirty page refresh information.
  • the program 910 is further configured to enable the processor 902 to generate a log file according to information about historical operations within a set time period, and store the log file in the shared storage.
  • the program 910 is further configured to cause the processor 902 to obtain the storage address of the log file in the shared storage after the storage of the log file in the shared storage, and Generating indication information corresponding to the log file according to the storage address; sending the indication information of the log file to the second node.
  • the indication information is pointer information indicating the storage address of the log file.
  • the dirty page refresh information is acquired from the first node, the dirty page indicated by the dirty page refresh information in the cache pool is determined according to the dirty page refresh information, and the determined dirty page in the cache pool is determined
  • the dirty pages are cleaned up. Since the dirty page refresh information is generated after the first node performs storage on at least one dirty page, the dirty page refresh information mainly contains the information of the dirty pages that are stored.
  • other nodes other than the first node can be timely , Quickly determine the dirty pages that have been stored, and can perform centralized processing on these dirty pages, for example, clean up operations, improve processing efficiency; on the other hand, make other nodes clean up dirty pages based on dirty page refresh information to ensure
  • the dirty pages that are cleaned up are dirty pages that have been stored by the first node, which avoids that other nodes in the prior art can only synchronize the dirty pages of the first node through the WAL log, and it is impossible to determine whether the dirty pages are stored in time, which leads to cleanup.
  • the dirty page is dirty, the dirty page data is directly discarded, resulting in poor data consistency between other nodes and the first node, and the problem of MVCC in the distributed system cannot be ensured.
  • the first node after the first node stores at least one of the dirty pages, it generates dirty page refresh information, and sends the dirty page refresh information to the second node, so that the second node refreshes the information according to the dirty page Perform a cleanup operation on the dirty pages in the second node.
  • the dirty page refresh information mainly indicates the dirty pages to be stored, so that the second node can complete the dirty page storage after the first node Determine the stored dirty pages in a timely and fast manner, and can process dirty pages in a centralized manner based on the dirty page refresh information, such as clean up dirty pages, and ensure the safety of dirty page cleanup, and ensure that the second node and the first node Data consistency.
  • each component/step described in the embodiment of the present invention can be split into more components/steps, or two or more components/steps or partial operations of components/steps can be combined into New components/steps to achieve the purpose of the embodiments of the present invention.
  • the above method according to the embodiments of the present invention can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented by
  • a recording medium such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk
  • the computer code downloaded from the network is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium, so that the method described here can be stored using a general-purpose computer, a dedicated processor or a programmable Or such software processing on a recording medium of dedicated hardware (such as ASIC or FPGA).
  • a computer, processor, microprocessor controller, or programmable hardware includes storage components (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is used by the computer, When the processor or hardware accesses and executes, the data synchronization processing method described here is realized.
  • storage components for example, RAM, ROM, flash memory, etc.
  • the execution of the code converts the general-purpose computer into a dedicated computer for executing the processing method of data synchronization shown here.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

一种数据同步的处理方法、装置、电子设备及计算机存储介质。其中，所述数据同步的处理方法包括：从第一节点获取脏页刷新信息（S102），其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页（S104）；对确定的所述缓存池中的脏页进行清理操作（S106）。该方法可以有效保证多个节点间的数据一致性。

Description

数据同步的处理方法、装置、电子设备及计算机存储介质
本申请要求2019年05月31日递交的申请号为201910467267.4、发明名称为“数据同步的处理方法、装置、电子设备及计算机存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及物流技术领域，尤其涉及一种数据同步的处理方法、装置、电子设备及计算机存储介质。
背景技术
在包括多个节点的分布式系统中，如何保证多个节点间的数据一致性和整个分布式系统的多版本并发控制(MVCC，Multi-Version Concurrency Control)是其面临的关键问题。
图1中示出了一种分布式系统的结构示意图，此种分布式系统包括读写节点(RW NODE)、只读节点(RO NODE)和存储节点(STORAGE NODE)。该分布式系统中读写节点和只读节点间的脏页(数据发生变化的缓存页)同步依赖外部的存储节点完成。
在这种脏页同步过程中，当只读节点的缓存池(Buffer Pool)满时，会触发只读节点的脏页清理，进行脏页清理，此时只读节点直接将部分脏页中数据丢弃掉，并将其恢复为可用缓存页。这种脏页同步过程使得只读节点无法及时有效地获取脏页是否已被读写节点持久化存储的信息，导致只读节点从磁盘中读取该脏页数据时，其读取到可能是过时的数据，造成分布式系统的只读节点和读写节点的数据一致性差，且无法确保分布式系统的MVCC。
发明内容
有鉴于此，本发明实施例提供一种脏页处理方案，以解决上述部分或全部问题。
根据本发明实施例的第一方面，提供了一种数据同步的处理方法，其包括：从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页；对确定的所述缓存池中的脏页进行清理操作。
根据本发明实施例的第二方面，提供了一种数据同步的处理方法，其包括：对至少 一个脏页进行存储，并生成脏页刷新信息；将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
根据本发明实施例的第三方面，提供了一种数据同步的处理装置，其包括：第一获取模块，用于从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；第一确定模块，用于根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页；清理模块，用于对确定的所述缓存池中的脏页进行清理操作。
根据本发明实施例的第四方面，提供了一种数据同步的处理装置，其包括：存储与生成模块，用于对第一节点中的至少一个脏页进行存储，并生成脏页刷新信息；信息发送模块，用于将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
根据本发明实施例的第五方面，提供了一种电子设备，包括：处理器、存储器、通信接口和通信总线，所述处理器、所述存储器和所述通信接口通过所述通信总线完成相互间的通信；所述存储器用于存放至少一可执行指令，所述可执行指令使所述处理器执行如第一方面或第二方面所述的数据同步的处理方法对应的操作。
根据本发明实施例的第六方面，提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现如第一方面或第二方面所述的数据同步的处理方法。
根据本发明实施例提供的数据同步的处理方案，从第一节点获取脏页刷新信息，根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页，对确定的所述缓存池中的脏页进行清理操作。由于在第一节点对至少一个脏页执行存储后会生成脏页刷新信息，脏页刷新信息中主要是被执行存储的脏页的信息，一方面，使得第一节点之外的其他节点可以及时、快速地确定已被存储的脏页，且可以对这些脏页进行集中处理，例如，进行清理操作，提升了处理效率；另一方面，使得其他节点根据脏页刷新信息进行脏页清理，保证清理掉的脏页是已经被第一节点执行存储的脏页，避免了现有技术中其他节点只能通过WAL日志同步第一节点的脏页，不能及时确定脏页是否被存储，而导致清理脏页时直接丢弃脏页数据，造成其他节点与第一节点间数据一致性差，且无法确保分布式系统的MVCC的问题。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有 技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明实施例中记载的一些实施例，对于本领域普通技术人员来讲，还可以根据这些附图获得其他的附图。
图1为现有技术中的分布式系统的结构示意图；
图2为根据本发明实施例一的一种数据同步的处理方法的步骤流程图；
图3为根据本发明实施例二的一种数据同步的处理方法的步骤流程图；
图4为根据本发明实施例三的一种数据同步的处理方法的步骤流程图；
图5为根据本发明实施例四的一种数据同步的处理方法的步骤流程图；
图6为根据本发明实施例五的一种数据同步的处理装置的结构框图；
图7为根据本发明实施例六的一种数据同步的处理装置的结构框图；
图8为根据本发明实施例七的一种数据同步的处理装置的结构框图；
图9为根据本发明实施例八的一种数据同步的处理装置的结构框图；
图10为根据本发明实施例九的一种电子设备的结构示意图；
图11为一写多读分析型数据库的结构框图；
图12为一种读写节点主动同步脏页刷新信息的步骤示意图；
图13为一种读写节点被动同步脏页刷新信息的步骤示意图。
具体实施方式
为了使本领域的人员更好地理解本发明实施例中的技术方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本发明实施例一部分实施例，而不是全部的实施例。基于本发明实施例中的实施例，本领域普通技术人员所获得的所有其他实施例，都应当属于本发明实施例保护的范围。
下面结合本发明实施例附图进一步说明本发明实施例具体实现。
实施例一
参照图2，示出了根据本发明实施例一的一种数据同步的处理方法的步骤流程图。
本实施例的数据同步的处理方法包括以下步骤：
步骤S102：从第一节点获取脏页刷新信息。
为了便于理解，在本实施例的分布式系统中的第一节点为具有读数据和写数据权限的节点。第一节点之外的其他节点记作第二节点，第二节点只具有读数据的权限。
在本实施例中，以分布式系统的第二节点为执行主体，对数据同步的处理方法进行说明。
分布式系统中的第二节点只具有读取分布式系统中的数据的权限，所述数据可以由第一节点写入。相对应地，数据库中的第一节点具有读取分布式系统中的数据和向分布式系统中写入数据的权限。
在分布式系统中，为了提升数据处理效率，减小响应用时，第一节点和第二节点可以将所需数据读取到缓存池(buffer pool)中，通过其中的缓存页(buffer)存储。
若缓存页中的数据被更新，则该缓存页被称为脏页。在脏页被存储之前，脏页中的数据与原始数据会存在差异。通常，为了保证数据可靠性，脏页被持久化存储到外部存储器中，即脏页的持久化存储是指将脏页中的数据写入到第一节点的外部存储器中，这一持久化存储操作又称脏页刷新操作，通常是由分布式系统中的第一节点执行。
本实施例中脏页的存储以持久化存储为例进行说明。当然，在其他实施例中，脏页被存储也可以存储到共享的内部存储器中，并在需要时可以将共享的内部存储器中的脏页数据写入到外部存储器中，或者也可以使脏页数据始终保存在共享的内部存储器中。
在脏页被持久化存储后，第一节点可以生成对应的脏页刷新信息，该脏页刷新信息用于指示被所述第一节点执行持久化存储的至少一个脏页。与传统的通过日志文件传递被执行持久化存储的脏页的信息不同，本发明实施例中的脏页刷新信息中的主要内容为被持久化存储的脏页的信息，而不包括第一节点执行的其它操作的信息(如第一节点的交互操作的信息、读取数据的信息，等等)。并且，该脏页刷新信息在一个或多个(两个及两个以上)脏页进行持久化存储后即可生成，因此，可以及时告知第二节点脏页的刷新情况。通过该脏页刷新信息，第二节点能够及时、快速地确定被执行持久化存储的脏页，进而可以根据该脏页刷新信息进行脏页清理，保证脏页清理的及时性和安全性。
其中，脏页刷新信息中的脏页的信息可以通过任意适当的方式被携带。
例如，若脏页的信息中包括脏页的标识，则脏页刷新信息可以根据脏页的标识生成，利用脏页的标识指示被持久化存储的脏页。
又例如，若脏页的信息中包括脏页的数据在外部存储器的地址，则脏页刷新信息可以根据脏页中数据在外部存储器的地址生成，利用地址指示被持久化存储的脏页。
当然，脏页的信息并不限于上述列举的内容，本领域技术人员可以根据需要配置脏页的信息，并根据脏页的信息生成适应的脏页刷新信息。
步骤S104：根据所述脏页刷新信息，确定第二节点缓存池中所述脏页刷新信息所指 示的脏页。
通过获取的脏页刷新信息，第二节点可以确定其自身缓存池中，与被第一节点执行持久化存储的脏页相对应的脏页。
例如，若脏页刷新信息中携带有脏页A1、B1、和C1的信息如脏页标识，则第二节点可以根据该脏页刷新信息，从其缓存池中找到与A1对应的脏页A2、与B1对应的脏页B2和与C1对应的脏页C2。可选地，也可以采用相同的标识来标示第一节点和第二节点中对应的脏页，如第一节点中的脏页A、B、C，和，第二节点中的脏页A、B、C。
需要说明的是，针对不同的脏页刷新信息，本领域技术人员可以采用任何适当的方式确定第二节点的缓存池中脏页刷新信息所指示的脏页。
步骤S106：对确定的所述缓存池中的脏页进行清理操作。
由于确定的第二节点的缓存池中脏页刷新信息所指示的脏页是已经被第一节点执行存储的脏页，因而第二节点可以对确定的脏页进行清理操作，如丢弃掉脏页中的数据，而不会造成第二节点与第一节点间的数据不一致。而对于第二节点的缓存池中的其它缓存页和未进行存储的脏页，可以不进行处理。
需要说明的是，本实施例提供的脏页数据方法可以应用于一写多读分布式系统的场景，但不限于此，可同样适用于其他包括多个节点的分布式系统的场景。
通过本实施例，从第一节点获取脏页刷新信息，根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页，对确定的所述缓存池中的脏页进行清理操作。由于在第一节点对至少一个脏页执行存储后会生成脏页刷新信息，脏页刷新信息中主要是被执行存储的脏页的信息，一方面，使得第一节点之外的其他节点可以及时、快速地确定已被存储的脏页，且可以对这些脏页进行集中处理，例如，进行清理操作，提升了处理效率；另一方面，使得其他节点根据脏页刷新信息进行脏页清理，保证清理掉的脏页是已经被第一节点执行存储的脏页，避免了现有技术中其他节点只能通过WAL日志同步第一节点的脏页，不能及时确定脏页是否被存储，而导致清理脏页时直接丢弃脏页数据，造成其他节点与第一节点间数据一致性差，且无法确保分布式系统的MVCC的问题。
本实施例的数据同步的处理方法可以由任意适当的具有数据处理能力的电子设备执行，包括但不限于：服务器、移动终端(如平板电脑、手机等)和PC机等。
实施例二
参照图3，示出了根据本发明实施例二的一种数据同步的处理方法的步骤流程图。
本实施例仍从第二节点的角度，对本发明实施例提供的数据同步的处理方法进行说 明。其中，第二节点为只读节点，即只具有读取数据的权限，第一节点为读写节点，即具有读取数据的权限和写数据的权限。
本实施例的数据同步的处理方法包括以下步骤：
步骤S202：当确定进行脏页清理时，根据缓存池中的脏页的信息，生成并向第一节点发送脏页清理请求消息，以请求脏页刷新信息。
在只读节点中，当缓存池中可用缓存页不足时，确定需要进行脏页清理。其中，可用缓存页指缓存池中不为脏页的缓存页。
本领域技术人员可以根据业务需求设置适当的确定缓存页是否不足的判断条件。例如，设置可用缓存页阈值，当可用缓存页数量小于可用缓存页阈值时，确定为可用缓存页不足。
当确定进行脏页清理时，为了避免只读节点和读写节点间的数据不一致，需要在读写节点将脏页的数据执行存储(本实施例中，该存储以持久化存储为例)后，再清理脏页。为此，所述步骤S202包括以下子步骤：
子步骤S2021：当确定进行脏页清理时，扫描缓存池中的脏页。
在本实施例中，为了提升只读节点访问效率，只读节点可以通过多个进程扫描自身的缓存池中的脏页。
针对扫描出的脏页，将其记录到只读节点的共享内存(即共享buffer，其可以对只读节点的多个进程共享)中的刷新请求列表(flush request list)中，以备后续将这些脏页的信息发送给读写节点，使其对部分或全部扫描出的脏页进行持久化存储，从而使只读节点可以安全地清理脏页。采用共享内存使只读节点的多进程间可以方便地数据交互，降低了通信成本和资源开销。
当然，在其他实施例中，根据不同的业务需求，可以针对每个扫描出的脏页直接进行后续处理，而不将其记录到共享内存中。
子步骤S2022：根据扫描出的脏页的信息和预设的消息类型，生成并向所述第一节点发送脏页清理请求消息。
在本实施例中，由只读节点的接收进程(receiver process)从共享内存中读取扫描出的脏页的信息(如脏页的标识等)，根据其及预设的消息类型生成脏页清理请求消息，并将该消息发送给第一节点(即读写节点)，例如，发送给读写节点的发送进程(sender process)。因传统的脏页清理方案中，并没有专门用于脏页清理的消息，为此，本发明实施例中，通过专设消息类型的方式，利用现有的请求消息格式，形成专用于脏页清理 的消息。既实现了专有消息的生成和处理，又有效利用了已有消息，降低了脏页清理请求消息的实现成本。
该脏页清理请求消息用于请求读写节点对所述请求消息所指示的脏页执行存储(如，持久化存储)。在其他实施例中，可以由只读节点的其他进程生成并发送脏页清理请求消息，本实施例对此不作限定。
在一具体实现中，脏页清理请求消息包括消息头和消息体，其中，消息头中包含有指示该消息的消息类型的信息，其根据预设的消息类型生成，例如，脏页清理请求消息的消息类型为’f’，用于指示该消息与脏页清理相关。消息体中包含消息内容，其根据扫描出的脏页的信息生成，例如，脏页清理请求消息中的消息内容为扫描出的脏页的标识等。
通过上述过程，实现了脏页清理请求消息的生成和发送。只读节点通过发送脏页清理请求信息，使读写节点能够对脏页进行被动方式的脏页刷新。
但需要说明的是，本步骤为可选步骤，在另一种可行方式中，本步骤可以不被执行，只读节点直接执行步骤S204，以实现读写节点采用主动方式刷新脏页后，只读节点从读写节点获取信息并对脏页进行清理。
步骤S204：从第一节点获取脏页刷新信息。
在一具体实现中，所述步骤S204包括：从所述第一节点接收同步消息，若所述同步消息的消息类型为所述脏页刷新信息对应的消息类型，则从所述同步消息中获取所述脏页刷新信息。
本领域技术人员可以根据需要采用合适的方式指示与所述脏页刷新信息对应的消息类型，本实施例对此不作限定。例如，以标识’f’指示与所述脏页刷新信息对应的消息类型。若同步消息的消息类型的标识为’f’，则根据该消息类型采用对应的解析或提取方式从所述同步消息中获取所述脏页刷新信息。
与脏页清理请求消息类似，脏页刷新消息用于传递进行了存储的脏页的信息，其设置新的消息类型并利用现有的消息格式，形成专用于传输脏页刷新信息的同步消息，既实现了专有消息的生成和处理，又有效利用了已有消息，降低了脏页刷新信息的传输成本。
需要说明的是，在一种称为被动方式刷新脏页的方式中，该脏页刷新信息可以是读写节点响应于只读节点的脏页清理请求消息，对脏页清理请求消息中指示的一个或多个脏页执行存储(如，持久化存储)后，根据被存储的脏页的信息生成的信息。例如，根 据被存储的脏页的标识，生成脏页刷新信息。
在另一种称为主动方式刷新脏页的方式中，脏页刷新信息可以是读写节点主动对脏页进行存储(如，持久化存储)后，根据被存储的脏页的信息生成的脏页刷新信息并主动发送给只读节点。
步骤S206：根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页。
本步骤中，可以采用实施例一中步骤S104中所述的实现方式确定缓存池中所述脏页刷新信息所指示的脏页，故在此不再进行赘述。
步骤S208：对确定的所述缓存池中的脏页进行清理操作。
本步骤中，可以采用实施例一中步骤S106中所述的实现方式实现；或者，通过将确定的所述缓存池中的脏页的脏页标记清零，以指示该页为可复用状态的方式，实现对确定的所述缓存池中的脏页进行清理操作；或者采用其他任意适当的方式实现。
在对脏页进行清理时，可以采用clock sweep算法或者其他脏页清理算法进行清理。
通过上述过程，实现了只读节点与读写节点间的数据(如脏页数据)的同步，使只读节点可以根据脏页刷新信息安全地进行脏页清理。基于此，可选地，还可以执行以下可选步骤。
步骤S210：从所述第一节点获取日志文件的指示信息。
首先需要说明的是，本步骤为可选步骤，其可以在步骤S202-S208执行过程中的任意时机执行。
针对不同的分布式系统，所述日志文件可以是任何适当类型的文件，只要能够记录所述读写节点在设定时间段内的历史操作的信息即可。例如，日志文件可以是WAL(Write Ahead Log)日志文件。
读写节点历史操作除包括脏页数据的更新操作，例如，脏页数据的修改、删除等外，还可能包括读写节点的交互操作，等等。
由于读写节点中的所有操作在执行之前都被先写入日志文件中，因此，在分布式系统发生断电等情况时，可以根据日志文件实现分布式系统中数据的恢复，提升分布式系统的容灾能力和数据可靠性。
只读节点可以获取读写节点的日志文件，并根据日志文件进行重放操作，从而获取读写节点的脏页同步的具体内容信息，以根据这些信息更新只读节点的缓存中的具体脏页数据。
与传统读写节点将完整的日志文件发送给只读节点不同，在本实施例中，发送给只 读节点的只是日志文件的指示信息，如日志文件的地址信息、指针信息、标识信息等中的一种或多种。只读节点可以根据日志文件的指示信息获取到读写节点的日志文件。例如，若读写节点将日志文件持久化存储到共享存储(如共享存储磁盘)中，则可以根据日志文件的指示信息，如日志文件在共享存储中的存储地址，获得日志文件的完整内容。使得只读节点和读写节点间的传输数据较小，且只读节点可以根据该指示信息快速获取到日志文件，这样既实现了日志文件的持久化存储，又实现了只读节点和读写节点间日志文件的同步。
步骤S212：根据所述指示信息，获取所述日志文件。
针对不同的指示信息，可以采用适当的方式获取对应的日志文件。例如，若所述指示信息为所述日志文件的指针信息，则步骤S212包括：从所述读写节点和所述只读节点的共享存储中，获取所述指针信息所指示的存储地址中所述日志文件。
由于只读节点和读写节点使用共享存储，读写节点和只读节点使用相同的文件系统并共享存储磁盘，使得读写节点和只读节点间的通信效率更高，解决了现有技术中，额外增加存储节点，既增加了硬件资源，又使得通信网络速度成为通信效率瓶颈的问题。
通过本实施例，从第一节点获取脏页刷新信息，根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页，对确定的所述缓存池中的脏页进行清理操作。由于在第一节点对至少一个脏页执行存储后会生成脏页刷新信息，脏页刷新信息中主要是被执行存储的脏页的信息，一方面，使得第一节点之外的其他节点可以及时、快速地确定已被存储的脏页，且可以对这些脏页进行集中处理，例如，进行清理操作，提升了处理效率；另一方面，使得其他节点根据脏页刷新信息进行脏页清理，保证清理掉的脏页是已经被第一节点执行存储的脏页，避免了现有技术中其他节点只能通过WAL日志同步第一节点的脏页，不能及时确定脏页是否被存储，而导致清理脏页时直接丢弃脏页数据，造成其他节点与第一节点间数据一致性差，且无法确保分布式系统的MVCC的问题。
在使读写节点进行脏页刷新(即对脏页进行持久化存储)时，读写节点可以采用主动方式刷新脏页，即读写节点主动扫描脏页，并对扫描到的脏页进行持久化存储；也可以采用被动方式刷新脏页，即读写节点根据只读节点发送的脏页清理请求消息对脏页进行持久化存储。用户可以根据需要配置使用某一种脏页刷新方式或者两种脏页刷新方式配合使用，使得脏页刷新更加灵活多样，刷新时机更加丰富，可以在多种情况下保证只读节点与读写节点间的数据一致性。
只读节点与读写节点间通过脏页刷新信息进行脏页同步，使得只读节点可以及时根 据脏页刷新信息进行脏页清理，提升了只读节点脏页清理的安全性和及时性。
读写节点与只读节点采用共享存储，读写节点在向只读节点发送日志文件时，只需要将日志文件的指示信息发送给只读节点，只读节点即可从共享存储中获取到日志文件，并实现脏页数据同步，提升了通信效率，减少了数据传输，且无需增加额外硬件资源，无需对分布式系统内核做深入改造，适用性更好。
本实施例的数据同步的处理方法可以由任意适当的具有数据处理能力的电子设备执行，包括但不限于：服务器、移动终端(如平板电脑、手机等)和PC机等。
实施例三
参照图4，示出了根据本发明实施例三的一种数据同步的处理方法的步骤流程图。
本实施例的数据同步的处理方法包括以下步骤：
步骤S302：对至少一个脏页进行存储，并生成脏页刷新信息。
在本实施例中，以分布式系统的第一节点为执行主体，对数据同步的处理方法进行说明。其中，第一节点具有读数据的权限和写数据的权限，分布式系统中还包括第二节点，第二节点只具有读数据的权限。
第一节点可以主动对其缓存池中的至少一个脏页进行存储(如持久化存储到外部存储器，或者存储到内部存储器中)，本实施例中以将脏页的数据写入第一节点的外部存储器中为例。或者，第一节点也可以被动地根据第二节点发送的脏页清理请求消息，对其缓存池中至少一个脏页进行持久化存储。
无论第一节点主动进行持久化存储还是被动进行持久化存储，在其进行持久化存储后，均可以根据被持久化存储的脏页的信息，生成脏页刷新信息。例如，根据脏页的信息中的脏页的标识(key值)，生成脏页刷新信息。
如前所述，脏页刷新信息与现有日志文件不同，其中的主要内容为被持久化存储的脏页的信息，而不包括第一节点执行的其它操作的信息(如第一节点的交互操作的信息、读取数据的信息，等等)。该脏页刷新信息在一个或多个(两个及两个以上)脏页进行持久化存储后即可生成。通过该脏页刷新信息，可以及时地告知第二节点脏页的刷新情况，使第二节点能够及时、快速地确定被执行持久化存储的脏页，进而可以根据该脏页刷新信息进行脏页清理，保证脏页清理的及时性和安全性。
步骤S304：将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
第一节点通过将脏页刷新信息发送给第二节点，使第二节点能够确定被持久化存储 的脏页，进而可以在进行脏页清理操作时，将被持久化存储的脏页清理掉，这样既能够使第二节点进行脏页的清理操作，提升缓存页的复用率，又能够充分保证第二节点与第一节点间的数据一致性。
通过本实施例，第一节点对其中的至少一个脏页进行存储后，生成脏页刷新信息，并将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。由于第一节点针对其对脏页的存储操作生成了对应的脏页刷新信息，脏页刷新信息中主要是指示被执行存储的脏页，使得第二节点能够在第一节点完成脏页存储后及时、快速地确定被存储的脏页，并可以根据脏页刷新信息集中对脏页进行处理，例如进行脏页清理，且可以保证脏页清理的安全性，确保第二节点与第一节点间的数据一致性。
本实施例的数据同步的处理方法可以由任意适当的具有数据处理能力的电子设备执行，包括但不限于：服务器、移动终端(如平板电脑、手机等)和PC机等。
实施例四
参照图5，示出了根据本发明实施例四的一种数据同步的处理方法的步骤流程图。
在本实施例中，仍从分布式系统的第一节点的角度，对数据同步的处理方法进行说明。其中，第一节点为分布式系统中的读写节点，第二节点为分布式系统中的只读节点。
本实施例的数据同步的处理方法包括以下步骤：
步骤S402：对至少一个脏页进行存储，并生成脏页刷新信息。
第一种情况中，读写节点主动对脏页进行存储(如，持久化存储)时，步骤S402包括：对缓存池进行扫描，以确定至少一个脏页；将确定的所述至少一个脏页中的数据存储到共享存储中；根据所述已存储的脏页的信息，生成所述脏页刷新信息。其中，所述脏页的信息包括但不限于：脏页的标识等。
在一具体实现中，读写节点中的写入进程(writer process)周期性扫描读写节点的缓存池，以确定读写节点中的至少一个脏页，并将确定的所述至少一个脏页中的数据存储到共享存储中。
其中，写入进程一次扫描脏页数默认为100个(即当从缓存池中扫描出100个脏页时确定完成一次扫描)，单次最大扫描个数为缓存池中总缓存页数的1/4(四分之一)。当然，本领域技术人员可以根据需要配置其他的默认一次扫描的脏页数据和单次最大扫描个数。
在写入进程将脏页中的数据存储到读写节点和只读节点共享的共享存储中后，可以 将被存储的脏页的信息写入读写节点自身的共享内存中，并唤醒发送进程(sender process)，发送进程从共享内存中获取被写入的脏页的信息，并生成所述脏页刷新信息。
例如，根据已存储的脏页的标识或者可标识脏页的其他信息，生成脏页刷新信息，等等。
该读写节点可以采用多个进程模式进行脏页的存储，可以提升工作效率，充分利用硬件资源，且采用共享内存实现多个进程间的数据传输，保证了通信效率。
第二种情况中，读写节点被动对脏页进行存储(如，持久化存储)时，所述步骤S402包括：从所述第二节点接收脏页清理请求消息，并根据所述脏页清理请求消息，确定缓存池中所述脏页清理请求消息所指示的至少一个脏页；对确定的所述至少一个脏页进行存储，并生成所述脏页刷新信息。其中，所述脏页清理请求消息如实施例二的步骤S202中所述，在此不再赘述。
在一具体实现中，读写节点通过发送进程从只读节点接收脏页清理请求消息，根据该脏页清理请求消息中包含的脏页的标识，确定读写节点的缓存池中脏页清理请求消息指示的至少一个脏页。读写节点确定至少一个脏页后，可以直接对其进行存储，并根据已存储的脏页的信息生成脏页刷新信息。其生成脏页刷新信息的方式可以与前述第一种情况中的方式相同，在此不再赘述。
需要说明的是，根据该脏页清理请求消息，确定读写节点的缓存池中脏页清理请求消息指示的至少一个脏页时，若脏页清理请求消息中指示有读写节点的缓存池中未包含的脏页，则可以直接将该未包含的脏页确定为已存储的脏页。
例如，脏页清理请求消息中包括脏页A、B和C，读写节点的缓存池中包含对应的脏页A和B，则确定的脏页清理请求消息指示的至少一个脏页为脏页A和B，并将脏页C直接确定为已存储的脏页。在生成脏页刷新信息时，根据脏页A、B和C的信息生成。
步骤S404：将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
在一具体实现方式中，所述将所述脏页刷新信息发送给只读节点包括以下子步骤：
子步骤S4041：根据所述脏页刷新信息和预设的消息类型，生成同步消息。
通过同步消息，只读节点可以根据其中的脏页刷新信息进行脏页清理。该同步消息采用消息类型(如’f’)，利用现有的消息格式，形成专用于传输脏页刷新信息的同步消息，既实现了专有消息的生成和处理，又有效利用了已有消息，降低了脏页刷新信息的传输成本。但需要说明的是，“f”仅为示例性说明，在实际应用中，本领域技术人员 可以采用其它适当的具体形式来指示该消息类型，与现有消息类型不同即可。
具体地，该同步消息包括消息头和消息体，消息头中包含该同步消息的消息类型，消息体中包含有脏页刷新信息。或者，本领域技术人员可以根据需要采用其他结构的消息，并将消息类型和脏页刷新信息配置到适当位置，本实施例对此不作限定。
子步骤S4042：将所述同步消息发送给所述第二节点。
同步消息可以由读写节点中的发送进程生成，并发送给只读节点。例如，发送给只读节点中的接收进程。需要说明的是，读写节点中的每个发送进程对应一个只读节点，当分布式系统中包含多个只读节点时，读写节点中包含多个发送进程。
通过将同步消息发送给只读节点，使得只读节点可以从中获取脏页刷新信息，该脏页刷新信息用于传递被执行存储(如，持久化存储)的脏页的信息，与传统的通过日志文件传递被执行存储的脏页的信息不同，脏页刷新信息中不包括读写节点执行的其它操作的信息，因此，其在脏页进行存储后即可生成并被发送，故而，可以及时告知只读节点脏页的刷新情况，使只读节点能够及时、快速地确定被执行存储的脏页，进而可以根据该脏页刷新信息进行脏页清理，保证脏页清理的及时性和安全性。
基于此，可选地，还可以执行以下可选步骤。
步骤S406：根据在设定时间段内的历史操作的信息，生成日志文件，并将所述日志文件存储到共享存储中。
需要说明的是，本步骤可以在步骤S402-S404执行过程中的任意时机执行。
为了确保分布式系统的容灾能力，读写节点采用日志文件记录所述读写节点在设定时间段内的历史操作的信息，以便在分布式系统发生意外断电等情况时可以根据日志文件恢复分布式系统中的数据。
针对不同的分布式系统，所述日志文件可以是任何适当类型的文件，例如，日志文件可以是WAL(Write Ahead Log)日志文件。
日志文件中记录的历史操作包括但不限于脏页数据的更新操作。例如，脏页数据的修改、删除等。
读写节点可以将生成的日志文件存储(如，持久化存储)到共享存储中。这样一方面可以实现日志文件的存储，提升数据可靠性；另一方面，便于后续只读节点获取该日志文件，并根据该日志文件同步读写节点中脏页的内容数据。
步骤S408：获取所述日志文件在所述共享存储中的存储地址，并根据所述存储地址生成与所述日志文件对应的指示信息。
将日志文件存储后，可以获取日志文件的共享存储中的存储地址，并生成对应的指示信息，以便后续将该指示信息发送给只读节点，使其根据指示信息从共享存储中获取日志文件。
在本实施例中，所述指示信息为指示所述日志文件的存储地址的指针信息。由于只读节点和读写节点采用共享存储，因此，只读节点根据指针信息即可从共享存储中获取日志文件，这样使得读写节点只需向只读节点发送指针信息即可实现日志文件的共享，相较于现有技术中需要发送整个日志文件的方案，可以减少传输的数据量，提升通信效率和通信可靠性。
步骤S410：将所述日志文件的指示信息发送给所述第二节点。
在本实施例中，可以由读写节点的发送进程将该指示信息发送给只读节点，使只读节点可以根据指示信息从共享存储中获取日志文件，并根据日志文件进行重放操作，以同步脏页数据。
通过本实施例，第一节点对其中的至少一个脏页进行存储后，生成脏页刷新信息，并将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。由于第一节点针对其对脏页的存储操作生成了对应的脏页刷新信息，脏页刷新信息中主要是指示被执行存储的脏页，使得第二节点能够在第一节点完成脏页存储后及时、快速地确定被存储的脏页，并可以根据脏页刷新信息集中对脏页进行处理，例如进行脏页清理，且可以保证脏页清理的安全性，确保第二节点与第一节点间的数据一致性。
此外，该数据同步的处理方法应用于基于共享存储的分布式系统中，由于共享存储采用分布式文件系统，以很好的支撑分布式系统集中存储资源的弹性扩展，而且能够实现高效的读写节点和只读节点间的脏页同步，脏页同步更加直接高效，不需要大规模改造数据库内核，不依赖其他组件和硬件资源，保证了数据一致性；具有很好的普适性，不需要复杂的网络、硬件环境，不增加新的组件，能够适应任何一写多读的分布式系统。
下面以该数据同步的处理方法应用于分析型数据库(AnalyticDB)的具体使用场景为例进行说明。
分析型数据库是一种海量数据实时高并发在线分析(Realtime OLAP，联机分析技术)云计算服务的分布式系统。其可以使用户可以在毫秒级针对千亿级数据进行即时的多维分析透视和业务探索。分析型数据库对海量数据的自由计算和极速响应能力，能让用户在瞬息之间进行灵活的数据探索，快速发现数据价值，并可直接嵌入业务系统为终端客 户提供分析服务。
为了满足用户需求，且实现存储容量的无限扩展，并能支持非常灵活的弹性伸缩，分析型数据库采用共享存储，且使用一写多读架构。其中，共享存储是指多个实例间使用相同的文件系统，并共享存储磁盘。读写节点和只读节点间通过共享存储来共享读写节点的日志文件和数据库的数据文件，读写节点负责写入、更新数据，只读节点可以读取数据。
在这种一写多读的分析型数据库中，需要只读节点高效地同步读写节点的脏页信息，否则可能出现只读节点从数据库中读取到的数据是过期数据从而导致数据不一致和MVCC被破坏等问题。
一种基于共享存储的一写多读分析型数据库的结构如图11所示，该数据库中，读写节点(RW节点)和只读节点(RO节点)具有共享存储，且其使用DBFS分布式文件系统，共享存储内部使用适当的存储系统完成数据的落盘(即持久化存储)。在数据库内核上适配了分布式文件系统的API，整个数据库集群共同使用相同的文件系统，读写相同的数据目录。
一方面，如图11所示，在数据库中实现读写节点和只读节点间的脏页数据同步时，读写节点完成WAL(Write Ahead Log，是数据库中用来保证数据一致性和故障恢复的方式)日志落盘，即将其写入到分布式文件系统中，同时会将WAL日志发送给只读节点，发送的仅仅是WAL日志在共享存储中的指针，只读节点根据指针读取对应的WAL日志并进行重放操作，以使只读节点同步读写节点中的至少一个脏页的内容数据。
另一方面，读写节点也会按照一定的策略将脏页中的数据写入到共享存储中，实现脏页数据的落盘。
在本使用场景中，读写节点将脏页进行数据落盘的过程可以称为脏页刷新。读写节点将脏页数据落盘后，需要及时将脏页刷新信息同步给只读节点，以便只读节点尽快释放其缓存池中的脏页，使脏页变为可复用的缓存页。
如图12所示，其示出了一种读写节点主动同步脏页刷新信息的步骤示意图。在该过程中，读写节点主动将脏页数据落盘，并向只读节点同步脏页刷新信息，以便其进行脏页清理操作。具体步骤为：
步骤A1：读写节点的写入进程(writer process)周期性的扫描读写节点缓存池，如果扫描到脏页就将其落盘，并将已落盘(已持久化存储)的脏页的信息写到读写节点的共享内存里面，如果没有扫描到脏页，则写入进程会进行短暂的休息。其中，写入进 程一次扫描脏页数默认100个，最大扫描个数为总缓存页个数的1/4。
步骤B1：扫描到脏页的情况下，唤醒发送进程(sender process)进行处理并根据处理结果生成同步消息。
例如，发送进程根据共享内存中的已落盘脏页的信息，生成脏页刷新信息，并根据脏页刷新信息生成同步消息，将其发送给对应的只读节点的接收进程(receiver process)。需要注意的是每个发送进程对应一个只读节点，所以具有多个只读节点时，发送进程会有多个。
为了实现脏页刷新信息的顺利传递，在通信机制上增加新的消息类型'f'，表示与脏页刷新信息对应的同步消息。
步骤C1：只读节点的接收进程接收到同步消息后，从中获取并处理脏页刷新信息，根据脏页刷新信息的指示，对只读节点的缓存池中的脏页清理操作，即将脏页的标记清零，标记为可复用状态。
通过该过程，只读节点可以安全地清理脏页，避免现有技术中，只读节点将没有落盘的脏页清理掉导致只读节点和读写节点数据不一致的问题。保证只读节点从共享存储重新读取到的数据是最新的数据。
在只读节点中，接收到用户的数据查询请求时，会从只读节点的缓存池中获取数据查询请求所请求的数据所在的缓存页，如果数据查询请求需要的缓存页不在缓存池中，则需要从共享存储中读取数据到缓存页中。这一过程中存在着只读节点缓存页可用数量较少的情况，例如，只读节点接收了大量的数据查询请求，同时读写节点向只读节点同步了大量脏页数据，此时只读节点的缓存池中可用的缓存页会变少，会触发脏页清理。
在脏页清理时，若某个脏页在读写节点没有将其写入到共享存储中时，被只读节点清理了，那么只读节点接收到查询请求需要查询这个脏页的数据时，只读节点去共享存储中读取的数据与读写节点中保存的脏页数据就不一致，为了避免数据不一致和MVCC破坏等问题，只读节点的脏页只有在读写节点将对应脏页落盘后才能安全清理。
如图13所示，其示出了一种读写节点被动同步脏页刷新信息的步骤示意图。在该过程中，读写节点被动将脏页数据落盘，并向只读节点同步脏页刷新信息，以便其进行脏页清理操作。具体步骤为：
步骤A2：只读节点的用户查询进程(client query process)请求可用缓存页时，若发现没有可用的缓存页，则需要使用clock sweep算法进行脏页清理操作，在清理过程中将发现的只读节点的缓存池中的脏页记录到只读节点的共享内存中的刷新请求列表 (flush request list)中。
步骤B2：当用户查询进程确定刷新请求列表中有脏页信息，唤醒接收进程，使接收进程根据脏页信息生成脏页清理请求消息，并发送给读写节点的发送进程。
该脏页清理请求消息的消息类型为增加的新消息类型'f'，以标识该脏页清理请求消息与脏页清理相关。
步骤C2：读写节点的发送进程收到脏页清理请求消息后，将直接进行脏页的落盘操作。
需要注意的是只读节点和读写节点间的脏页可能不对等，会存在脏页清理请求消息中的脏页不在读写节点的缓存池中的情况，这时可以默认该脏页已经被落盘。
步骤D2：读写节点的发送进程根据脏页刷新信息生成同步消息，并发送给对应的只读节点的接收进程。
步骤E2：只读节点的接收进程接收同步消息，并根据其中的脏页刷新信息进行脏页清理。
本使用场景中的脏页处理过程，基于物理复制技术(物理复制指数据库多个实例间进行数据同步的机制，保证物理磁盘上的数据同步)，结合分布式文件系统，实现了一种高效的读写节点与只读节点间的脏页同步，同时支持了读写节点的主动同步脏页机制和由只读节点引发的被动同步脏页机制，实现方式稳定高效。
此外，其解决了图1的现有技术的方案中将日志重放工作下沉到了存储节点，增加硬件资源的同时，新增一个存储节点，每次只读节点或者读写节点缓存未命中的时候都需要去存储节点获取数据，使得网络传输速度成为了通信速度瓶颈的问题；而且解决了该方案中，只读节点根据日志同步脏页数据时，若其缓存池中不存在该缓存页则直接忽略该条操作，导致后续只读节点中的数据可能与读写节点中的数据不一致的问题；还解决了需要对数据库内核做非常深入的改造，数据库复杂度、数据库集群管理难度增加的问题。
在本使用场景中的数据同步的处理方法可以由任意适当的具有数据处理能力的电子设备执行，包括但不限于：服务器、移动终端(如平板电脑、手机等)和PC机等。
实施例五
参照图6，示出了根据本发明实施例五的一种数据同步的处理装置的结构框图。
本实施例的数据同步的处理装置包括：第一获取模块502，用于从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后 生成；第一确定模块504，用于根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页；清理模块506，用于对确定的所述缓存池中的脏页进行清理操作。
通过本实施例，从第一节点获取脏页刷新信息，根据所述脏页刷新信息，确定第二节点的缓存池中所述脏页刷新信息所指示的脏页，对确定的所述缓存池中的脏页进行清理操作。由于在第一节点对至少一个脏页执行存储后会生成脏页刷新信息并发送给第二节点，脏页刷新信息中主要是被执行存储的脏页的信息，一方面，使得第二节点可以及时、快速地确定已被存储的脏页，且可以对这些脏页进行集中处理，例如，进行清理操作，提升了处理效率；另一方面，使得第二节点根据脏页刷新信息进行脏页清理，保证清理掉的脏页是已经被第一节点执行存储的脏页，避免了现有技术中第二节点只能通过WAL日志同步第一节点的脏页，不能及时确定脏页是否被存储，而导致清理脏页时直接丢弃脏页数据，造成第二节点与第一节点间数据一致性差，且无法确保分布式系统的MVCC的问题。
实施例六
参照图7，示出了根据本发明实施例六的一种数据同步的处理装置的结构框图。
本实施例的数据同步的处理装置包括：第一获取模块602，用于从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；第一确定模块604，用于根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页；清理模块606，用于对确定的所述缓存池中的脏页进行清理操作。
可选地，第一节点为读写节点，所述第一获取模块602用于从所述读写节点接收同步消息，若所述同步消息的消息类型为所述脏页刷新信息对应的消息类型，则从所述同步消息中获取所述脏页刷新信息。
可选地，所述装置还包括：请求发送模块608，用于在从第一节点获取脏页刷新信息之前，当确定进行脏页清理时，根据所述缓存池中的脏页的信息，生成并向所述第一节点发送脏页清理请求消息，以请求所述脏页刷新信息。
可选地，所述请求发送模块608包括：扫描模块6081，用于当确定进行脏页清理时，扫描缓存池中的脏页；请求生成模块6082，用于根据扫描出的脏页的信息和预设的消息类型，生成并向所述第一节点发送脏页清理请求消息。
可选地，所述装置还包括：第二获取模块610，用于从所述第一节点获取日志文件的指示信息，其中，所述日志文件用于记录所述第一节点在设定时间段内的历史操作的信息；第三获取模块612，用于根据所述指示信息，获取所述日志文件。
可选地，所述指示信息为所述日志文件的指针信息；所述第三获取模块612用于从所述第一节点和第二节点的共享存储中，获取所述指针信息所指示的存储地址中所述日志文件。
本实施例的数据同步的处理装置用于实现前述多个方法实施例中相应的数据同步的处理方法，并具有相应方法实施例的有益效果，在此不再赘述。
实施例七
参照图8，示出了根据本发明实施例七的一种数据同步的处理装置的结构框图。
本实施例的数据同步的处理装置包括：存储与生成模块702，用于对至少一个脏页进行存储，并生成脏页刷新信息；信息发送模块704，用于将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
通过本实施例，对至少一个脏页进行存储后，生成脏页刷新信息，并将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。由于第一节点针对其对脏页的存储操作生成了对应的脏页刷新信息，脏页刷新信息中主要是指示被执行存储的脏页，使得第二节点能够在第一节点完成脏页存储后及时、快速地确定被存储的脏页，并可以根据脏页刷新信息集中对脏页进行处理，例如进行脏页清理，且可以保证脏页清理的安全性，确保第二节点与第一节点间的数据一致性。
实施例八
参照图9，示出了根据本发明实施例八的一种数据同步的处理装置的结构框图。
本实施例的数据同步的处理装置包括：存储与生成模块802，用于对至少一个脏页进行存储，并生成脏页刷新信息；信息发送模块804，用于将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
可选地，所述存储与生成模块802包括：第二确定模块8021，用于对缓存池进行扫描，以确定至少一个脏页；第一存储模块8022，用于将确定的所述至少一个脏页中的数据存储到共享存储中；信息生成模块8023，用于根据所述已存储的脏页的信息，生成所述脏页刷新信息。
可选地，所述第二节点为只读节点，所述信息发送模块804包括：消息生成模块8041，用于根据所述脏页刷新信息和预设的消息类型，生成同步消息；消息发送模块8042，用 于将所述同步消息发送给所述第二节点。
可选地，所述存储与生成模块802包括：接收模块8024，用于从所述第二节点接收脏页清理请求消息，并根据所述脏页清理请求消息，确定缓存池中所述脏页清理请求消息所指示的至少一个脏页；第二存储模块8025，用于对确定的所述至少一个脏页进行存储，并生成所述脏页刷新信息。
可选地，所述装置还包括：日志生成模块806，用于根据所述第一节点在设定时间段内的历史操作的信息，生成日志文件，并将所述日志文件存储到共享存储中。
可选地，所述装置还包括：指示生成模块808，用于在所述将所述日志文件存储到共享存储中之后，获取所述日志文件在所述共享存储中的存储地址，并根据所述存储地址生成与所述日志文件对应的指示信息；指示发送模块810，用于将所述日志文件的指示信息发送给所述第二节点。
可选地，所述指示信息为指示所述日志文件的存储地址的指针信息。
本实施例的数据同步的处理装置用于实现前述多个方法实施例中相应的数据同步的处理方法，并具有相应方法实施例的有益效果，在此不再赘述。
实施例九
参照图10，示出了根据本发明实施例九的一种电子设备的结构示意图，本发明具体实施例并不对电子设备的具体实现做限定。
如图10所示，该电子设备可以包括：处理器(processor)902、通信接口(Communications Interface)904、存储器(memory)906、以及通信总线908。
其中：
处理器902、通信接口904、以及存储器906通过通信总线908完成相互间的通信。
通信接口904，用于与其它电子设备如终端设备或服务器进行通信。
处理器902，用于执行程序910，具体可以执行上述数据同步的处理方法实施例中的相关步骤。
具体地，程序910可以包括程序代码，该程序代码包括计算机操作指令。
处理器902可能是中央处理器CPU，或者是特定集成电路ASIC(Application Specific Integrated Circuit)，或者是被配置成实施本发明实施例的一个或多个集成电路。电子设备包括的一个或多个处理器，可以是同一类型的处理器，如一个或多个CPU；也可以是不同类型的处理器，如一个或多个CPU以及一个或多个ASIC。
存储器906，用于存放程序910。存储器906可能包含高速RAM存储器，也可能还包 括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。
程序910具体可以用于使得处理器902执行以下操作：从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页；对确定的所述缓存池中的脏页进行清理操作。
在一种可选的实施方式中，第一节点为读写节点，程序910还用于使得处理器902在从第一节点获取脏页刷新信息时，从所述读写节点接收同步消息，若所述同步消息的消息类型为所述脏页刷新信息对应的消息类型，则从所述同步消息中获取所述脏页刷新信息。
在一种可选的实施方式中，程序910还用于使得处理器902在从第一节点获取脏页刷新信息之前，当确定进行脏页清理时，根据所述缓存池中的脏页的信息，生成并向所述第一节点发送脏页清理请求消息，以请求所述脏页刷新信息。
在一种可选的实施方式中，程序910还用于使得处理器902在当确定进行脏页清理时，根据所述缓存池中的脏页，生成并向所述第一节点发送脏页清理请求消息时，当确定进行脏页清理时，扫描缓存池中的脏页；根据扫描出的脏页的信息和预设的消息类型，生成并向所述第一节点发送脏页清理请求消息。
在一种可选的实施方式中，程序910还用于使得处理器902从所述第一节点获取日志文件的指示信息，其中，所述日志文件用于记录所述第一节点在设定时间段内的历史操作的信息；根据所述指示信息，获取所述日志文件。
在一种可选的实施方式中，程序910还用于使得处理器902在所述根据所述指示信息，获取所述日志文件时，若指示信息为所述日志文件的指针信息；则从所述第一节点和第二节点的共享存储中，获取所述指针信息所指示的存储地址中所述日志文件。
或者，程序910具体可以用于使得处理器902执行以下操作：对至少一个脏页进行存储，并生成脏页刷新信息；将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
在一种可选的实施方式中，程序910还用于使得处理器902在对至少一个脏页进行存储，并生成脏页刷新信息时，对缓存池进行扫描，以确定所述读写节点中的至少一个脏页；将确定的所述至少一个脏页中的数据存储到共享存储中；根据所述已存储的脏页的信息，生成所述脏页刷新信息。
在一种可选的实施方式中，程序910还用于使得处理器902在将所述脏页刷新信息 发送给第二节点时，根据所述脏页刷新信息和预设的消息类型，生成同步消息；将所述同步消息发送给所述第二节点。
在一种可选的实施方式中，程序910还用于使得处理器902在对至少一个脏页进行存储，并生成脏页刷新信息时，从所述第二节点接收脏页清理请求消息，并根据所述脏页清理请求消息，确定缓存池中所述脏页清理请求消息所指示的至少一个脏页；对确定的所述至少一个脏页进行存储，并生成所述脏页刷新信息。
在一种可选的实施方式中，程序910还用于使得处理器902根据在设定时间段内的历史操作的信息，生成日志文件，并将所述日志文件存储到共享存储中。
在一种可选的实施方式中，程序910还用于使得处理器902在所述将所述日志文件存储到共享存储中之后，获取所述日志文件在所述共享存储中的存储地址，并根据所述存储地址生成与所述日志文件对应的指示信息；将所述日志文件的指示信息发送给所述第二节点。
在一种可选的实施方式中，所述指示信息为指示所述日志文件的存储地址的指针信息。
程序910中各步骤的具体实现可以参见上述数据同步的处理方法实施例中的相应步骤和单元中对应的描述，在此不赘述。所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的设备和模块的具体工作过程，可以参考前述方法实施例中的对应过程描述，在此不再赘述。
通过本实施例的电子设备，从第一节点获取脏页刷新信息，根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页，对确定的所述缓存池中的脏页进行清理操作。由于在第一节点对至少一个脏页执行存储后会生成脏页刷新信息，脏页刷新信息中主要是被执行存储的脏页的信息，一方面，使得第一节点之外的其他节点可以及时、快速地确定已被存储的脏页，且可以对这些脏页进行集中处理，例如，进行清理操作，提升了处理效率；另一方面，使得其他节点根据脏页刷新信息进行脏页清理，保证清理掉的脏页是已经被第一节点执行存储的脏页，避免了现有技术中其他节点只能通过WAL日志同步第一节点的脏页，不能及时确定脏页是否被存储，而导致清理脏页时直接丢弃脏页数据，造成其他节点与第一节点间数据一致性差，且无法确保分布式系统的MVCC的问题。
或者，第一节点对其中的至少一个脏页进行存储后，生成脏页刷新信息，并将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节 点中的脏页进行清理操作。由于第一节点针对其对脏页的存储操作生成了对应的脏页刷新信息，脏页刷新信息中主要是指示被执行存储的脏页，使得第二节点能够在第一节点完成脏页存储后及时、快速地确定被存储的脏页，并可以根据脏页刷新信息集中对脏页进行处理，例如进行脏页清理，且可以保证脏页清理的安全性，确保第二节点与第一节点间的数据一致性。
需要指出，根据实施的需要，可将本发明实施例中描述的各个部件/步骤拆分为更多部件/步骤，也可将两个或多个部件/步骤或者部件/步骤的部分操作组合成新的部件/步骤，以实现本发明实施例的目的。
上述根据本发明实施例的方法可在硬件、固件中实现，或者被实现为可存储在记录介质(诸如CD ROM、RAM、软盘、硬盘或磁光盘)中的软件或计算机代码，或者被实现通过网络下载的原始存储在远程记录介质或非暂时机器可读介质中并将被存储在本地记录介质中的计算机代码，从而在此描述的方法可被存储在使用通用计算机、专用处理器或者可编程或专用硬件(诸如ASIC或FPGA)的记录介质上的这样的软件处理。可以理解，计算机、处理器、微处理器控制器或可编程硬件包括可存储或接收软件或计算机代码的存储组件(例如，RAM、ROM、闪存等)，当所述软件或计算机代码被计算机、处理器或硬件访问且执行时，实现在此描述的数据同步的处理方法。此外，当通用计算机访问用于实现在此示出的数据同步的处理方法的代码时，代码的执行将通用计算机转换为用于执行在此示出的数据同步的处理方法的专用计算机。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及方法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明实施例的范围。
以上实施方式仅用于说明本发明实施例，而并非对本发明实施例的限制，有关技术领域的普通技术人员，在不脱离本发明实施例的精神和范围的情况下，还可以做出各种变化和变型，因此所有等同的技术方案也属于本发明实施例的范畴，本发明实施例的专利保护范围应由权利要求限定。

Claims (28)

  1. 一种数据同步的处理方法，其特征在于，包括：
    从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；
    根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指示的脏页；
    对确定的所述缓存池中的脏页进行清理操作。
  2. 根据权利要求1所述的方法，其特征在于，所述第一节点为读写节点，所述从第一节点获取脏页刷新信息，包括：
    从所述读写节点接收同步消息，若所述同步消息的消息类型为所述脏页刷新信息对应的消息类型，则从所述同步消息中获取所述脏页刷新信息。
  3. 根据权利要求1所述的方法，其特征在于，在从第一节点获取脏页刷新信息之前，所述方法还包括：
    当确定进行脏页清理时，根据所述缓存池中的脏页的信息，生成并向所述第一节点发送脏页清理请求消息，以请求所述脏页刷新信息。
  4. 根据权利要求3所述的方法，其特征在于，当确定进行脏页清理时，根据所述缓存池中的脏页，生成并向读写节点发送脏页清理请求消息，包括：
    当确定进行脏页清理时，扫描所述缓存池中的脏页；
    根据扫描出的脏页的信息和预设的消息类型，生成并向所述第一节点发送脏页清理请求消息。
  5. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    从所述第一节点获取日志文件的指示信息，其中，所述日志文件用于记录所述第一节点在设定时间段内的历史操作的信息；
    根据所述指示信息，获取所述日志文件。
  6. 根据权利要求5所述的方法，其特征在于，所述指示信息为所述日志文件的指针信息；
    所述根据所述指示信息，获取所述日志文件，包括：
    从所述第一节点和第二节点的共享存储中，获取所述指针信息所指示的存储地址中所述日志文件。
  7. 一种数据同步的处理方法，其特征在于，包括：
    对至少一个脏页进行存储，并生成脏页刷新信息；
    将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
  8. 根据权利要求7所述的方法，其特征在于，所述对至少一个脏页进行存储，并生成脏页刷新信息，包括：
    对缓存池进行扫描，以确定至少一个脏页；
    将确定的所述至少一个脏页中的数据存储到共享存储中；
    根据已存储的脏页的信息，生成所述脏页刷新信息。
  9. 根据权利要求7所述的方法，其特征在于，所述第二节点为只读节点，所述将所述脏页刷新信息发送给第二节点，包括：
    根据所述脏页刷新信息和预设的消息类型，生成同步消息；
    将所述同步消息发送给所述第二节点。
  10. 根据权利要求7所述的方法，其特征在于，所述对至少一个脏页进行存储，并生成脏页刷新信息，包括：
    从所述第二节点接收脏页清理请求消息，并根据所述脏页清理请求消息，确定缓存池中所述脏页清理请求消息所指示的至少一个脏页；
    对确定的所述至少一个脏页进行存储，并生成所述脏页刷新信息。
  11. 根据权利要求7所述的方法，其特征在于，所述方法还包括：
    根据在设定时间段内的历史操作的信息，生成日志文件，并将所述日志文件存储到共享存储中。
  12. 根据权利要求11所述的方法，其特征在于，在所述将所述日志文件存储到共享存储中之后，所述方法还包括：
    获取所述日志文件在所述共享存储中的存储地址，并根据所述存储地址生成与所述日志文件对应的指示信息；
    将所述日志文件的指示信息发送给所述第二节点。
  13. 根据权利要求12所述的方法，其特征在于，所述指示信息为指示所述日志文件的存储地址的指针信息。
  14. 一种数据同步的处理装置，其特征在于，包括：
    第一获取模块，用于从第一节点获取脏页刷新信息，其中，所述脏页刷新信息通过所述第一节点在执行至少一个脏页的存储后生成；
    第一确定模块，用于根据所述脏页刷新信息，确定缓存池中所述脏页刷新信息所指 示的脏页；
    清理模块，用于对确定的所述缓存池中的脏页进行清理操作。
  15. 根据权利要求14所述的装置，其特征在于，所述第一节点为读写节点，所述第一获取模块用于从所述读写节点接收同步消息，若所述同步消息的消息类型为所述脏页刷新信息对应的消息类型，则从所述同步消息中获取所述脏页刷新信息。
  16. 根据权利要求14所述的装置，其特征在于，所述装置还包括：
    请求发送模块，用于在从第一节点获取脏页刷新信息之前，当确定进行脏页清理时，根据所述缓存池中的脏页的信息，生成并向所述第一节点发送脏页清理请求消息，以请求所述脏页刷新信息。
  17. 根据权利要求16所述的装置，其特征在于，所述请求发送模块包括：
    扫描模块，用于当确定进行脏页清理时，扫描所述缓存池中的脏页；
    请求生成模块，用于根据扫描出的脏页的信息和预设的消息类型，生成并向所述第一节点发送脏页清理请求消息。
  18. 根据权利要求14所述的装置，其特征在于，所述装置还包括：
    第二获取模块，用于从所述第一节点获取日志文件的指示信息，其中，所述日志文件用于记录所述第一节点在设定时间段内的历史操作的信息；
    第三获取模块，用于根据所述指示信息，获取所述日志文件。
  19. 根据权利要求18所述的装置，其特征在于，所述指示信息为所述日志文件的指针信息；
    所述第三获取模块用于从所述第一节点和第二节点的共享存储中，获取所述指针信息所指示的存储地址中所述日志文件。
  20. 一种数据同步的处理装置，其特征在于，包括：
    存储与生成模块，用于对第一节点中的至少一个脏页进行存储，并生成脏页刷新信息；
    信息发送模块，用于将所述脏页刷新信息发送给第二节点，以使所述第二节点根据所述脏页刷新信息对所述第二节点中的脏页进行清理操作。
  21. 根据权利要求20所述的装置，其特征在于，所述存储与生成模块包括：
    第二确定模块，用于对缓存池进行扫描，以确定至少一个脏页；
    第一存储模块，用于将确定的所述至少一个脏页中的数据存储到共享存储中；
    信息生成模块，用于根据已存储的脏页的信息，生成所述脏页刷新信息。
  22. 根据权利要求20所述的装置，其特征在于，所述第二节点为只读节点，所述信息发送模块包括：
    消息生成模块，用于根据所述脏页刷新信息和预设的消息类型，生成同步消息；
    消息发送模块，用于将所述同步消息发送给所述第二节点。
  23. 根据权利要求20所述的装置，其特征在于，所述存储与生成模块包括：
    接收模块，用于从所述第二节点接收脏页清理请求消息，并根据所述脏页清理请求消息，确定缓存池中所述脏页清理请求消息所指示的至少一个脏页；
    第二存储模块，用于对确定的所述至少一个脏页进行存储，并生成所述脏页刷新信息。
  24. 根据权利要求20所述的装置，其特征在于，所述装置还包括：
    日志生成模块，用于根据在设定时间段内的历史操作的信息，生成日志文件，并将所述日志文件存储到共享存储中。
  25. 根据权利要求24所述的装置，其特征在于，所述装置还包括：
    指示生成模块，用于在所述将所述日志文件存储到共享存储中之后，获取所述日志文件在所述共享存储中的存储地址，并根据所述存储地址生成与所述日志文件对应的指示信息；
    指示发送模块，用于将所述日志文件的指示信息发送给所述第二节点。
  26. 根据权利要求25所述的装置，其特征在于，所述指示信息为指示所述日志文件的存储地址的指针信息。
  27. 一种电子设备，包括：处理器、存储器、通信接口和通信总线，所述处理器、所述存储器和所述通信接口通过所述通信总线完成相互间的通信；
    所述存储器用于存放至少一可执行指令，所述可执行指令使所述处理器执行如权利要求1-6中任一项所述的数据同步的处理方法对应的操作，或者，执行如权利要求7-13中任一项所述的数据同步的处理方法对应的操作。
  28. 一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1-6中任一所述的数据同步的处理方法，或者，实现如权利要求7-13中任一所述的数据同步的处理方法。
PCT/CN2020/091523 2019-05-31 2020-05-21 数据同步的处理方法、装置、电子设备及计算机存储介质 Ceased WO2020238748A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910467267.4A CN112015807B (zh) 2019-05-31 2019-05-31 数据同步的处理方法、装置、电子设备及计算机存储介质
CN201910467267.4 2019-05-31

Publications (1)

Publication Number Publication Date
WO2020238748A1 true WO2020238748A1 (zh) 2020-12-03

Family

ID=73501067

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/091523 Ceased WO2020238748A1 (zh) 2019-05-31 2020-05-21 数据同步的处理方法、装置、电子设备及计算机存储介质

Country Status (2)

Country Link
CN (1) CN112015807B (zh)
WO (1) WO2020238748A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112650589A (zh) * 2020-12-29 2021-04-13 成都科来网络技术有限公司 一种均衡系统资源和落盘实时性的方法及可读存储介质
CN114595286A (zh) * 2020-12-04 2022-06-07 腾讯科技（深圳）有限公司 一种数据同步方法、装置、电子设备及存储介质
CN119248833A (zh) * 2024-09-24 2025-01-03 本原数据(北京)信息技术有限公司 数据库缓存管理方法、数据库、设备及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113904913B (zh) * 2021-08-19 2024-10-18 济南浪潮数据技术有限公司 一种基于管道的告警处理的方法、装置、设备、存储介质
US20250103613A1 (en) * 2024-12-11 2025-03-27 Lemon Inc. Efficient synchronization mechanisms for graph databases

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104111958A (zh) * 2013-04-22 2014-10-22 中国移动通信集团山东有限公司 一种数据查询方法及装置
US20150058295A1 (en) * 2012-05-02 2015-02-26 Huawei Technologies Co., Ltd. Data Persistence Processing Method and Apparatus, and Database System
CN104408126A (zh) * 2014-11-26 2015-03-11 杭州华为数字技术有限公司 一种数据库的持久化写入方法、装置和系统
CN104881418A (zh) * 2014-02-28 2015-09-02 阿里巴巴集团控股有限公司 用于MySQL的快速回收回滚空间的方法和装置
CN109086425A (zh) * 2018-08-09 2018-12-25 北京百度网讯科技有限公司 用于数据库的数据处理方法和装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7962703B1 (en) * 2007-08-08 2011-06-14 Symantec Corporation Techniques for improving dirty page logging
GB2515501A (en) * 2013-06-25 2014-12-31 Ibm Replication for on-line hot-standby database
CN104504147B (zh) * 2015-01-04 2018-04-10 华为技术有限公司 一种数据库集群的资源协调方法、装置及系统
KR20170054767A (ko) * 2015-11-10 2017-05-18 엘지전자 주식회사 데이터베이스 관리 시스템 및 그의 데이터 변경 및 복구 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150058295A1 (en) * 2012-05-02 2015-02-26 Huawei Technologies Co., Ltd. Data Persistence Processing Method and Apparatus, and Database System
CN104111958A (zh) * 2013-04-22 2014-10-22 中国移动通信集团山东有限公司 一种数据查询方法及装置
CN104881418A (zh) * 2014-02-28 2015-09-02 阿里巴巴集团控股有限公司 用于MySQL的快速回收回滚空间的方法和装置
CN104408126A (zh) * 2014-11-26 2015-03-11 杭州华为数字技术有限公司 一种数据库的持久化写入方法、装置和系统
CN109086425A (zh) * 2018-08-09 2018-12-25 北京百度网讯科技有限公司 用于数据库的数据处理方法和装置

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114595286A (zh) * 2020-12-04 2022-06-07 腾讯科技（深圳）有限公司 一种数据同步方法、装置、电子设备及存储介质
CN114595286B (zh) * 2020-12-04 2025-10-28 腾讯科技（深圳）有限公司 一种数据同步方法、装置、电子设备及存储介质
CN112650589A (zh) * 2020-12-29 2021-04-13 成都科来网络技术有限公司 一种均衡系统资源和落盘实时性的方法及可读存储介质
CN119248833A (zh) * 2024-09-24 2025-01-03 本原数据(北京)信息技术有限公司 数据库缓存管理方法、数据库、设备及存储介质

Also Published As

Publication number Publication date
CN112015807A (zh) 2020-12-01
CN112015807B (zh) 2024-07-02

Similar Documents

Publication Publication Date Title
WO2020238748A1 (zh) 数据同步的处理方法、装置、电子设备及计算机存储介质
KR102153804B1 (ko) 데이터 동기화 방법, 장치, 및 시스템
US11442961B2 (en) Active transaction list synchronization method and apparatus
CN103180852B (zh) 分布式数据处理方法及装置
CN102265277B (zh) 数据存储系统的操作方法和装置
US20190334850A1 (en) Systems and methods for storing and transferring message data
CN201682522U (zh) 会话信息存储系统及应用服务器
KR20210002574A (ko) 데이터 백업 방법, 저장 매체 및 컴퓨팅 기기
CN102741843A (zh) 从数据库中读取数据的方法及装置
CN113094430B (zh) 一种数据处理方法、装置、设备以及存储介质
CN103138912B (zh) 数据同步方法及系统
CN102780603B (zh) 网站流量控制方法及装置
CN104580226A (zh) 一种共享会话数据的系统和方法
CN111177159B (zh) 一种数据处理的系统、方法和数据更新设备
EP2987387A1 (en) System, method, and device for exposing wireless module data storage
CN113961510B (zh) 一种文件处理方法、装置、设备及存储介质
CN112307119A (zh) 数据同步方法、装置、设备及存储介质
CN107562385A (zh) 分布式存储客户端读取数据的方法、装置和设备
CN105138679A (zh) 一种基于分布式缓存的数据处理系统及处理方法
CN111737254B (zh) 一种提高应用缓存利用率的方法及装置
CN107612982A (zh) 一种降低功能页面请求次数的方法、存储介质、设备及系统
CN104866528A (zh) 多平台数据采集方法及系统
CN115858423A (zh) 缓存淘汰策略的切换方法、装置、电子设备及存储介质
CN107180118A (zh) 一种文件系统缓存数据管理方法及装置
CN114925078A (zh) 数据更新方法、系统、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20815443

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20815443

Country of ref document: EP

Kind code of ref document: A1