WO2024093263A1

WO2024093263A1 - Data processing system, method and apparatus, and related device

Info

Publication number: WO2024093263A1
Application number: PCT/CN2023/101428
Authority: WO
Inventors: 王伟; 任仁; 曹宇
Original assignee: 华为技术有限公司
Priority date: 2022-11-02
Filing date: 2023-06-20
Publication date: 2024-05-10
Also published as: CN117992467A

Abstract

A data processing system, comprising a computing cluster and a storage cluster (102), wherein the computing cluster comprises a master computing node (1011) and a slave computing node (1012), and the storage cluster (102) comprises at least one storage node. The master computing node (1011) is used for generating a binlog in response to a data update request, and sending the binlog to the storage cluster (102) for storage; and the slave computing node (1012) is used for reading the binlog stored in the storage cluster (102) and updating persistently stored data in the storage cluster (102) by means of playing back the binlog.

Description

Data processing system, method, device and related equipment

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 2, 2022, with application number 202211363509.3 and application name “A method for managing binlog”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on December 14, 2022, with application number 202211608424.7 and application name “Data processing systems, methods, devices and related equipment”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of database technology, and in particular to a data processing system, method, apparatus and related equipment.

Background technique

With the development of information technology, data processing systems, such as MySQL, PostgreSQL, openGauss, etc., are widely used in finance, communications, medical care, logistics, e-commerce and other fields for persistent storage of business data in various fields.

At present, data processing systems are usually deployed with a main center (or production center) and at least one disaster recovery center. The main center includes the main computing node and the main storage node, and the disaster recovery center includes the main computing node and the main storage node. During normal operation, the main center uses the main computing node and the main storage node to provide data reading and writing services to the outside world; the backup center is responsible for backing up the data stored in the main center, and when the main center fails, the disaster recovery center can use the backup data to continue to provide data reading and writing services to the outside world to avoid data loss, thereby ensuring the reliability of data storage.

Normally, when the master computing node updates the persistently stored data in the master storage node, it will send a binary log (binlog) file to the slave computing node, so that the slave computing node can complete the data update in the slave storage node by replaying the binlog file, thereby achieving data synchronization between the master center and the disaster recovery center. However, in actual application scenarios, when the master center fails, the data stored in the disaster recovery center is often inconsistent with the data in the master center before the failure, which causes the recovery point object (RPO) of the data processing system to fail to reach 0, affecting the reliability of the data processing system. Among them, RPO can be used to measure the maximum amount of data loss that occurs during disaster recovery of the data processing system.

Summary of the invention

A data processing system is provided to achieve consistency between the data stored in the disaster recovery center and the data in the main center before the failure, so as to improve the reliability of the data processing system and achieve an RPO of 0 for the data processing system. In addition, corresponding data processing methods, devices, computing device clusters, chips, computer-readable storage media, and computer program products are also provided.

In a first aspect, an embodiment of the present application provides a data processing system, which includes a computing cluster and a storage cluster, wherein the computing cluster and the storage cluster are connected through a network, such as through a wired network or a wireless network, and the computing cluster includes a master computing node and a slave computing node. Generally, the slave computing node serves as a disaster recovery node for the master computing node, and the storage cluster includes at least one storage node; the master computing node is used to generate a binlog (binary log) in response to a data update request, and the data update request can be sent to the master computing node by a user through a client, etc., and the master computing node is also used to send the binlog to the storage cluster for storage; the slave computing node is used to read the binlog stored in the storage cluster, and update the data persistently stored in the storage cluster by replaying the binlog (specifically replaying the database statements recorded in the binlog). For example, the slave computing node can synchronize data with the master computing node by timely replaying the binlog generated by the master computing node, or the slave computing node can replay the binlog when the master computing node fails to achieve data recovery, etc.

Since the master computing node and the slave computing node transmit binlogs through the storage cluster, and the master computing node does not need to send binlogs directly to the slave computing node, this can avoid excessive load on the master computing node, or unstable data transmission links between the master computing node and the slave computing node, resulting in the slave computing node failing to obtain the binlog generated by the master computing node. The slave computing node can achieve data synchronization between the master computing node and the slave computing node by replaying the binlog that can be obtained, or achieve data recovery when the master computing node fails. In this way, when the master computing node fails, the slave computing node can take over the business on the master computing node based on the data of the master computing node at the time of failure, thereby achieving an RPO of 0 for the data processing system and improving the reliability of the data processing system.

In a possible implementation, the storage cluster includes a log storage area, which is accessed by the master computing node and the slave computing node, that is, the master computing node and the slave computing node can share the log storage area, so that when the master computing node sends binlog to the storage cluster, it specifically sends the binlog to the log storage area for storage; correspondingly, the slave computing node specifically reads the binlog from the log storage area; wherein the storage cluster includes not only the log storage area, but also the data storage area, and the data storage area Used to store business data, such as business data processed during the normal operation of the master computing node, the data storage area can be accessed by the master computing node and the slave computing node, or the data storage area can only be accessed by the master computing node (the slave computing node can only access the data storage area when the master computing node fails). In this way, between the master computing node and the slave computing node, binlog transmission can be achieved through a shared storage area in the storage cluster, so as to ensure that the slave computing node can obtain the binlog generated by the master computing node as much as possible, thereby improving the reliability of the data processing system.

In a possible implementation, the slave computing node is specifically used to synchronize data with the master computing node by replaying binlog, that is, during the normal operation of the master computing node, the slave computing node can continuously replay the binlog generated by the master computing node to achieve data synchronization with the master computing node; or, the slave computing node is used to recover the data lost when the master computing node fails by replaying binlog, that is, before the master computing node fails, the slave computing node may not perform the operation of replaying binlog, and after the master computing node fails, the slave computing node recovers the data lost when the master computing node fails by replaying binlog, so that the RPO of the data processing system is 0. In this way, the reliability of the data processing system can be effectively improved.

In a possible implementation, the storage nodes included in the storage cluster include a master storage node and a slave storage node, wherein the slave storage node serves as a disaster recovery for the master storage node, and the master storage node and the slave storage node are deployed in the same data center or the same availability zone. Normally, the master storage node is used to provide read and write data services for the master computing node, and the slave storage node is used to provide read and write data services for the slave computing node; thus, the slave computing node, during the normal operation of the master computing node, will read the binlog generated and sent by the master computing node from the log storage area of the storage cluster, and update the data persistently stored in the slave storage node by replaying the binlog. In this way, the slave computing node can synchronize data with the master computing node by continuously replaying the binlog generated by the master computing node, so that when the master computing node fails, the slave computing node can take over the business on the master computing node based on the synchronized data, thereby achieving an RPO of 0 for the data processing system, thereby improving the reliability of the data processing system.

In one possible implementation, a storage cluster includes a target storage node among the storage nodes, which is used to persistently store the data written by the master computing node; the slave computing node may not perform the binlog playback operation during the normal operation of the master computing node, but when the master computing node fails, it reads the binlog from the log storage area and updates the data persistently stored in the target storage node by playing back the binlog, so as to restore the data lost by the master computing node when the failure occurs, thereby improving the reliability of the data processing system.

In a possible implementation, the storage cluster includes a main storage node and a slave storage node in the storage nodes, the slave storage node serves as a disaster recovery for the main storage node, the main storage node and the slave storage node are deployed in different data centers, or the main storage node and the slave storage node are deployed in different availability zones, usually, the main storage node is used to provide read and write data services to the main computing node, and the slave storage node is used to provide read and write data services to the slave computing node; in the process of transmitting binlog, the main computing node, specifically, sends the binlog to the main storage node for storage, and then the main storage node sends the binlog to the slave storage node, so that the slave computing node can read the binlog stored in the slave storage node. In this way, between the main computing node and the slave computing node, by transmitting binlog between the main storage node and the slave storage node, it is possible to ensure that the slave computing node can obtain the binlog generated by the main computing node, thereby improving the reliability of the data processing system.

In one possible implementation, the master storage node is also used to send baseline data to the slave storage node before sending the binlog to the slave storage node. The baseline data generally includes the data persistently stored by the master storage node at a certain moment (that is, the data stored on the data page) and the binlog generated by the master computing node before that moment. The slave storage node stores the baseline data after receiving it. In this way, the slave computing node can update the baseline data stored in the slave storage node by replaying the binlog to achieve data synchronization with the master center.

In a possible implementation, a target application is running on the main computing node, and the binlog transmitted through the storage cluster is generated during the operation of the target application. The target application includes a relational database management system RDBMS, and the RDBMS includes at least one of MySQL, PostgreSQL, OpenGauss, and Oracle. In this way, for the main computing node running any type of application, the reliability of the data processing system can be guaranteed by using the storage cluster to transmit binlog, thereby improving the compatibility and scalability of the data processing system for database applications.

In a possible implementation, the storage nodes in the storage cluster are specifically storage arrays, which are used to persistently store data. Since the storage array is usually configured with technologies based on independent disk redundant array technology, erasure coding technology, deduplication compression, data backup, etc., the reliability of persistent storage of data in the storage cluster can be further improved.

In a second aspect, an embodiment of the present application provides a data processing method, the method is applied to a data processing system, the data processing system includes a computer The method comprises: the master computing node generates a binary log binlog in response to a data update request; the master computing node sends the binlog to the storage cluster for storage; the slave computing node reads the binlog stored in the storage cluster; and the slave computing node updates the data persistently stored in the storage cluster by replaying the binlog.

In a possible implementation, the storage cluster includes a log storage area, which is accessed by a master computing node and a slave computing node; the master computing node sends binlog to the storage cluster for storage, including: the master computing node sends binlog to the log storage area for storage; the slave computing node reads the binlog stored in the storage cluster, including: the slave computing node reads the binlog from the log storage area; wherein the storage cluster also includes a data storage area, which is used to store business data, and the data storage area is accessed by the master computing node and the slave computing node, or the data storage area is only accessed by the master computing node.

In a possible implementation, replaying binlog from a computing node includes: replaying binlog from a computing node to synchronize data with a master computing node, or replaying binlog to recover data lost when a master computing node fails.

In a possible implementation, at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in the same data center or the same availability zone; the secondary computing node reads the binlog stored in the storage cluster, including: the secondary computing node reads the binlog from the log storage area during the normal operation of the primary computing node; the secondary computing node updates the data persistently stored in the storage cluster by replaying the binlog, including: the secondary computing node updates the data persistently stored in the secondary storage node by replaying the binlog.

In one possible implementation, at least one storage node includes a target storage node, and the target storage node is used to persistently store data written by the main computing node; the slave computing node reads the binlog stored in the storage cluster, including: when the main computing node fails, the slave computing node reads the binlog from the log storage area; the slave computing node updates the data persistently stored in the storage cluster by replaying the binlog, including: the slave computing node updates the data persistently stored in the target storage node by replaying the binlog.

In a possible implementation, at least one storage node includes a master storage node and a slave storage node, the slave storage node serves as a disaster recovery for the master storage node, the master storage node and the slave storage node are deployed in different data centers, or the master storage node and the slave storage node are deployed in different availability zones; the master computing node sends the binlog to the storage cluster for storage, including: the master computing node sends the binlog to the master storage node for storage; the slave computing node reads the binlog stored in the storage cluster, including: the slave computing node is specifically used to read the binlog stored in the slave storage node, and the binlog in the slave storage node is sent by the master storage node.

In a possible implementation, the method further includes: before the master storage node sends the binlog to the slave storage node, the master storage node sends the baseline data to the slave storage node for storage; the slave computing node updates the data persistently stored in the storage cluster by replaying the binlog, including: the slave computing node updates the baseline data stored in the slave storage node by replaying the binlog.

In a possible implementation, a target application is running on the main computing node, binlog is generated during the running of the target application, the target application includes a relational database management system RDBMS, and the RDBMS includes at least one of MySQL, PostgreSQL, OpenGauss, and Oracle.

In a possible implementation, the storage node is a storage array, and the storage array is used to persistently store data.

In a third aspect, an embodiment of the present application provides a data processing device, which is applied to a data processing system, wherein the data processing system includes a computing cluster and a storage cluster, the computing cluster and the storage cluster are connected through a network, the computing cluster includes a master computing node and a slave computing node, and the storage cluster includes at least one storage node; the data processing device includes: a storage module, used to instruct the master computing node to send the binary log binlog generated in response to the data update request to the storage cluster for storage; a reading module, used to instruct the slave computing node to read the binlog stored in the storage cluster to the slave computing node; and a playback module, used to instruct the slave computing node to update the data persistently stored in the storage cluster by replaying the binlog.

In one possible implementation, the storage cluster includes a log storage area, which is accessed by the master computing node and the slave computing node; a storage module, which is specifically used to instruct the master computing node to send binlog to the log storage area for storage; a reading module, which is specifically used to instruct the slave computing node to read binlog from the log storage area to the slave storage node; wherein the storage cluster also includes a data storage area, which is used to store business data, and the data storage area is accessed by the master computing node and the slave computing node, or the data storage area is only accessed by the master computing node.

In a possible implementation, the revisit module is specifically used to instruct the slave computing node to synchronize data with the master computing node by replaying the binlog, or is specifically used to instruct the slave computing node to recover data lost when the master computing node fails by replaying the binlog.

In a possible implementation, at least one storage node includes a master storage node and a slave storage node, wherein the slave storage node serves as the master storage node. For disaster recovery of storage nodes, the master storage node and the slave storage node are deployed in the same data center or the same availability zone; the reading module is specifically used to instruct the slave computing node to read the binlog from the log storage area to the slave computing node during the normal operation of the master computing node; the playback module is specifically used to instruct the slave computing node to update the data persistently stored in the slave storage node by replaying the binlog.

In one possible implementation, at least one storage node includes a target storage node, which is used to persistently store data written by the main computing node; a reading module, which is specifically used to instruct the slave computing node to read binlog from the log storage area when the main computing node fails; and a playback module, which is specifically used to instruct the slave computing node to update the data persistently stored in the target storage node by replaying the binlog.

In one possible implementation, at least one storage node includes a master storage node and a slave storage node, the slave storage node serves as a disaster recovery for the master storage node, the master storage node and the slave storage node are deployed in different data centers, or the master storage node and the slave storage node are deployed in different availability zones; a storage module is specifically used to instruct the master computing node to send the binlog to the master storage node for storage; a reading module is specifically used to instruct the slave computing node to read the binlog stored in the slave storage node, and the binlog in the slave storage node is sent by the master storage node.

In one possible implementation, the storage module is further used to: instruct the master storage node to send the baseline data to the slave storage node for storage before sending the binlog to the slave storage node; the playback module is specifically used to instruct the slave computing node to update the baseline data stored in the slave storage node by replaying the binlog.

In a fourth aspect, an embodiment of the present application provides a computing device cluster, the computing device cluster includes at least one computing device, each computing device in the at least one computing device includes: a processor and a memory; the memory is used to store instructions, and the processor executes the instructions stored in the memory so that the computing device cluster executes the data processing method described in the above second aspect or any implementation of the second aspect, or implements the data processing device described in the above third aspect or any implementation of the third aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via a bus. The memory may include a readable memory and a random access memory.

In a fifth aspect, an embodiment of the present application provides a chip, comprising a power supply circuit and a processing circuit, wherein the power supply circuit is used to power the processing circuit, and the processing circuit implements the data processing device described in the above third aspect or any implementation method of the third aspect.

In the sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program or instruction is stored. When the computer-readable storage medium is run on a computer, the data processing method described in the first device in the above-mentioned second aspect or any implementation of the second aspect is executed.

In the seventh aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data processing method described in the above second aspect or any implementation of the second aspect.

In addition, the technical effects brought about by any implementation method in the second to seventh aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

FIG1 is a schematic diagram of the structure of an exemplary data processing system provided in an embodiment of the present application;

FIG2 is a schematic diagram of the structure of another exemplary data processing system provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of another exemplary data processing system provided in an embodiment of the present application;

FIG4 is a schematic diagram of the structure of another exemplary data processing system provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of another exemplary data processing system provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of another exemplary data processing system provided in an embodiment of the present application;

FIG7 is a flow chart of an exemplary data processing method provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of an exemplary data processing device provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of the structure of an exemplary computing device provided in an embodiment of the present application.

Detailed ways

In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, various non-limiting implementation methods in the embodiments of the present application are exemplarily described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained based on the above content belong to the scope of protection of the present application.

Referring to Fig. 1, it is a schematic diagram of the structure of an exemplary data processing system 100, and the data processing system 100 may adopt a storage-computing separation architecture. As shown in Fig. 1, the data processing system 100 includes a computing cluster 101 and a storage cluster 102, and the computing cluster 101 and the storage cluster 102 may communicate with each other through a network, such as a wired network or a wireless network.

Among them, the computing cluster 101 includes multiple computing nodes, different computing nodes can communicate with each other, and each computing node can be a computing device including a processor, such as a server, a desktop computer, etc. In the computing cluster 101, some computing nodes can be used as disaster recovery for another part of the computing nodes. For ease of explanation, FIG1 takes the computing cluster 101 including the main computing node 1011 and the slave computing node 1012 as an example for exemplary explanation, and the slave computing node 1012 is used as the disaster recovery for the main computing node 1011. Exemplarily, the slave computing node 1012 can be used as a hot standby or a cold standby for the main computing node 1011. Among them, when the slave computing node 1012 is used as a hot standby, the slave computing node 1012 and the main computing node 1011 are continuously in operation; in this way, when the main computing node 1011 fails, the slave computing node 1012 can use the backup data to immediately take over the business on the main computing node 1011, specifically, to process the requests that the main computing node 1011 has not completed when it fails. When the slave computing node 1012 is used as a cold standby, during the normal operation of the master computing node 1011, the slave computing node 1012 may not be running (such as being in a dormant state, etc.), or the slave computing node 1012 may release the computing resources thereon and use the released computing resources to process other services, such as offline computing services, etc. When the master computing node 1011 fails, the slave computing node 1012 starts to run/recover computing resources, and uses the backed-up data to take over the services on the master computing node 1011. In actual application, the master computing node 1011 may have multiple slave computing nodes as disaster recovery nodes, so that some slave computing nodes can be used as cold standby for the master computing node 1011, and another part of the slave computing nodes can be used as hot standby for the master computing node 1011, etc.

The storage cluster 102 may include one or more storage nodes, each of which may be a device including a persistent storage medium, such as a network attached storage (NAS), a storage server, etc., which may be used to persistently store data. Among them, the persistent storage medium in the storage node may be, for example, a hard disk, such as a solid state disk or a shingled magnetic recording hard disk, etc. In actual application, each storage node may be constructed by one or more devices for persistently storing data. When the storage cluster 102 includes multiple storage nodes, some storage nodes may be used as disaster recovery for another part of the storage nodes. For ease of explanation, FIG1 takes the storage cluster 102 including a master storage node 1021 and a slave storage node 1022 as an example, and the master storage node 1021 is used as a disaster recovery for the slave storage node 1022, and the data storage areas on the master storage node 1021 and the slave storage node 1022 are respectively used to store business data, and the data storage area on the master storage node 1021 is accessed by the master computing node 1011, and the data storage area on the slave storage node 1022 is accessed by the slave computing node 1012. Among them, the master storage node 1021 and the slave storage node 1022 can be deployed in the same data center, or in the same availability zone (AZ), etc. At this time, by creating multiple copies of persistently stored data in the same data center or the same AZ, the reliability of data storage in the local area can be improved. Alternatively, the master storage node 1021 and the slave storage node 1022 can be deployed in different data centers, for example, the master storage node 1021 is deployed in data center A, and the slave storage node 1022 is deployed in data center B; or, the master storage node 1021 and the slave storage node 1022 can be deployed in different AZs, such as the master storage node 1021 is deployed in AZ ₁ , and the slave storage node 1022 is deployed in AZ ₂ , etc. In this way, data disaster recovery can be achieved across data centers or across AZs, thereby improving the reliability of data storage in a different location.

The master computing node 1011 uses the master storage node 1021 to provide data read and write services, and after the master computing node 1011 fails, the slave computing node 1012 takes over the business on the master computing node 1011 using the data backed up from the storage node 1022. In actual application scenarios, the master computing node 1011 and the master storage node 1021 can constitute a master center (usually belonging to a production site), and the slave computing node 1012 and the slave storage node 1022 can constitute a disaster recovery center (usually belonging to a disaster recovery site). In other possible data processing systems, the storage cluster 102 can also include a storage node. In this case, the master computing node 1011 and the slave computing node 1012 can share the storage node, that is, the business data stored in the data storage area on the storage node can be accessed by the master computing node 1011 and the slave computing node 1012, so that after the master computing node 1011 fails, the slave computing node 1012 can continue to provide data read and write services using the data stored in the storage node.

One or more applications (not shown in FIG. 1 ) may be deployed on the main computing node 1011. The deployed applications may be, for example, database applications or other applications. For example, the database application may be, for example, a relational database management system (RDBMS). Management system (RDBMS), etc., which RDBMS may include at least one of MySQL, PostgreSQL, OpenGauss, Oracle, or other types of database systems. During the application operation, the main computing node 1011 usually receives a data update request sent by a client or other device on the user side, such as receiving a data update request sent by a client on the user side for reading or modifying data in the main storage node 1021. At this time, the application on the main computing node 1011 can respond to the data update request and provide corresponding data read and write services for the client or other devices. Among them, when the data update request received by the main computing node 1011 is used to request to write new data to the data processing system 100, or to request to modify the data that has been persistently stored in the data processing system 100, or to request to delete the data that has been persistently stored in the data processing system 100, the application on the main computing node 1011 will generate a binary log (binlog) and save the binlog in a local storage area. Among them, binlog is a logical log, which is used to record database statements, such as SQL statements, for updating the data persistently stored in the main storage node 1021. In actual scenarios, the application may include a service layer and a storage engine layer, and the service layer may generate and save binlog. Then, the master computing node 1011 will send the generated binlog to the slave computing node 1012, and the slave computing node 1012 will update the data in the slave storage node 1022 by executing the database statements in the binlog, so that the data in the slave storage node 1022 is consistent with the data in the master storage node 1021, that is, the data in the master storage node 1021 is copied to the slave storage node 1022.

After the failure of the master computing node 1011, the slave computing node 1012 needs to run/reclaim computing resources and use the computing resources to start running the application on the slave computing node 1012. Then, the application on the slave computing node 1012 executes the database statements recorded in the binlog sent by the master computing node 1011 before the failure, so that the data in the slave storage node 1022 is consistent with the data before the failure of the storage node 1021. In this way, the slave computing node 1012 can take over the unfinished requests on the master computing node 1011 based on the data stored in the slave storage node 1022.

However, in actual application scenarios, some binlogs may exist in the master computing node 1011 and fail to be successfully transmitted to the slave computing node 1012. For example, when the data transmission link between the master computing node 1011 and the slave computing node 1012 is unstable (such as high data transmission jitter or high communication network transmission pressure), the binlog sent by the master computing node 1011 may be lost during the transmission process of the communication network, making it difficult for the slave computing node 1012 to receive the binlog. For another example, when the business load on the master computing node 1011 is large, the master computing node 1011 may have difficulty in sending multiple binlogs stored locally to the slave computing node 1012 in a timely manner while continuously generating new binlogs, and the storage space of the local storage area for storing binlogs in the master computing node 1011 is limited, which causes the master computing node 1011 to eliminate the first part of the binlogs stored in the local storage area in order to store new binlogs. At this time, since part of the binlog on the master computing node 1011 has not been sent to the slave computing node 1012, it is difficult for the slave computing node 1012 to synchronize the data in the master storage node 1021 with the data between the slave storage node 1022 by replaying the part of the binlog, that is, the data between the master center and the disaster recovery center are inconsistent. In this way, when the master computing node 1011 of the master center fails, since the disaster recovery center cannot restore the data to the state when the master computing node 1011 fails, the RPO of the data processing system 100 cannot reach 0, that is, some data is lost, affecting the reliability of the data processing system 100.

Based on this, in the data processing system 100 provided by the present application, after generating the binlog, the master computing node 1011 will send the binlog to the storage cluster 102, and the storage cluster 102 will store the binlog. Then, the slave computing node 1012 reads the binlog from the storage cluster 102, and updates the data persistently stored in the storage cluster 102 by replaying the binlog (i.e., replaying the database statements recorded in the binlog), specifically updating the data in the slave storage node 1021, so as to achieve the consistency of the data in the slave storage node 1022 with the data in the master storage node 1021, or with the data in the master storage node 1021 + the data cached by the master computing node 1011. Since the binlog generated by the master computing node 1011 is transmitted to the slave computing node 1012 through the storage side, this can avoid the overload of the master computing node 1011, or the instability of the data transmission link between the master computing node 1011 and the slave computing node 1012, resulting in the data synchronization problem caused by the slave computing node 1012 not obtaining the binlog generated by the master computing node 1011. In this way, when the main computing node 1011 fails, since the data in the slave storage node 1022 remains consistent with the data in the main storage node 1021, the slave computing node 1012 can take over the business on the main computing node 1011 based on the data in the slave storage node 1022, thereby achieving an RPO of 0 for the data processing system 100 and improving the reliability of the data processing system 100.

In addition, when the master computing node 1011 and the slave computing node 1012 simultaneously include multiple database applications such as MySQL, PostgreSQL, OpenGauss, and Oracle, a unified processing logic can be used between the master computing node 1011 and the slave computing node 1012 to copy the binlog generated by the master computing node 1011 to the slave computing node 1012, thereby improving the compatibility of the data processing system 100 with database applications and reducing the difficulty of deploying database applications on the computing cluster 101.

It is worth noting that the data processing system 100 shown in Fig. 1 is only an exemplary description, and in actual application, the data processing system 100 may also be implemented in other ways. For ease of understanding, this embodiment provides the following implementation examples.

In a first implementation example, the storage cluster 102 may include only one storage node for the master computing node 1011 and the slave computing node 1012 , so that the master computing node 1011 and the slave computing node 1012 can share access to data pages in the storage node.

In the second implementation example, a metadata management cluster may be included between the computing cluster 101 and the storage cluster 102, and the metadata management cluster is responsible for managing the metadata stored in the storage cluster 102; accordingly, the computing nodes in the computing cluster 101 may first access the metadata from the metadata management cluster, and then access the data stored in the storage cluster 102 based on the metadata.

In the third implementation example, the computing cluster and the storage cluster in the data processing system 200 may include three or more nodes, as shown in FIG2 . Specifically, the computing cluster includes a plurality of computing nodes 410, each of which can communicate with each other, and some computing nodes 410 can serve as disaster recovery for another computing node 410. Each computing node 410 is a computing device including a processor, such as a server, a desktop computer, etc. In terms of hardware, as shown in FIG2 , the computing node 410 includes at least a processor 412, a memory 413, a network card 414, and a storage medium 415. Among them, the processor 412 is a central processing unit (CPU) for processing data access requests from outside the computing node 410, or requests generated inside the computing node 410. The processor 412 reads data from the memory 413, or, when the total amount of data in the memory 413 reaches a certain threshold, the processor 412 sends the data stored in the memory 413 to the storage node 400 for persistent storage. FIG2 shows only one CPU 412. In actual applications, there are often multiple CPUs 412, wherein one CPU 412 has one or more CPU cores. This embodiment does not limit the number of CPUs or CPU cores. In addition, the processor 412 in the computing node 410 can also be used to implement the above-mentioned functions of writing binlog to the storage cluster and/or reading and replaying binlog from the storage cluster, so as to achieve data synchronization between different storage nodes 400 in the storage cluster.

Memory 413 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as a temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either a random access memory or a read-only memory (ROM). In actual applications, multiple memories 413 and different types of memories 413 can be configured in the computing node 410. This embodiment does not limit the number and type of memory 413.

The network card 414 is used to communicate with the storage node 400. For example, when the total amount of data in the memory 413 reaches a certain threshold, the computing node 410 can send a request to the storage node 400 through the network card 414 to store the data persistently. In addition, the computing node 410 can also include a bus for communication between components inside the computing node 410. In actual implementation, the computing node 410 can also have a small number of hard disks built in, or a small number of hard disks connected externally.

Each computing node 410 can access the storage node 400 in the storage cluster through the network. The storage cluster includes multiple storage nodes 400, and some storage nodes 400 can be used as disaster recovery for another part of the storage nodes 400. A storage node 400 includes one or more controllers 401, a network card 404 and multiple hard disks 405. The network card 404 is used to communicate with the computing node 410. The hard disk 405 is used for persistent storage of data, and can be a disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The controller 401 is used to write data to the hard disk 405 or read data from the hard disk 405 according to the read/write data request sent by the computing node 410. In the process of reading and writing data, the controller 401 needs to convert the address carried in the read/write data request into an address that the hard disk can recognize.

To facilitate understanding and explanation, the following is a detailed introduction to the process of achieving data synchronization between the main center (including the main computing node 1011 and the main storage node 1021) and the disaster recovery center (including the slave computing node 1012 and the slave storage node 1022) based on the data processing system 100 shown in Figure 1.

Normally, one or more applications (such as MySQL, etc.) are running on the main computing node 1011. For ease of understanding, the following is an example of running a target application on the main computing node 1011. When the target application is running, it can support the main computing node 1011 to provide data read and write services for users. Taking a user request to modify data as an example, after the main computing node 1011 receives a data update request for modifying data sent by the user through the client, the target application can first read the data page where the data requested to be modified by the data update request is located from the main storage node 1021 to the buffer pool in the main computing node 1011, and complete the modification of the data page in the buffer pool according to the data update request, specifically modifying the data on the data page to new data (the new data can be empty, in which case the data on the data page is deleted). At this time, the target application will generate a binlog for the data modification content, which is used to record the database statement indicating the modification of the data. For ease of distinction and description, the new data is referred to as target data below.

After completing the modification of the data in the buffer pool and generating the binlog, the master computing node 1011 can feedback to the client that the data has been written/modified successfully. Since the speed of writing data to the buffer pool is usually higher than the speed of persistently storing the data, this can speed up the master computing node 1011 to respond to data update requests. In actual application, when the amount of data accumulated in the buffer pool reaches a threshold, the master computing node 1011 sends the data in the buffer pool to the master storage node 1021 for persistent storage, and can delete the binlog corresponding to the data. Alternatively, the master The computing node 1011 may also feedback to the client that the data has been written/modified successfully, etc., after successfully sending the binlog to the storage cluster 102 for storage.

The data in the master storage node 1021 (and the slave storage node 1022) may be persistently stored in the format of a file. In this case, a corresponding file system (FS) may be deployed in the master storage node 1021 (and the slave storage node 1022), and the FS is used to manage the persistently stored files. Alternatively, the data in the master storage node 1021 (and the slave storage node 1022) may be persistently stored in the format of a data block. That is, when the master storage node 1021 (and the slave storage node 1022) stores data, the data is divided into blocks according to a fixed size. The data volume of each block may be, for example, 512 bytes or 4 kilobytes (KB). Alternatively, the data in the master storage node 1021 (and the slave storage node 1022) may be stored in the format of an object. In this case, an object may be the basic unit for storing data in a storage node. Each object may include a combination of data and the attributes of the data. The attributes of the data may be set according to the requirements of the application in the computing node, including data distribution, quality of service, etc. In this embodiment, the storage format of data is not limited.

In this embodiment, the following exemplary implementations are provided for the process in which the master computing node 1011 sends the generated binlog to the storage cluster 102 and stores the binlog in the storage cluster 102 .

In a first possible implementation, the master computing node 1011 may send the binlog to the master storage node 1021 in the storage cluster 102 so that the master storage node 1021 saves the binlog. In the storage cluster 102, since the slave storage node 1022 is a disaster recovery for the master storage node 1021, after the master computing node 1011 writes the binlog to the master storage node 1021, the master storage node 1021 may back up the written binlog and send the backed-up binlog to the slave storage node 1022 via wired or wireless means. When it is determined that the slave storage node 1022 has successfully written the backed-up binlog, the master storage node 1021 may feedback to the master computing node 1011 that the binlog has been written successfully.

Exemplarily, as shown in FIG3 , the master storage node 1021 and the slave storage node 1022 can be deployed in the same data center or the same AZ. At this time, a wired or wireless connection can be established between the master storage node 1021 and the slave storage node 1022, and the backed-up binlog can be sent to the slave storage node 1022 for storage through the wired or wireless connection.

Alternatively, as shown in FIG4 , the master storage node 1021 and the slave storage node 1022 may be deployed in different AZs, such as the master storage node 1021 is deployed in AZ ₁ and the slave storage node 1022 is deployed in AZ ₂ , etc. At this time, the master storage node 1021 may send the backed-up binlog to the slave storage node 1022 for storage via a network card or a network interface.

The implementations shown in Figures 3 and 4 above are only some exemplary explanations. For example, in other possible implementations, the master storage node 1021 may have multiple slave storage nodes as disaster recovery. In this case, some of the multiple slave storage nodes may be deployed in the same physical area as the master storage node 1021, such as in the same data center or the same AZ, and another part of the multiple slave storage nodes may be deployed in a different physical area from the master storage node 1021, such as in a different data center/AZ, etc., so as to improve the reliability of data storage both locally and remotely.

In this way, the computing node 1012 can read the binlog stored in the storage node 1022, and update the data stored in the storage node 1022 (specifically the data on the data page) by replaying the database statements recorded in the binlog, thereby realizing data synchronization between the data in the storage node 1022 and the main storage node 1021, or it can be called data synchronization between the disaster recovery center and the main center.

As an example, an input/output (IO) thread and a database thread (such as an SQL thread, etc.) can be created in the slave computing node 1012. Then, the slave computing node 1012 can use the IO thread to access the slave storage node 1022 to read the binlog in the slave storage node 1022 and store the read binlog in the local storage area. Then, the slave computing node 1012 can use the database thread to perform playback operations on each binlog in the local storage area in sequence. For example, each binlog in the local storage area can have a log sequence number (LSN), so that the database thread can play back each binlog in order from small to large LSN. Specifically, when playing back each binlog, the database thread can parse the database statement to be executed from the binlog, such as an SQL statement, and perform semantic analysis and grammatical analysis on the database statement to determine the legitimacy of the database statement. Among them, grammatical analysis refers to the grammatical rules of the database language to check whether the database statement has grammatical errors; semantic analysis refers to analyzing whether the semantics of the database statement is legal. After passing the legality check, the database thread can generate a plan tree for the database statement, which indicates the execution plan for processing the data. Finally, the database thread can update the data from the storage node 1022 according to the optimized plan tree after completing the optimization of the plan tree.

In actual application, the master computing node 1011 and the slave computing node 1012 may have the same configuration. For example, when creating a disaster recovery center, the configuration files in the master computing node 1011 and the master storage node 1021 may be backed up, and the backed-up configuration files may be sent to the slave computing node 1012 and the slave storage node 1022 in the disaster recovery center, so that the slave computing node 1012 may have the same configuration based on the received configuration files. The slave storage node 1022 has the same configuration as the master computing node 1012, and the slave storage node 1022 has the same configuration as the master storage node 1021 based on the received configuration file. In this way, when the master computing node 1011 sends the binlog to the master storage node 1021, the binlog in the master storage node 1021 can be mounted to the specified directory, so that after receiving the binlog sent by the master storage node 1021, the slave storage node 1022 can store the binlog to the storage location corresponding to the directory. In this way, the slave computing node 1012 can read the binlog belonging to the directory stored in the slave storage node 1022 based on the directory uniformly configured with the master computing node 1011.

In a second possible implementation, as shown in FIG5 , a log storage area 501 may be configured in the storage cluster 102, and the log storage area 501 can be accessed by the master computing node 1011 and the slave computing node 1012. Exemplarily, as shown in FIG5 , the log storage area 501 may be a partial storage area on the master storage node 1021 or the slave storage node 1022, or may be a storage area on other storage nodes independent of the master storage node 1021 and the slave storage node 1022, etc., which is not limited in this embodiment. At this time, the master computing node 1011 may send the binlog to the log storage area 501 in the storage cluster 102 for storage, for example, it may be sent to the log storage area 501 under the specified directory for storage. Then, the slave computing node 1012 may obtain the binlog generated by the master computing node 1011 by accessing the log storage area 501, for example, it may be accessing the corresponding log storage area 501 according to the specified directory to obtain the binlog, etc. Furthermore, the master computing node 1012 can achieve data synchronization between the master storage node 1021 and the slave storage node 1022 by replaying the binlog. The specific implementation process of the master computing node 1012 replaying the binlog can be found in the above-mentioned related description, which will not be repeated here.

It is worth noting that the above two implementation methods are only used as an exemplary description. In actual application, the slave computing node 1012 can also obtain the binlog generated by the master computing node 1011 from the storage cluster 102 in other ways.

In a further possible implementation, before the master computing node 1011 synchronizes the currently generated binlog to the slave computing node 1012 through the storage cluster 102, the master storage node 1021 and the slave storage node 1022 can also complete the baseline replication in advance. Among them, baseline replication refers to sending all the data persistently stored by the master storage node 1021 at a certain point in time (such as the current moment) and the binlog that the master computing node 1011 has generated to the disaster recovery center. For example, when creating the slave storage node 1022, the data synchronization process is performed for the first time between the master storage node 1021 and the slave storage node 1022. Then, the master storage node 1021 and the slave storage node 1022 can use the baseline replication method to achieve data synchronization.

In specific implementation, the main storage node 1021 can determine the first moment (such as the current moment) as the moment corresponding to the baseline, and determine the baseline data based on the moment. Among them, the baseline data includes the data persistently stored by the main storage node 1021 at the first moment (that is, the data stored on the data page), and the binlog generated by the main computing node 1011 before the first moment, that is, the binlog whose LSN is less than or equal to the LSN corresponding to the first moment. Usually, the updated data indicated by the binlog in the baseline data is stored in the buffer pool and has not yet been sunk to the main storage node 1021 for persistent storage. Then, the main storage node 1021 can send the baseline data to the slave storage node 1022 by wired or wireless means. After the slave storage node 1022 successfully stores the baseline data, the slave computing node 1012 can perform the process of replaying each binlog in the baseline data in sequence according to the order of LSN from small to large, so as to complete the update of the data belonging to the data page in the baseline data, thereby realizing the synchronization of the data stored by the main center at the first moment to the disaster recovery center. In this way, when the main computing node 1011 generates a binlog based on the data newly written by the user, the main computing node 1011 sends the binlog to the slave computing node 1012 through the storage cluster 102, and then the slave computing node 1012 replays the binlog to update the baseline data stored in the slave storage node 1022, thereby achieving timely synchronization of data between the main center and the disaster recovery center.

In actual application, when some binlogs in the main storage node 1021 are not sent to the slave storage node 1022 in time and are deleted in the main storage node 1021, such as the life cycle of this part of the binlog in the main storage node 1021 exceeds the preset time and is deleted, at this time, the slave storage node 1022 cannot obtain the deleted part of the binlog to maintain data synchronization with the main storage node 1021. Therefore, data synchronization can be achieved between the main storage node 1021 and the slave storage node 1022 by re-executing the baseline replication.

During the operation of the data processing system 100, the slave computing node 1012 can detect in real time or periodically whether the master computing node 1011 has a fault. For example, the slave computing node 1012 can determine whether the master computing node 1011 has a fault when it does not receive a heartbeat message sent by the master computing node 1011, or determine whether the master computing node 1011 has a fault when it receives a fault notification sent by a third-party arbitration server. When it is determined that the master computing node 1011 has a fault, the slave computing node 1012 is upgraded to the master computing node, and the storage node 1022 is instructed to upgrade to the master storage node. At this time, the slave computing node 1012 can first check whether there is any binlog that has not been fully replayed stored in the slave storage node 1022 or the log storage area 501. If so, the slave computing node 1012 will first read and replay the binlog to complete the update of the data persistently stored in the slave storage node 1022. The updated data is the data stored in the main center when the main computing node 101 fails; then, the slave computing node 1012 continues to take over the business on the main computing node 1011 based on the data in the storage node 1022.

In the process of taking over the business from the computing node 1012, when the user requests to perform operations on the data persistently stored in the data processing system 100, During the modification, the slave computing node 1012 can generate the corresponding binlog and write it to the slave storage node 1022 or the log storage area 501 in the storage cluster 102, so that the master computing node 1011 can synchronize data according to the binlog stored in the storage cluster 102 after the failure recovery. After the failure recovery, the master computing node 1011 can be used as a disaster recovery for the slave computing node 1012; or, the master computing node 1011 can be restored to the master node again through the master-slave switching, etc., which is not limited in this embodiment.

In this embodiment, since the binlog generated by the master computing node 1011 is transmitted to the slave computing node 1012 through the storage side, this can avoid the overload of the master computing node 1011, or the instability of the data transmission link between the master computing node 1011 and the slave computing node 1012, resulting in the data synchronization problem caused by the slave computing node 1012 not obtaining the binlog generated by the master computing node 1011. In this way, when the master computing node 1011 fails, since the data in the slave storage node 1022 is consistent with the data in the master storage node 1021, the slave computing node 1012 can take over the business on the master computing node 1011 based on the data in the slave storage node 1022, thereby achieving an RPO of 0 for the data processing system 100, and improving the reliability of the data processing system 100.

In addition, when the master computing node 1011 and the slave computing node 1012 include multiple database applications at the same time, a unified processing logic can be used between the master computing node 1011 and the slave computing node 1012 to copy the binlog generated by the master computing node 1011 to the slave computing node 1012, thereby improving the compatibility of the data processing system 100 with database applications and reducing the difficulty of deploying database applications on the computing cluster 101.

In actual application, the master storage node 1021 and the slave storage node 1022 can be implemented by a memory array, or can be implemented by a device including the memory array, so that the master storage node 1021 and the slave storage node 1022 can persistently store data based on the memory array, and can further improve the reliability of persistent storage of data in the master storage node 1021 and the slave storage node 1022 based on the memory array based on Redundant Arrays of Independent Disks (RAID) technology, erasure coding (EC) technology, deduplication and compression, data backup and other technologies.

In the data processing system 100 shown in FIGS. 1 to 5 above, the master computing node 1011 and the slave computing node 1012 are respectively configured with their own storage nodes, and the master storage node 1021 can only be accessed by the master computing node 1011, and the slave storage node 1022 can only be accessed by the slave computing node 1012. In other possible data processing systems, the master computing node 1011 and the slave computing node 1012 can also share the same storage node, that is, the data on the data page in the storage node is allowed to be accessed by the master computing node 1011 and the slave computing node 1012. The data processing system is described in detail below in conjunction with FIG. 6.

Referring to Figure 6, a schematic diagram of the structure of another data processing system provided by the present application is shown. As shown in Figure 6, the data processing system 600 still adopts a storage-computing separation structure, including a computing cluster 601 and a storage cluster 602, wherein the computing cluster 601 and the storage cluster 602 can communicate with each other through a network.

The computing cluster 601 includes multiple computing nodes. For ease of description, FIG6 takes the computing cluster 601 including a master computing node 6011 and a slave computing node 6012 as an example for illustrative explanation, and the slave computing node 1012 serves as a disaster recovery for the master computing node 1011, which can be a hot backup or a cold backup.

The storage cluster 602 includes at least one storage node, and FIG. 6 takes the storage node 6021 as an example, and the storage cluster 602 also includes a log storage area 6022. As shown in FIG. 6, the log storage area 6022 can be deployed in the storage node 6021, or it can be deployed independently of the storage node 6021, such as deployed in other storage nodes in the storage cluster 602, etc., and this embodiment does not limit this. In addition, the storage node 6021 also includes a data storage area (not shown in FIG. 6), and the data storage area and the log storage area 6022 can be accessed by the main computing node 6011 and the slave computing node 6012. In addition, the storage node 6021 is used to use the data storage area to persistently store data, such as persistently storing the data generated by the main computing node 6011 when processing business. The log storage area 6022 is used to store the binlog generated by the main computing node 6011 during operation.

After receiving a data update request for requesting to update the data persistently stored in the storage node 6021 (including data addition, deletion, modification, etc.), the master computing node 6011 can read the data page where the data requested to be modified by the data update request is located from the storage node 6021 to the buffer pool in the master computing node 6011, and complete the modification of the data page in the buffer pool according to the data update request, and generate a binlog for the data modification content. Then, the master computing node 6011 can feedback the data update success to the user side, and send the generated binlog to the log storage area 6022 in the storage cluster 602 for storage (and send the updated data page to the storage node 6021 for persistent storage); or, the master computing node 6011 can send the generated binlog to the log storage area 6022 in the storage cluster 602 for storage, and feedback the data update success to the user side, etc., which is not limited in this embodiment.

It is worth noting that since the master computing node 6011 and the slave computing node 6012 share the data page in the storage node 6021, after the master computing node 6011 successfully writes the binlog into the log storage area 6022, if the master computing node 6011 operates normally, the slave computing node 6012 It is not necessary to read and replay the binlog in the log storage area 6022. In actual application, when the main computing node 6011 writes the data in the buffer pool to the storage node 6021, the binlog corresponding to the data in the buffer pool can be eliminated from the log storage area 6022.

The slave computing node 6012 can detect in real time or periodically whether the master computing node 6011 fails, and when it is determined that the master computing node 6011 fails, the slave computing node 6012 is upgraded to the master node, and detects whether there is a binlog in the log storage area 6022. If there is, it indicates that when the master computing node 6011 fails, there may be new data in the buffer pool of the master computing node 6011 that has not been persistently stored in the storage node 6021. At this time, the slave computing node 6012 can read the binlog in the log storage area 6022, and update the data in the storage node 6021 by replaying the binlog to restore the data cached in the buffer pool of the master computing node 6011 when the failure occurs, thereby realizing the restoration of the data in the data processing system 600 to the state when the master computing node 6011 fails, that is, realizing the RPO of the data processing system 600 to be 0.

It is worth noting that, when the slave computing node 6012 plays back the binlog in the log storage area 6022, it can first read the binlog in the log storage area 6022 to the local storage space of the slave computing node 6012, and then perform the playback operation on each binlog in the local storage space in order of LSN from small to large. Alternatively, the slave computing node 6012 can also directly read each binlog in the log storage area 6022 in order of LSN from small to large, and directly perform the playback operation on the read binlog. In this way, the resource consumption required for playing back the binlog can be further reduced, the data recovery delay can be reduced, and the recovery point objective (RTO) of the data processing system 600 can be reduced. Among them, RTO refers to the time interval between the moment when the data processing system 600 business is suspended and the moment when the data processing system 600 resumes business after the disaster occurs.

It should be noted that in the data processing system shown in Figures 1 to 6 above, the operations performed by the master computing node and the slave computing node can be implemented by an application deployed thereon, which application can be, for example, the above-mentioned database application such as MySQL, PostgreSQL, OpenGauss, or Oracle, or can be other applications.

In this way, by updating the versions of existing applications deployed on the master computing node and the slave computing node, the binlog can be transmitted from the master computing node to the slave computing node through the storage side, thereby realizing data synchronization between the main center and the disaster recovery center.

Alternatively, the operations performed by the master computing node and the slave computing node may also be performed by a data processing device deployed separately in the computing cluster, that is, the master computing node may write the generated binlog to the storage cluster under the control of the data processing device, and the slave computing node may read the binlog from the storage cluster and replay the binlog under the control of the data processing device.

By way of example, the data processing device may be implemented by software or hardware.

Among them, when implemented by software, the data processing device can be, for example, a program code deployed on a hardware device. In actual application, the data processing device can be, for example, deployed in the main computing node and/or the slave computing node in the form of software such as a plug-in, component or application (for example, deployed in the controller of the main computing node and/or the slave computing node). At this time, by deploying the data processing device on the main computing node and/or the slave computing node, the transmission of binlog can be completed between the main computing node and the slave computing node through the storage side, which can reduce or eliminate the need to modify the database application deployed on the main computing node and the slave computing node, reducing the difficulty of implementing the solution.

Alternatively, the above-mentioned data processing device can be implemented by a physical device, wherein the physical device can be, for example, a CPU, or can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), or any other processor or any combination thereof, and this embodiment does not limit this.

The above, in combination with Figures 1 to 6, introduces the process of realizing data synchronization between the main center and the disaster recovery center in the data processing system by transmitting binlog on the storage side. Below, in combination with Figure 7, an exemplary description of the process of realizing data synchronization between the main center and the disaster recovery center is given from the perspective of the method flow. Referring to Figure 7, a flow chart of a data processing method provided by an embodiment of the present application is shown. For ease of understanding, Figure 7 is used as an example to illustrate the data processing system 100 shown in Figure 1. As shown in Figure 7, the method may specifically include:

S701: The main computing node 1011 receives a data update request, where the data update request is used to request the data processing system 100 to update persistently stored data.

In actual application, the main computing node 1011 can receive a data update request sent by a client or other device on the user side. The data update request can be used to request modification of data persistently stored in the data processing system 100, or can be used to request writing new data to the data processing system 100, etc.

S702: The main computing node 1011 responds to the data update request, completes the data update in the buffer pool, and generates a corresponding binlog for the data update request.

S703: The main computing node 1011 sends the binlog to the storage cluster 102 for storage.

In the first implementation example, the storage cluster 102 includes a master storage node 1021 and a slave storage node 1022, wherein the master storage node 1021 supports the data reading and writing of the master computing node 1011, and the slave storage node 1022 supports the data reading and writing of the slave computing node 1012. In this way, the master computing node 1011 can write the binlog to the master storage node 1021, and the master storage node 1021 backs up the binlog, and then sends the backed-up binlog to the slave storage node 1022 for storage. The master storage node 1021 and the slave storage node 1022 can be deployed in the same physical area (such as the same data center/AZ), or can be deployed in different physical areas.

In the second implementation example, a log storage area, such as the above-mentioned log storage area 501, can be deployed in the storage cluster 102, so that the master computing node 1011 can write the generated binlog into the log storage area, and the log storage area can be accessed by the master computing node 1011 and the slave computing node 1012.

S704 : The slave computing node 1012 reads the binlog from the storage cluster 102 .

In specific implementation, the slave computing node 1012 may read binlogs etc. from the slave storage node 1022 or the log storage area in the storage cluster 102 .

The slave computing node 1012 may have the same configuration as the master computing node 1011. For example, in the process of creating the slave computing node 1012, the master computing node 1011 may back up its own configuration files and other data, and send the backed-up data to the slave computing node 1012, so that the slave computing node 1012 completes the corresponding configuration according to the received backup data, such as configuring the logic of processing services, the application running on the slave computing node 1012, and the directory where the binlog in the storage cluster 102 is mounted.

S705 : Play back the read binlog from the computing node 1012 to update the data persistently stored in the slave storage node 1022 in the storage cluster 102 .

In the data processing system 100 shown in Figure 1, the slave computing node 1012 can specifically read the binlog from the storage node 1022 or the log storage area, and by replaying the binlog, the data in the slave storage node 1022 can be kept in a synchronized state with the data in the main center. In specific implementation, the slave computing node 1012 can parse database statements, such as SQL statements, from the binlog, and perform semantic analysis and grammatical analysis on the database statements to determine the legitimacy of the database statements. After passing the legitimacy check, the slave computing node 1012 can generate a plan tree for the database statement, which indicates an execution plan for processing the data. Finally, after completing the optimization of the plan tree, the slave computing node 1012 can implement the update of the data in the slave storage node 1022 according to the optimized plan tree.

In this way, after the main computing node 1011 fails, the slave computing node 1012 can take over the business on the main computing node 1011 by using the data in the storage node 1022 that is synchronized with the main center, thereby realizing fault recovery of the data processing system 100.

Among them, before replaying the binlog from the computing node 1012, the master computing node 1011 can control the master storage node 1021 to send baseline data to the slave storage node 1022 to complete the baseline replication. The specific implementation process of the baseline replication can be found in the relevant description of the aforementioned embodiment and will not be repeated here.

In other possible data processing systems, such as the data processing system 600 shown in FIG6 , the master computing node 1011 and the slave computing node 1012 may share the same storage node, which is referred to as the target storage node below. Then, when the master computing node 1011 is in normal operation, for the binlog written by the master computing node 1011 into the storage cluster 102, the slave computing node 1012 does not need to perform the operation of reading the binlog from the storage cluster 102 and replaying the binlog. At this time, the binlog stored in the storage cluster 102 is the binlog corresponding to the updated data stored in the buffer pool of the master computing node 1011; when the data in the buffer pool is written into the storage cluster 102, the binlog corresponding to the data in the buffer pool can be eliminated from the storage cluster 102. When the main computing node 1011 fails, the data cached in the buffer pool of the main computing node 1011 may be lost due to the failure of the main computing node 1011 because it has not yet completed persistent storage. At this time, the slave computing node 1012 reads the binlog from the storage cluster 102 and performs the operation of replaying the binlog to update the data persistently stored in the target storage node, thereby restoring the data in the buffer pool of the main computing node 1011 that has not yet completed persistent storage in the target storage node.

It should be noted that steps S701 to S705 shown in FIG. 7 correspond to the system embodiments shown in FIG. 1 to FIG. 6 above. Therefore, the specific implementation process of steps S701 to S705 can be found in the relevant description of the aforementioned embodiments and will not be repeated here.

The data processing system provided by the present application is described in detail above in conjunction with Figures 1 to 7. The data processing apparatus and data processing device provided by the present application will be described below in conjunction with Figures 8 and 9.

With the same inventive concept as the above method, the embodiment of the present application also provides a data processing device. A schematic diagram of a data processing device provided in the embodiment. The data processing device 800 shown in FIG8 is located in a data processing system, such as the data processing system 100 shown in FIG1 , the data processing system 600 shown in FIG6 , etc. The data processing system includes a computing cluster and a storage cluster, the computing cluster and the storage cluster are connected via a network, the computing cluster includes a master computing node and a slave computing node, the storage cluster includes at least one storage node, and usually, the slave computing node serves as a disaster recovery for the master computing node.

As shown in FIG8 , the data processing device 800 includes:

The storage module 801 is used to instruct the main computing node to send the binary log binlog generated in response to the data update request to the storage cluster for storage;

A reading module 802 is used to instruct the slave computing node to read the binlog stored in the storage cluster to the slave computing node;

The playback module 803 is used to instruct the slave computing node to update the persistently stored data in the storage cluster by replaying the binlog.

In one possible implementation, the storage cluster includes a log storage area, and the log storage area is accessed by the master computing node and the slave computing node;

Storage module 801 is specifically used to instruct the main computing node to send the binlog to the log storage area for storage;

The reading module 802 is specifically used to instruct the slave computing node to read the binlog from the log storage area to the slave storage node; wherein, the storage cluster also includes a data storage area, the data storage area is used to store business data, the data storage area is accessed by the master computing node and the slave computing node, or the data storage area is only accessed by the master computing node.

In a possible implementation, the revisit module 803 is specifically used to instruct the slave computing node to synchronize data with the master computing node by replaying the binlog, or is specifically used to instruct the slave computing node to recover data lost when the master computing node fails by replaying the binlog.

In a possible implementation, at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in the same data center or the same availability zone;

The reading module 802 is specifically used to instruct the slave computing node to read the binlog from the log storage area to the slave computing node during the normal operation of the master computing node;

The playback module 803 is specifically used to instruct the slave computing node to update the data persistently stored in the slave storage node by replaying the binlog.

In a possible implementation, the at least one storage node includes a target storage node, and the target storage node is used to persistently store data written by the primary computing node;

The reading module 802 is specifically used to instruct the slave computing node to read the binlog from the log storage area when the master computing node fails;

The playback module 803 is specifically used to instruct the slave computing node to update the persistently stored data in the target storage node by replaying the binlog.

In a possible implementation, at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in different data centers, or the primary storage node and the secondary storage node are deployed in different availability zones;

Storage module 801 is specifically used to instruct the main computing node to send the binlog to the main storage node for storage;

The reading module 802 is specifically used to instruct the slave computing node to read the binlog stored in the slave storage node, where the binlog in the slave storage node is sent by the master storage node.

In a possible implementation, the storage module 801 is further used to: instruct the master storage node to send the baseline data to the slave storage node for storage before sending the binlog to the slave storage node;

The playback module 803 is specifically used to instruct the slave computing node to update the baseline data stored in the slave storage node by replaying the binlog.

The data processing device 800 provided in this embodiment corresponds to the data processing system in the above-mentioned embodiments, and is used to implement the data processing process performed in the above-mentioned embodiments. Therefore, the functions of each module in this embodiment and the technical effects thereof can be found in the relevant descriptions in the above-mentioned embodiments, and will not be elaborated here.

In addition, the present application embodiment further provides a computing device. As shown in FIG9 , the computing device 900 may include a communication interface 910 and a processor 920. Optionally, the computing device 900 may also include a memory 930. The memory 930 may be disposed in the computing device 900. The part may also be arranged outside the computing device 900. Exemplarily, each action that the data processing device instructs the master computing node and the slave computing node (and the master storage node) to perform in the above-mentioned embodiment can be implemented by the processor 920. In the implementation process, each step of the processing flow can complete the method in the above-mentioned embodiment by the hardware integrated logic circuit in the processor 920 or the instruction in the form of software. For the sake of brevity, it will not be repeated here. The program code executed by the processor 920 to implement the above-mentioned method can be stored in the memory 930. The memory 930 is connected to the processor 920, such as a coupling connection.

Some features of the embodiments of the present application may be completed/supported by the processor 920 executing program instructions or software codes in the memory 930. The software components loaded on the memory 930 may be summarized functionally or logically, for example, the functions of the storage module 801 and the playback module 803 shown in FIG8 , and the reading module 802 shown in FIG8 may be implemented by the communication interface 910.

Any communication interface involved in the embodiments of the present application may be a circuit, a bus, a transceiver or any other device that can be used for information exchange. For example, the communication interface 910 in the computing device 900, illustratively, the other device may be a device connected to the computing device 900, etc.

An embodiment of the present application also provides a computing device cluster, which may include one or more computing devices, each of which may have a hardware structure of a computing device 900 as shown in FIG. 9 , and during operation, the computing device cluster can be used to implement the data processing method in the embodiment shown in FIG. 7 .

Based on the above embodiments, the embodiments of the present application further provide a chip, including a power supply circuit and a processing circuit, wherein the power supply circuit is used to supply power to the processing circuit, and the processing circuit is used to implement the functions of the data processing device 800 shown in FIG. 8 .

The processor involved in the embodiments of the present application may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, modules or modules, which can be electrical, mechanical or other forms, and is used for information exchange between devices, modules or modules.

The processor may operate in conjunction with a memory. The memory may be a non-volatile memory, such as a hard disk or a solid-state drive, or a volatile memory, such as a random access memory. The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

The specific connection medium between the communication interface, processor and memory is not limited in the embodiments of the present application. For example, the memory, processor and communication interface may be connected via a bus. The bus may be divided into an address bus, a data bus, a control bus, etc.

Based on the above embodiments, the embodiments of the present application further provide a computer storage medium, in which a software program is stored, and when the software program is read and executed by one or more computing devices, the method performed by the data processing device 102 provided in any one or more of the above embodiments can be implemented. The computer storage medium may include: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and other media that can store program codes.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

The present application is described with reference to the flowchart and/or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, and the combination of the process and/or box in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions can also be loaded into a computer or other programmable device so that the computer or other programmable device can execute the program. A series of operation steps are performed to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows in the flowchart and/or one or more blocks in the block diagram.

The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

A data processing system, characterized in that the data processing system comprises a computing cluster and a storage cluster, the computing cluster and the storage cluster are connected via a network, the computing cluster comprises a master computing node and a slave computing node, and the storage cluster comprises at least one storage node;

The master computing node is used to generate a binary log binlog in response to a data update request, and send the binlog to the storage cluster for storage;

The slave computing node is used to read the binlog stored in the storage cluster, and update the data persistently stored in the storage cluster by replaying the binlog.
The data processing system according to claim 1, characterized in that the storage cluster includes a log storage area, and the log storage area is accessed by the master computing node and the slave computing node;

The main computing node is specifically used to send the binlog to the log storage area for storage;

The slave computing node is specifically used to read the binlog from the log storage area;

The storage cluster further includes a data storage area, which is used to store business data. The data storage area is accessed by the master computing node and the slave computing node, or the data storage area is only accessed by the master computing node.
The data processing system according to claim 2 is characterized in that the slave computing node is specifically used to synchronize data with the master computing node by replaying the binlog, or to recover data lost when the master computing node fails by replaying the binlog.
The data processing system according to claim 3, characterized in that the at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in the same data center or the same availability zone;

The slave computing node is specifically used to read the binlog from the log storage area during the normal operation of the master computing node, and update the data persistently stored in the slave storage node by replaying the binlog.
The data processing system according to claim 3, characterized in that the at least one storage node includes a target storage node, and the target storage node is used to persistently store the data written by the main computing node;

The slave computing node is specifically used to read the binlog from the log storage area when the master computing node fails, and to update the data persistently stored in the target storage node by replaying the binlog.
The data processing system according to claim 1, characterized in that the at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in different data centers, or the primary storage node and the secondary storage node are deployed in different availability zones;

The main computing node is specifically used to send the binlog to the main storage node for storage;

The master storage node is used to send the binlog to the slave storage node;

The slave computing node is specifically used to read the binlog stored by the slave storage node.
The data processing system according to claim 6, characterized in that

The master storage node is further configured to send the baseline data to the slave storage node before sending the binlog to the slave storage node;

The slave storage node is used to store the baseline data;

The slave computing node is specifically used to update the baseline data stored in the slave storage node by replaying the binlog.
The data processing system according to any one of claims 1 to 7 is characterized in that a target application is running on the main computing node, the binlog is generated during the running of the target application, the target application includes a relational database management system RDBMS, and the RDBMS includes at least one of MySQL, PostgreSQL, OpenGauss, and Oracle.
The data processing system according to any one of claims 1 to 8, characterized in that the storage node is a storage array, and the storage array is used to persistently store data.
A data processing method, characterized in that the method is applied to a data processing system, the data processing system includes a computing cluster and a storage cluster, the computing cluster and the storage cluster are connected via a network, the computing cluster includes a master computing node and a slave computing node, and the storage cluster includes at least one storage node; the method includes:

The main computing node generates a binary log binlog in response to the data update request;

The main computing node sends the binlog to the storage cluster for storage;

The slave computing node reads the binlog stored in the storage cluster;

The slave computing node updates the data persistently stored in the storage cluster by replaying the binlog.
The data processing method according to claim 10, characterized in that the storage cluster includes a log storage area, and the log storage area is accessed by the master computing node and the slave computing node;

The main computing node sends the binlog to the storage cluster for storage, including:

The main computing node sends the binlog to the log storage area for storage;

The reading the binlog stored in the storage cluster from the computing node includes:

The slave computing node reads the binlog from the log storage area;

The storage cluster further includes a data storage area, which is used to store business data. The data storage area is accessed by the master computing node and the slave computing node, or the data storage area is only accessed by the master computing node.
The data processing method according to claim 11, characterized in that the slave computing node plays back the binlog, comprising:

The slave computing node synchronizes data with the master computing node by replaying the binlog, or is used to recover data lost when the master computing node fails by replaying the binlog.
The data processing method according to claim 12, characterized in that the at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in the same data center or the same availability zone;

The reading the binlog stored in the storage cluster from the computing node includes:

The slave computing node reads the binlog from the log storage area during the normal operation of the master computing node;

The slave computing node updates the persistently stored data in the storage cluster by replaying the binlog, including:

The slave computing node updates the data persistently stored in the slave storage node by replaying the binlog.
The data processing method according to claim 12, characterized in that the at least one storage node includes a target storage node, and the target storage node is used to persistently store the data written by the main computing node;

The reading the binlog stored in the storage cluster from the computing node includes:

When the master computing node fails, the slave computing node reads the binlog from the log storage area;

The slave computing node updates the persistently stored data in the storage cluster by replaying the binlog, including:

The slave computing node updates the data persistently stored in the target storage node by replaying the binlog.
The data processing method according to claim 10, characterized in that the at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in different data centers, or the primary storage node and the secondary storage node are deployed in different availability zones;

The main computing node sends the binlog to the storage cluster for storage, including:

The main computing node sends the binlog to the main storage node for storage;

The reading the binlog stored in the storage cluster from the computing node includes:

The slave computing node is specifically used to read the binlog stored in the slave storage node, and the binlog in the slave storage node is sent by the master storage node.
The data processing method according to claim 15, characterized in that the method further comprises:

The master storage node sends the baseline data to the slave storage node for storage before sending the binlog to the slave storage node;

The slave computing node updates the persistently stored data in the storage cluster by replaying the binlog, including:

The slave computing node updates the baseline data stored in the slave storage node by replaying the binlog.
The data processing method according to any one of claims 10 to 16 is characterized in that a target application is running on the main computing node, the binlog is generated during the running of the target application, the target application includes a relational database management system RDBMS, and the RDBMS includes at least one of MySQL, PostgreSQL, OpenGauss, and Oracle.
The data processing method according to any one of claims 10 to 17 is characterized in that the storage node is a storage array, and the storage array is used to persistently store data.
A data processing device, characterized in that the data processing device is applied to a data processing system, the data processing system includes a computing cluster and a storage cluster, the computing cluster and the storage cluster are connected via a network, the computing cluster includes a main computing cluster and a storage cluster. Node, slave computing node, the storage cluster includes at least one storage node; the data processing device includes:

A storage module, used to instruct the master computing node to send a binary log binlog generated in response to a data update request to the storage cluster for storage;

A reading module, used for instructing the slave computing node to read the binlog stored in the storage cluster to the slave computing node;

The playback module is used to instruct the slave computing node to update the data persistently stored in the storage cluster by playing back the binlog.
The data processing device according to claim 19, characterized in that the storage cluster includes a log storage area, and the log storage area is accessed by the master computing node and the slave computing node;

The storage module is specifically used to instruct the main computing node to send the binlog to the log storage area for storage;

The reading module is specifically used to instruct the slave computing node to read the binlog from the log storage area to the slave storage node;

The storage cluster further includes a data storage area, which is used to store business data. The data storage area is accessed by the master computing node and the slave computing node, or the data storage area is only accessed by the master computing node.
The data processing device according to claim 20 is characterized in that the revisit module is specifically used to instruct the slave computing node to synchronize data with the master computing node by replaying the binlog, or is specifically used to instruct the slave computing node to recover data lost when the master computing node fails by replaying the binlog.
The data processing device according to claim 21, characterized in that the at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in the same data center or the same availability zone;

The reading module is specifically used to instruct the slave computing node to read the binlog from the log storage area to the slave computing node during the normal operation of the master computing node;

The playback module is specifically used to instruct the slave computing node to update the data persistently stored in the slave storage node by replaying the binlog.
The data processing device according to claim 21, characterized in that the at least one storage node includes a target storage node, and the target storage node is used to persistently store the data written by the main computing node;

The reading module is specifically used to instruct the slave computing node to read the binlog from the log storage area when the master computing node fails;

The playback module is specifically used to instruct the slave computing node to update the data persistently stored in the target storage node by replaying the binlog.
The data processing device according to claim 19, characterized in that the at least one storage node includes a primary storage node and a secondary storage node, the secondary storage node serves as a disaster recovery for the primary storage node, and the primary storage node and the secondary storage node are deployed in different data centers, or the primary storage node and the secondary storage node are deployed in different availability zones;

The storage module is specifically used to instruct the main computing node to send the binlog to the main storage node for storage;

The reading module is specifically used to instruct the slave computing node to read the binlog stored in the slave storage node, where the binlog in the slave storage node is sent by the master storage node.
The data processing device according to claim 24, characterized in that the storage module is further used for:

Instructing the master storage node to send the baseline data to the slave storage node for storage before sending the binlog to the slave storage node;

The playback module is specifically used to instruct the slave computing node to update the baseline data stored in the slave storage node by replaying the binlog.
The data processing device according to any one of claims 19 to 25 is characterized in that a target application is running on the main computing node, the binlog is generated during the running of the target application, the target application includes a relational database management system RDBMS, and the RDBMS includes at least one of MySQL, PostgreSQL, OpenGauss, and Oracle.
The data processing device according to any one of claims 19 to 26 is characterized in that the storage node is a storage array, and the storage array is used to persistently store data.
A computing device cluster, characterized in that the computing device cluster includes at least one computing device, and each computing device in the at least one computing device includes a processor and a memory;

The processor is configured to execute instructions stored in the memory, so that the computing device cluster executes the method according to any one of claims 10 to 18.
A computer-readable storage medium, characterized in that it includes instructions, wherein the instructions are used to implement the method according to any one of claims 10 to 18.
A computer program product comprising instructions, characterized in that when the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 10 to 18.