CN116450648A

CN116450648A - Data storage method and system

Info

Publication number: CN116450648A
Application number: CN202310327855.4A
Authority: CN
Inventors: 章颖强; 陈浩
Original assignee: Alibaba Cloud Computing Ltd
Current assignee: Alibaba Cloud Computing Ltd
Priority date: 2023-03-24
Filing date: 2023-03-24
Publication date: 2023-07-18

Abstract

The embodiment of the specification provides a data storage method and a system, wherein the data storage method comprises the following steps: acquiring data to be processed; carrying out data processing on the data to be processed to obtain log data; and sending the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and storing the data processing result and the log data, wherein a persistent storage medium is configured on the memory node. By separating the computing node and the memory node of the architecture, the memory node serves as a node which abstracts the memory characteristic independently, provides data storage service for the computing node, has good expansibility, and transmits log data to the memory node after finishing data processing on the computing node, so that the memory node plays back the log data, restores the data processing result, reduces the data quantity transmitted to the memory node, fully utilizes the limited bandwidth, reduces the blocking probability and improves the efficiency and stability of data storage.

Description

Data storage method and system

Technical Field

The embodiment of the specification relates to the technical field of computers, in particular to a data storage method.

Background

With the development of computer technology, multi-tier computing storage architectures represented by "processor + dynamic storage media + hard disk" have changed tremendously with the advent of persistent storage media (PersistentMemory, PM). The application of the persistent storage medium brings about various improvements to the database system: compared with a dynamic storage medium typified by a DRAM (dynamic random access memory), a persistent storage medium has a larger storage capacity and a persistent storage content; compared with a hard disk typified by an SSD (solid state disk), a persistent storage medium provides byte addressing capability, and is faster in read and write speed.

At present, the memory in the computing node is directly replaced by a persistent storage medium, and the expansibility is insufficient due to the limitation of the hardware quantity of the computing node. The distributed persistent storage nodes are constructed by utilizing the persistent storage medium, and as a large number of data read-write operations exist in the computing nodes and a large number of log read-write operations exist in the computing nodes, the distributed persistent storage nodes are extremely high in data read-write pressure for the limited bandwidth between the computing nodes and the persistent storage nodes, the blockage is easy to cause, and the data storage efficiency and the stability are insufficient. Therefore, there is a need for a data storage method that can be highly scalable, efficient, and stable.

Disclosure of Invention

In view of this, the present specification embodiments provide a data storage method. One or more embodiments of the present specification relate to a data storage system, a data storage device, a computing apparatus, a computer-readable storage medium, and a computer program.

According to a first aspect of embodiments of the present disclosure, there is provided a data storage method applied to a computing node in a data storage system, the data storage system further including an independently configured memory node, including:

acquiring data to be processed;

carrying out data processing on the data to be processed to obtain log data;

and sending the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and storing the data processing result and the log data, wherein a persistent storage medium is configured on the memory node.

According to a second aspect of embodiments of the present disclosure, there is provided a data storage method applied to an independently configured memory node in a data storage system, where the memory node is configured with a persistent storage medium, including:

receiving log data sent by a computing node;

performing log playback on the log data to obtain a data processing result;

And storing the data processing result and the log data.

According to a third aspect of embodiments of the present specification, there is provided a data storage system comprising: a computing node and a memory node;

the computing node is used for acquiring data to be processed, performing data processing on the data to be processed to obtain log data, and sending the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and storing the data processing result and the log data, wherein a persistent storage medium is configured on the memory node;

the memory node is used for receiving the log data sent by the computing node, carrying out log playback on the log data to obtain a data processing result, and storing the data processing result and the log data.

According to a fourth aspect of embodiments of the present disclosure, there is provided a data storage device for use in a computing node in a data storage system, the data storage system further comprising a separately provided memory node, comprising:

the acquisition module is configured to acquire data to be processed;

the processing module is configured to perform data processing on the data to be processed to obtain log data;

the first sending module is configured to send the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and stores the data processing result and the log data, wherein a persistent storage medium is configured on the memory node.

According to a fifth aspect of embodiments of the present disclosure, there is provided a data storage device for use in an independently configured memory node in a data storage system, the memory node having a persistent storage medium disposed thereon, comprising:

the receiving module is configured to receive log data sent by the computing node;

the playback module is configured to carry out log playback on the log data to obtain a data processing result;

and the first storage module is configured to store the data processing result and the log data.

According to a sixth aspect of embodiments of the present specification, there is provided a computing device comprising:

a memory and a processor;

the memory is configured to store computer-executable instructions that, when executed by the processor, perform the steps of the data storage method described above.

According to a seventh aspect of embodiments of the present specification, there is provided a computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the data storage method described above.

According to an eighth aspect of embodiments of the present specification, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the data storage method described above.

In one or more embodiments of the present disclosure, data to be processed is obtained; carrying out data processing on the data to be processed to obtain log data; and sending the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and storing the data processing result and the log data, wherein a persistent storage medium is configured on the memory node. By separating the computing node and the memory node of the architecture, the memory node serves as a separate node for abstracting the memory characteristic, provides data storage service for the computing node, provides good expansibility, and transmits log data to the memory node after finishing data processing on the computing node, so that the memory node plays back the log data, restores the data processing result, reduces the data quantity transmitted to the memory node, fully utilizes the limited bandwidth, reduces the blocking probability and improves the efficiency and stability of data storage.

Drawings

FIG. 1 is a flow chart of a method of data storage provided in one embodiment of the present disclosure;

FIG. 2 is a flow chart of another method of data storage provided in one embodiment of the present disclosure;

FIG. 3 is a process flow diagram of a data storage method applied to a distributed database according to one embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a data storage system according to one embodiment of the present disclosure;

FIG. 5 is a block diagram of a data storage system according to one embodiment of the present disclosure;

FIG. 6 is a data structure diagram of a computing node in a data storage system according to one embodiment of the present disclosure;

FIG. 7 is a block diagram illustrating the memory nodes of a data storage system according to one embodiment of the present disclosure;

FIG. 8 is a data structure diagram of a memory node in a data storage system according to one embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a data storage device according to one embodiment of the present disclosure;

FIG. 10 is a schematic diagram of another data storage device according to one embodiment of the present disclosure;

FIG. 11 is a block diagram of a computing device provided in one embodiment of the present description.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many other forms than described herein and similarly generalized by those skilled in the art to whom this disclosure pertains without departing from the spirit of the disclosure and, therefore, this disclosure is not limited by the specific implementations disclosed below.

The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

Furthermore, it should be noted that, user information (including, but not limited to, user equipment information, user personal information, etc.) and data (including, but not limited to, data for analysis, stored data, presented data, etc.) according to one or more embodiments of the present disclosure are information and data authorized by a user or sufficiently authorized by each party, and the collection, use, and processing of relevant data is required to comply with relevant laws and regulations and standards of relevant countries and regions, and is provided with corresponding operation entries for the user to select authorization or denial.

First, terms related to one or more embodiments of the present specification will be explained.

Persistent storage medium (PM, persistentMemory): a high-capacity storage medium with byte addressing capability has the characteristics of high read-write speed, high-capacity storage and persistent storage. The PM capacity is larger and also has endurance compared to DRAM. PM provides byte addressing capability compared to SSDs, and performance is higher compared to most SSD models. Meanwhile, the cost of PM products is still much higher than that of high-end SSDs, and the requirements on processors are higher. Alternatively or in addition to the DRAM-split architecture, PM can be used to build a scalable, resilient cloud-native database.

Persistent storage node (PMN, persistentMemoryNode): a remote storage node which is arranged separately from the computing node, wherein a persistent storage medium is configured on the persistent storage node, and a remote cache pool is arranged on the persistent storage node.

Computing node (CN, computingNode): the local node for data processing is provided with a processor and a dynamic storage medium (memory) on the computing node, and has the characteristics of higher read-write speed, small capacity storage and dynamic storage.

RDMA (remote direct memory Access): a remote data storage protocol is used for transferring data from a system to a remote system memory through RDMA directly into a storage area of a computer through a network, so that the problem of data processing delay of a remote storage node in the process of remote data storage is solved.

Unilateral RDMA: and one end directly transmits data to the other end under the precondition that the dynamic storage address of the other end is determined, and the participation of a processor of the other end is not needed.

Bilateral RDMA: one end transmits data to the other end, and the other end can transmit data to the other end only when the other end prepares a dynamic memory space in advance.

DRAM (dynamic random access memory): a dynamic storage medium has the characteristics of higher read-write speed, small capacity storage and dynamic storage.

WAL (Write-ahead logging, pre-Write log algorithm): when the database system adopting the WAL log submits the transaction, the disk write operation is only about half of the traditional rollback log, so that the efficiency of the database disk I/O operation is greatly improved, and the performance of the database is improved.

CDLog (ComputingDrivingLog, compute drive log): a logging mechanism driven by a compute node.

LSN (log sequence number): is an incrementally increasing shaping number that represents the total number of bytes written to the log by the data processing record.

Mini-page: and a mini data page obtained by dividing the data page of the data processing result into a plurality of sub pages.

SSD (solid state disk): a persistent storage medium is composed of a control unit and a storage unit.

HDD (hard disk drive): the permanent storage medium consists of disc, magnetic head, disc rotating shaft, control motor, magnetic head controller, data converter, interface, buffering unit, etc.

Data page: dividing the storage space with fixed size on the storage medium, and writing the data into the corresponding partial storage space to form a data page. For example, the 16K memory space is divided to obtain 7 partial memory spaces, and data is written into the corresponding partial memory spaces to form a data page.

Durability: the data processing results cannot be rolled back after transmission, and the data processing results are permanent.

Transaction commit: and the computing node sends the log to a persistent storage node, and performs the operation of persistent storage on the data processing result and the log data.

A database engine: the core service program for storing, processing and protecting data provides data access functions for users of clients, and in one or more embodiments of the present description the database engine is a MySQL database engine.

Dirty bit (DirtyBit): also known as a page overwrite flag bit, indicates whether the page of data has been written.

The metadata table (MetaDataTable) is an address index table of a piece of data, and is implemented by using a hash table and matched with an open addressing method. Where row key of metadata table = id of data page + data page storage address for positioning of data and some metadata services.

In the present specification, a data storage method is provided, and the present specification relates to a data storage system, a data storage device, a computing apparatus, a computer-readable storage medium, and a computer program, one by one, in the following embodiments.

Referring to fig. 1, fig. 1 shows a flowchart of a data storage method according to an embodiment of the present disclosure, where the method is applied to a computing node in a data storage system, and the data storage system further includes an independently configured memory node, and includes the following specific steps:

step 102: and obtaining data to be processed.

The present description applies to computing nodes in a data storage system, including, but not limited to, computing nodes having processors and dynamic storage media disposed thereon, memory nodes, which are persistent storage nodes configured with persistent storage media. The computing nodes are connected with the memory nodes through data transmission channels, the data transmission rate of the data transmission channels is high, and the high efficiency of data storage is guaranteed. The memory node is provided with a dynamic storage interface corresponding to the computing node. The memory nodes are generally a plurality of, are distributed storage nodes, and have high expansibility. The compute nodes may be connected to multiple memory nodes to ensure high availability in a distributed or log replication manner. Among a set of persistent storage nodes, there is one that will be designated as a primary copy, extracting log data for an application.

The data to be processed is initial data for performing data processing. For example, the data to be processed is table data, and the data processing is numerical statistics on the table data; the data to be processed is picture data, and the data processing is used for cutting the picture data. The data to be processed may be pre-written into a cache of the processor, may be pre-written into a dynamic storage medium of the computing node, or may be obtained from a data storage node and written into the dynamic storage medium of the computing node, which is not limited herein. The data to be processed may be the processing result of the previous data processing, the data to be processed used for this data processing, or the newly acquired data to be processed, which is not limited herein.

It should be noted that, the obtaining the data to be processed is to obtain the data to be processed through analysis and processing of the database engine after receiving the data processing request sent by the user.

Illustratively, the pre-written Data to be processed data_a is obtained from a dynamic storage medium of the computing node.

The data to be processed is obtained, log data is obtained for subsequent data processing, initial data is provided, and a foundation is laid.

Step 104: and carrying out data processing on the data to be processed to obtain log data.

The data processing is to perform task processing on data, for example, perform data classification on the data to be processed, perform numerical calculation on the data to be processed, perform data insertion, deletion, modification on the data to be processed, and the like.

The log data is recorded with data processing records of the data to be processed, and comprises at least one data processing record, wherein the log data can be physical log data or logic log data, and the log playback efficiency of the physical log data is higher than that of the logic log data.

The data to be processed is processed to obtain log data, and the specific mode is as follows: and carrying out data processing on the data to be processed to obtain a data processing record, and integrating the data processing record to obtain log data.

It should be noted that, performing data processing on the data to be processed to obtain a data processing record, integrating the data processing record to obtain log data is implemented by calling a database engine to perform read-write operation on a data page in the dynamic storage medium.

Illustratively, the Data to be processed data_a is subjected to numerical calculation and Data modification to obtain Data processing records Record (record_1, record_ … … record_n), and the Data processing records are integrated to obtain Log Data Log.

The data to be processed is processed to obtain log data, and a foundation of the log data is laid for subsequently sending the log data to the memory node for log playback and storage.

Step 106: and sending the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and storing the data processing result and the log data, wherein a persistent storage medium is configured on the memory node.

Because the storage capacity of the dynamic storage medium on the computing node is smaller and dynamic storage is performed, the data processing result can be covered on the computing node, the data processing result is stored on the persistent storage medium with higher storage capacity and higher reading and writing speed, the data storage persistence can be ensured, the storage space on the computing node cannot be occupied, and meanwhile, compared with the hard disk storage medium on the data storage node, the data storage method has higher reading and writing speed and higher data storage efficiency.

The data processing result is the result data obtained by carrying out data processing on the data to be processed. For example, the data to be processed is table data, the data processing is to carry out numerical statistics on the table data, and the data processing result is a numerical statistics result; the data to be processed is image data, the data processing is to cut the image data, and the data processing result is the cut image data. The data processing result may be written into a cache of the processor, or may be written into a dynamic storage medium of the computing node, or may not be written, that is, the data processing result is not obtained, which is not limited herein.

And the log playback is to process the record according to the data in the log data and then execute the data processing to obtain the data processing result.

The memory node is provided with a persistent storage medium for storing data processing results and a persistent storage medium for storing log data.

It should be noted that, step 102 obtains data to be processed, which is also a data processing, and also obtains log data, which is initialization data. And after the log data are sent to the persistent storage node, performing log playback to obtain initialized data to be processed, and performing log playback according to subsequent log data to obtain a data processing result.

It should be noted that, the data volume of the data to be processed and the data processing result is far greater than the data volume of the log data, and in the embodiment of the specification, by sending the log data with small data volume to the memory node, the data processing result can be obtained through log playback and restoration on the memory node, so that the large data volume is prevented from occupying the bandwidth between the computing node and the memory node, the blocking probability is reduced, and the efficiency and the stability of data storage are improved.

After the data processing result is stored in the memory node, if the subsequent computing node needs to reprocess the data processing result, the data processing result may be sent to the computing node, or log data corresponding to the data processing result may be sent to the computing node, and log playback is performed at the computing node to obtain the data processing result.

The log data is sent to the memory node in the following specific modes: and sending the log data to the memory node according to a preset remote data storage protocol. Among other things, remote data storage protocols include single-sided RDMA and double-sided RDMA. Because the bilateral RDMA can better ensure that the computing node drives the persistent storage node to perform data storage persistence, and a large number of processors on the persistent storage node are required to participate in data storage, the delay of the bilateral RDMA is higher, and the single-sided RDMA has lower delay.

The log data is subjected to log playback to obtain a data processing result, and the specific mode is as follows: and carrying out data processing on the data to be processed on the persistent storage node according to the data processing record in the log data to obtain a data processing result.

The data processing result and the log data are stored in the specific mode that: storing the data processing result to a corresponding data storage medium, and storing the log data to a corresponding log storage medium, wherein the data storage medium and the log storage medium are persistent storage media which are divided in advance and used for storing corresponding types of data.

The Log Data Log is sent to the memory node according to the unilateral RDMA, so that the memory node performs numerical calculation and Data modification on the Data to be processed Data data_a on the persistent storage node according to the Data processing records Record (record_1, record_ … … record_n) in the Log Data, a Data processing result data_b is obtained, the Data processing result is stored in a corresponding Data storage medium, and the Log Data is stored in the corresponding Log storage medium, wherein the Data storage medium and the Log storage medium are the persistent storage medium which is divided in advance and is used for storing the corresponding type of Data.

In the embodiment of the present specification, data to be processed is acquired; carrying out data processing on the data to be processed to obtain log data; and sending the log data to the memory node so that the memory node performs log playback on the log data to obtain a data processing result, and storing the data processing result and the log data, wherein a persistent storage medium is configured on the memory node. By separating the computing node and the memory node of the architecture, the memory node serves as a separate node for abstracting the memory characteristic, provides data storage service for the computing node, provides good expansibility, and transmits log data to the memory node after finishing data processing on the computing node, so that the memory node plays back the log data, restores the data processing result, reduces the data quantity transmitted to the memory node, fully utilizes the limited bandwidth, reduces the blocking probability and improves the efficiency and stability of data storage.

In an alternative embodiment of the present disclosure, step 102 includes the following specific steps:

and receiving the data to be processed sent by the data storage node.

The data storage node is a remote storage node configured with a hard disk storage medium, and the data storage node is arranged separately from the computing node. The computing nodes are connected with the data storage nodes by using a distributed file system to construct a data transmission channel. The data storage nodes are generally a plurality of distributed storage nodes, and compared with a persistent storage medium on the persistent storage node, the data storage node has slower read-write speed of a hard disk storage medium, but has larger storage capacity and lower cost.

The method for receiving the data to be processed sent by the data storage node comprises the following specific steps: and sending a data acquisition request to the data storage node, wherein the data acquisition request carries related information of the data to be processed, and receiving the data to be processed, which is sent by the data storage node and is acquired according to the related information.

It should be noted that, after receiving the data to be processed sent by the data storage node, the computing node writes the data to be processed into the dynamic storage medium for subsequent data processing.

Illustratively, a Data acquisition Request is sent to a Data storage node, where the Data acquisition Request carries relevant information infoData_A of Data to be processed, and the Data to be processed, which is sent by the Data storage node and acquired according to the relevant information, is received.

The data to be processed sent by the data storage node is received, and the basis is laid for subsequent data processing to obtain log data, and initial data is provided.

In an alternative embodiment of the present disclosure, step 104 includes the following specific steps:

and carrying out data processing on the data to be processed to obtain a data processing result and log data.

The data to be processed is processed to obtain a data processing result and log data, and the specific mode is as follows: and carrying out data processing on the data to be processed to obtain a data processing result and a data processing record, and integrating the data processing record to obtain log data.

It should be noted that, the specific process of data processing is to call the database engine to perform read-write operation on the data page on the dynamic storage medium to implement data processing, write the data processing record, and after completing data reading-writing, update the data page on the dynamic storage medium, that is, the data processing result, and obtain log data by integrating the data processing record for subsequent transaction submission.

Illustratively, the Data to be processed data_a is subjected to numerical calculation and Data modification to obtain a Data processing result data_b and a Data processing Record (record_1, record_2 … … record_n), and the Data processing records are integrated to obtain Log Data Log.

And carrying out data processing on the data to be processed to obtain a data processing result and log data, and laying a foundation of the data processing result and the log data for subsequently sending the log data to the memory node for log playback and storage.

In an optional embodiment of the present disclosure, the data to be processed includes a data page to be processed, the data processing result includes at least one data processing result sub-page, and the log data includes a data processing record corresponding to the at least one data processing result sub-page;

correspondingly, the data processing is carried out on the data to be processed to obtain a data processing result and log data, and the method comprises the following specific steps:

dividing a data page to be processed into a plurality of data sub-pages to be processed;

and carrying out data processing on at least one data sub-page to be processed to obtain at least one data processing result sub-page and a corresponding data processing record.

The data page to be processed is an initial data page on the dynamic storage medium for data processing. The data sub-page to be processed is an initial data sub-page after logically dividing the data page to be processed, namely a Mini-page.

The data processing result sub-page is an updated data sub-page obtained after data processing is carried out on the data sub-page to be processed.

And in the process of carrying out data processing on any data sub page to be processed to obtain an updated data processing result sub page, recording the data processing operation to obtain a corresponding data processing record.

It should be noted that, after dividing the data page to be processed into a plurality of data sub-pages to be processed, a dirty bit mark is added to each data sub-page to be processed for marking the data sub-page to be processed through the writing operation, and according to the dirty bit mark, a log serial number is allocated to the data sub-page to be processed through the writing operation for recording the corresponding data processing record. Only the data processing records corresponding to the data processing sub pages after the writing operation are integrated to obtain the log data and are sent, so that the data quantity of the sent data is reduced, the writing bandwidth of the memory node is prevented from being occupied excessively, the blocking probability is reduced, and the data storage efficiency is improved.

Dividing the data page to be processed into a plurality of data sub-pages to be processed, wherein the specific mode is as follows: the data page to be processed is logically divided into a plurality of data sub-pages to be processed. The logic division may be performed according to a data type, a data amount, a data read-write logic, and the like of the data page to be processed.

Illustratively, the Data page to be processed data_a.page is logically divided into 7 Data sub-pages to be processed Mini-pages (data_a1.page, data_a2.page … … data_a7.page), and Data processing is performed on at least one Data sub-page to obtain at least one Data processing result sub-page Mini-page (data_b1.page, data_b2.page … … data_b7.page) and corresponding Data processing Record (record_1, record_2 … … record_7).

Dividing the data page to be processed into a plurality of data sub-pages to be processed, carrying out data processing on at least one data sub-page to be processed to obtain at least one data processing result sub-page and a corresponding data processing record, reducing the transmitted data quantity, further avoiding excessive occupation of the writing bandwidth of the memory node, further reducing the blocking probability and further improving the data storage efficiency.

In an alternative embodiment of the present disclosure, step 106 includes the following specific steps:

configuring a corresponding first storage address for at least one data processing result sub-page;

and sending at least one data processing record and a corresponding first storage address to the memory node so that the memory node performs log playback on each data processing record to obtain a corresponding data processing result sub-page, and storing each data processing result sub-page into the corresponding first storage address, wherein the first storage address is a storage address on a persistent storage medium.

The first storage address is a storage address on a persistent storage medium on the memory node. And configuring corresponding first storage addresses for the data processing result sub-pages on the computing nodes in advance, so that the memory nodes accurately store the data processing result sub-pages obtained by log playback according to the first storage addresses.

The corresponding first storage address is configured for at least one data processing result sub-page, specifically, the following steps are: and configuring a corresponding first storage address for at least one data processing result sub-page according to a metadata table of a preset data page. Wherein, the metadata recorded on the metadata table for each data processing result sub-page comprises 4 fields: the second memory address represents a memory address of the current data sub-page in the local dynamic storage medium, and the dirty bit vector is used to track the data processing result sub-page of the dirty bit flag. The local log sequence number is used to record a newly modified log sequence number for the data sub-page, the first storage address being the storage address of the current data sub-page on the persistent storage medium.

At least one data processing record and a corresponding first storage address are sent to a memory node in the following specific modes: and sending at least one data processing record and a corresponding first storage address to the memory node according to a preset remote data storage protocol. Among other things, remote data storage protocols include single-sided RDMA and double-sided RDMA.

Performing log playback on each data processing record to obtain a corresponding data processing result sub page, wherein the specific mode is as follows: and carrying out data processing on the corresponding sub pages of the data to be processed on the persistent storage node according to each data processing record to obtain sub pages of data processing results.

For example, according to a metadata table MetaDataTable of a preset Data page, corresponding first storage addresses RBP (RBP 1, RBP2 … … RBP 7) are configured for 7 Data processing result sub-pages Mini-pages (data_b1. Page, data_b2.Page … … data_b7. Page), 7 Data processing records and corresponding first storage addresses are sent to a memory node according to single-sided RDMA, so that the memory node performs numerical calculation and Data modification on the corresponding 7 to-be-processed Data sub-pages Mini-pages (data_a1. Page, data_a2.Page … … data_a7. Page) on the persistent storage node according to each Data processing Record (record_1, record_2 … … record_n), and each Data processing result sub-page is obtained and stored in the corresponding first storage address.

And configuring a corresponding first storage address for at least one data processing result sub-page, and sending at least one data processing record and the corresponding first storage address to a memory node so that the memory node performs log playback on each data processing record to obtain a corresponding data processing result sub-page, and storing each data processing result sub-page into the corresponding first storage address, wherein the first storage address is a storage address on a persistent storage medium, thereby ensuring accurate storage of the data processing result sub-page obtained by log playback on the memory node and improving the stability of data storage.

In an alternative embodiment of the present specification, a dynamic storage medium is configured on a computing node;

correspondingly, after data processing is carried out on the data to be processed to obtain a data processing result and log data, the method further comprises the following specific steps:

and storing the data processing result into a dynamic storage medium.

Since the data processing result may be used for subsequent data processing, after the data processing result is obtained, the data processing result is stored in the dynamic storage medium of the computing node, so that the processing efficiency of subsequent data processing is ensured.

The data processing result is stored in the dynamic storage medium, namely the data processing result is obtained by calling a database engine to perform read-write operation on a data table on the dynamic storage medium, and updating the data page. Or writing the data to be processed in the dynamic storage medium into a cache of the processor, performing read-write operation on a data table on the cache by calling a database engine, updating a data page to obtain a data processing result, and storing the data processing result into the dynamic storage medium.

Optionally, a local buffer pool is configured in the dynamic storage medium, and is used for storing data processing results with high read-write frequency.

By storing the data processing result into the dynamic storage medium, the processing efficiency of the subsequent data processing is ensured.

In an alternative embodiment of the present disclosure, after performing data processing on data to be processed to obtain a data processing result and log data, the method further includes the following specific steps:

and sending the data processing result to the data storage node so that the data storage node stores the data processing result.

Because the dynamic storage medium is a storage medium for dynamic storage, updated data processing results can be obtained along with subsequent data processing, and further the initial data processing results are covered, so that in order to ensure the persistent storage of the data processing results, the data processing results need to be stored to a data storage node configured with a hardware storage medium, the persistence of the stored data processing results is ensured, and the stability of data storage is improved.

The data processing result is sent to the data storage node so that the data storage node stores the data processing result, and the specific mode is as follows: and configuring a corresponding third storage address for the data processing result, wherein the third storage address is a storage address on a hard disk storage medium of the data storage node, and sending the data processing result to the data storage node so that the data storage node stores the data processing result to the third storage address. Further, a corresponding third memory address is configured for the data processing result, specifically: and configuring a corresponding third storage address for the data processing result according to a preset metadata table. Further, the data processing result is sent to the data storage node by the following specific modes: and sending the data processing result to the data storage node according to a preset remote data storage protocol. Among other things, remote data storage protocols include single-sided RDMA and double-sided RDMA.

For example, according to the preset metadata table MetaDataTable, a corresponding third storage address Addr is configured for the Data processing result data_b, and the Data processing result is sent to the Data storage node according to single-side RDMA, so that the Data storage node stores the Data processing result to the third storage address.

And the data processing result is sent to the data storage node, so that the data storage node stores the data processing result, the durability of the stored data processing result is ensured, and the stability of data storage is improved.

In an alternative embodiment of the present disclosure, after step 106, the following specific steps are further included:

under the condition of carrying out data processing on the covered data processing results, acquiring historical data processing results from the memory nodes;

judging whether the historical data processing result is a covered data processing result or not;

if not, acquiring the history log data corresponding to the history data processing result from the memory node, and performing log playback on the history log data to obtain the data processing result.

Because the dynamic storage medium is a storage medium for dynamic storage, other data can be used for covering the data processing result along with subsequent data processing, and the data processing result can be obtained from the memory node when the covered data processing result is required to be reprocessed subsequently. However, when the memory node receives the log data sent by the computing node, a certain time is required for performing log playback on the log data, in the process of log playback, the history data processing result may not be the latest data processing result, the memory node cannot directly perform data processing on the log data, whether the data processing result is the covered data processing result is required to be judged, if yes, the memory node performs data processing on the covered data processing result, and if not, the history log data is required to be firstly subjected to log playback to obtain the latest data processing result, and then the data processing is performed on the latest data processing result.

It should be noted that, a dynamic storage interface corresponding to the computing node is left on the memory node, so that the computing node can directly obtain the history processing data and the history log data from the memory node.

The historical data processing result is a data processing result obtained by log playback on the memory node. The historical log data is recorded with data processing records for the historical data processing results, and comprises at least one data processing record, wherein the historical log data can be physical log data or logic log data, and the log playback efficiency of the physical log data is higher than that of the logic log data. For example, the covered data processing result is data 2, the historical data processing result is data 1, the historical log data is recorded with the data processing record for processing the data 1 to obtain the data 2, the computing node obtains the historical data processing result, and the log is replayed to obtain the data 2.

Performing log playback on the history log data to obtain a data processing result, wherein the specific mode is as follows: and carrying out data processing on the historical data processing result according to the data processing record in the historical log data to obtain the data processing result. For example, on a computer node, performing Data processing on the historical Data processing result data_A to obtain a Data processing result data_B, and obtaining a corresponding Data processing result; the method comprises the steps of receiving historical Log Data Log, sending the historical Log Data Log to a memory node for Log playback, at the moment, on a computing node, covering a Data processing result data_B, carrying out Data processing on the Data processing result data_B, obtaining a historical Data processing result data_A from the memory node, judging that the historical Data processing result data_A is not the covered Data processing result data_B, obtaining historical Log Data Log corresponding to the historical Log Data processing result data_A from the memory node, and carrying out Log playback on the historical Log Data Log to obtain a Data processing result data_B.

For example, in the case of performing Data processing on the covered Data processing result data_b, the history Data processing result data_a is acquired from the memory node, whether the history Data processing result is the covered Data processing result is determined, if not, the history Log Data Log corresponding to the history Data processing result data_a is acquired from the memory node, and the Data processing is performed on the history Data processing result data_a according to the Data processing Record (record_1, record_2 … … record_n) in the history Log Data, so as to obtain the Data processing result data_b.

Under the condition of carrying out data processing on the covered data processing result, acquiring a historical data processing result from the memory node, judging whether the historical data processing result is the covered data processing result, if not, acquiring historical log data corresponding to the historical data processing result from the memory node, and carrying out log playback on the historical log data to obtain the data processing result, thereby improving the stability and applicability of data processing.

Referring to fig. 2, fig. 2 is a flowchart of another data storage method according to an embodiment of the present disclosure, where the method is applied to a separately provided memory node in a data storage system, and the memory node is configured with a persistent storage medium, and includes the following specific steps:

Step 202: and receiving log data sent by the computing node.

The embodiment of the specification is applied to an independently arranged memory node in a data storage system, wherein the data storage system comprises, but is not limited to, a computing node, a memory node and a data storage node, the computing node is provided with a processor and a dynamic storage medium, the memory node is a persistent storage node, and the persistent storage medium is provided with the memory node. The memory nodes are connected with the computing nodes through data transmission channels, the data transmission rate of the data transmission channels is high, and the high efficiency of data storage is ensured. The memory node is provided with a dynamic storage interface corresponding to the computing node. The memory nodes are generally a plurality of, are distributed storage nodes, and have high expansibility.

Because the storage capacity of the dynamic storage medium on the computing node is smaller and dynamic storage is performed, the dynamic storage node on the computing node has large data throughput, other data can be used for covering the data processing result, the data processing result is stored on the persistent storage medium with higher storage capacity and reading and writing speed, the data persistence can be ensured, the storage space on the computing node can not be occupied, and meanwhile, compared with the hard disk storage medium on the data storage node, the data storage method has higher reading and writing speed and higher data storage efficiency.

The log data is data recorded with data processing records for data to be processed, and comprises at least one data processing record. The log data can be physical log data or logical log data, wherein the log playback efficiency of the physical log data is higher than that of the logical log data.

The log data sent by the computing node is received by the following specific modes: and receiving the log data sent by the computing node according to a preset remote data storage protocol. Among other things, remote data storage protocols include single-sided RDMA and double-sided RDMA. Because the bilateral RDMA can better ensure that the computing node drives the persistent storage node to perform data storage persistence, and a large number of processors on the persistent storage node are required to participate in data storage, the delay of the bilateral RDMA is higher, and the single-sided RDMA has lower delay.

Illustratively, according to single-sided RDMA, log data Log sent by a compute node is received.

And receiving log data sent by the computing node, and laying a foundation for subsequent log playback to obtain a data processing result.

Step 204: and performing log playback on the log data to obtain a data processing result.

It should be noted that, when the computing node obtains the data to be processed, which is also a data processing, log data is also obtained, that is, initialization data. And carrying out log playback on the log data sent by the receiving computing node to obtain initialized data to be processed, and carrying out log playback according to the subsequent log data to obtain a data processing result.

Illustratively, according to the Data processing records Record (record_1, record_2 … … record_n) in the Log Data Log, the Data data_a to be processed on the persistent storage node is subjected to numerical calculation and Data modification, so as to obtain a Data processing result data_b.

And performing log playback on the log data to obtain a data processing result, so that the data processing result is restored, and the durability of the data is ensured.

Step 206: and storing the data processing result and the log data.

The persistent storage node is provided with a persistent storage medium for storing data processing results and a persistent storage medium for storing log data.

Illustratively, the Data processing result data_b is stored to a corresponding Data storage medium, and the Log Data Log is stored to a corresponding Log storage medium, wherein the Data storage medium and the Log storage medium are persistent storage media which are divided in advance for storing the corresponding type of Data.

In the embodiment of the specification, log data sent by a computing node is received; performing log playback on the log data to obtain a data processing result; and storing the data processing result and the log data. By separating the computing node and the memory node of the architecture, the memory node serves as a node which abstracts the memory characteristic independently, provides data storage service for the computing node, provides good expansibility, receives log data sent by the computing node, performs log playback on the log data, restores a data processing result, reduces the data quantity sent by the receiving memory node, fully utilizes the limited bandwidth, reduces the blocking probability, and improves the efficiency and the stability of data storage.

In an optional embodiment of the present disclosure, the log data includes a data processing record corresponding to at least one sub page of data processing results;

correspondingly, the step 204 includes the following specific steps:

and performing log playback on each data processing record to obtain a corresponding data processing result sub-page, wherein each data processing record and the corresponding data processing result sub-page are obtained by dividing a data page to be processed into a plurality of data sub-pages to be processed by a computing node and performing data processing on at least one data sub-page to be processed.

And on the computing node, carrying out data processing on any sub page of the data to be processed, and recording data processing operation in the process of obtaining updated sub pages of the data processing result to obtain corresponding data processing records.

It should be noted that, on the computing node, after dividing the data page to be processed into a plurality of data sub-pages to be processed, a dirty bit mark is added to each data sub-page to be processed for marking the data sub-page to be processed through the writing operation, and according to the dirty bit mark, a log serial number is allocated to the data sub-page to be processed through the writing operation for recording the corresponding data processing record. Only the data processing records corresponding to the data processing sub pages after the writing operation are integrated to obtain the log data and are sent, so that the data quantity of the sent data is reduced, the writing bandwidth of the memory node is prevented from being occupied excessively, the blocking probability is reduced, and the data storage efficiency is improved.

Performing log playback on each data processing record to obtain a corresponding data processing result sub page, wherein the specific mode is as follows: and carrying out data processing on the sub-pages of the data to be processed on the persistent storage node according to each data processing record to obtain corresponding sub-pages of the data processing result.

Illustratively, according to 7 Data processing records Record (record_1, record_2 … … record_7), data processing is performed on 7 Data sub-pages to be processed on the persistent storage node (data_a1.page, data_a2.page … … data_a7.page), so as to obtain 7 corresponding Data processing result sub-pages Mini-page (data_b1.page, data_b2.page … … data_b7.page).

And performing log playback on each data processing record to obtain a corresponding data processing result sub-page, wherein each data processing record and the corresponding data processing result sub-page are obtained by dividing the data page to be processed into a plurality of data sub-pages to be processed by a computing node and performing data processing on at least one data sub-page to be processed, so that the received data volume is reduced, the writing bandwidth of a memory node is further prevented from being excessively occupied, the blocking probability is further reduced, and the data storage efficiency is further improved.

In an alternative embodiment of the present disclosure, step 202 includes the following specific steps:

receiving at least one data processing record and a corresponding first storage address sent by a computing node, wherein the first storage address is a storage address on a persistent storage medium and is distributed to at least one data processing result sub-page by the computing node;

correspondingly, step 206 includes the following specific steps:

and storing each data processing result sub page into a corresponding first storage address.

At least one data processing record and a corresponding first storage address sent by a computing node are received, and the specific mode is as follows: and receiving at least one data processing record and a corresponding first storage address sent by the computing node according to a preset remote data storage protocol. Among other things, remote data storage protocols include single-sided RDMA and double-sided RDMA.

Illustratively, according to the unilateral RDMA, 7 Data processing records Record (record_1, record_ … … record_7) and corresponding first storage addresses RBP (RBP 1, RBP2 … … RBP 7) sent by the computing node are received, according to the 7 Data processing records, 7 Data sub-pages Mini-pages (data_a1.page, data_a2.page … … data_a7.page) to be processed on the persistent storage node are subjected to Data processing, so as to obtain corresponding 7 Data processing result sub-pages Mini-page (data_b1.page, data_b2.page … … data_b7.page), and each Data processing result sub-page is stored in the corresponding first storage address.

And receiving at least one data processing record and a corresponding first storage address sent by the computing node, wherein the first storage address is a storage address on a persistent storage medium, the first storage address is distributed to at least one data processing result sub-page by the computing node, and each data processing result sub-page is stored in the corresponding first storage address, so that accurate storage of the data processing result sub-page obtained by log playback on the memory node is ensured, and the stability of data storage is improved.

In an alternative embodiment of the present disclosure, a log cache and an address cache are configured on a memory node;

after receiving at least one data processing record and a corresponding first storage address sent by the computing node, the method further comprises the following specific steps:

and writing each data processing record into the log cache, and writing the first storage address into the address cache.

The storage of CDLog includes two ring buffers: one is a log cache for storing data processing records and the other is an address cache for storing first memory addresses. The log cache is used to directly store data processing records, such as the log cache of MySQL database. Each first memory address mainly contains three fields: the first field is a first storage address a, records the log serial number of the data processing result sub-page at the current playback, the second field is a first storage address b, the record corresponding content is the data processing result sub-page, and the last field is the log serial number. When the computing node sends the data processing record to the memory node, corresponding contents are additionally written into the log cache and the address cache at the same time, and the writing operation is completed through single-side RDMA. And the memory node is provided with a corresponding log playback thread, and corresponding data processing records are played back to the data cache of the memory node by reading the contents of the log cache and the address cache, so that corresponding data processing result sub-pages are updated.

The memory node is also configured with a data cache, and each data processing result sub-page is stored in the data cache, wherein any data processing result sub-page comprises a corresponding first storage address, namely a current log serial number of current playback and a target log serial number required to be played back.

And each data processing record is written into the log cache, and the first storage address is written into the address cache, so that the accuracy of log playback and the accuracy of data storage are ensured, and the stability of data storage is improved.

The data storage method provided in the present specification will be further described with reference to fig. 3 by taking an application of the data storage method to data storage as an example. Fig. 3 is a flowchart of a processing procedure of a data storage method according to an embodiment of the present disclosure, where the method is applied to a computing node in a data storage system, and the data storage system further includes an independently configured memory node, and specifically includes the following steps.

Step 302: and receiving the data page to be processed sent by the data storage node.

Step 304: the data page to be processed is divided into a plurality of data sub-pages to be processed.

Step 306: and carrying out data processing on at least one data sub-page to be processed to obtain at least one data processing result sub-page and a corresponding data processing record.

Step 308: and configuring a corresponding first storage address for at least one data processing result sub-page.

Step 310: and sending at least one data processing record and the corresponding first storage address to the memory node so that the memory node performs log playback on each data processing record to obtain a corresponding data processing result sub-page, and storing each data processing result sub-page into the corresponding first storage address.

Wherein the first storage address is a storage address on a persistent storage medium.

Step 312: at least one sub-page of data processing results is stored in a dynamic storage medium.

Step 314: in the case of data processing on the covered data processing result sub-page, a history data processing result sub-page is acquired from the memory node.

Step 316: and judging whether the historical data processing result sub-page is an overlaid data processing result sub-page.

Step 318: if not, acquiring a history data processing record corresponding to the history data processing result from the memory node, and performing log playback on the history data processing record to obtain a sub page of the data processing result.

Step 320: and carrying out data processing on the data processing result sub page to obtain an updated data processing result sub page and a corresponding updated data processing record.

Step 322: and configuring a corresponding first storage address for the updated data processing result sub page.

Step 324: and sending the updated data processing record and the corresponding first storage address to the memory node so that the memory node performs log playback on the updated data processing record to obtain a corresponding updated data processing result sub-page, and storing the updated data processing result sub-page into the corresponding first storage address.

In the present embodiment, a log mechanism is driven by a computing node. The method allows the computing node to only write WAL data to a memory node which is arranged remotely and independently, and uninstalls log playback to the memory node, and redesigns the placement mode of the log data by utilizing the byte addressing capability of the persistent storage medium. Network traffic from the compute node to the memory node is reduced, overhead caused by log parsing is avoided, the compute node is allowed to optimistically perform remote reading, and if the retrieved data cached by the memory node is not up-to-date, the compute node remotely reads the data of the memory node and plays back the data locally to obtain an update processing result sub-page.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of a data storage system, and fig. 4 shows a schematic structural diagram of a data storage system provided in one embodiment of the present disclosure. As shown in fig. 4, the system includes:

The computing node 402 is configured to obtain data to be processed, perform data processing on the data to be processed to obtain log data, send the log data to the memory node 404, so that the memory node 404 performs log playback on the log data to obtain a data processing result, and store the data processing result and the log data, where a persistent storage medium is configured on the memory node 404;

the memory node 404 is configured to receive the log data sent by the computing node 402, perform log playback on the log data, obtain a data processing result, and store the data processing result and the log data.

It should be noted that, in the embodiment of the present disclosure, at least one of the computing node 402, the memory node 404, and the data storage node is not limited to the structure of fig. 4.

In this embodiment of the present disclosure, a computing node is configured to obtain data to be processed, perform data processing on the data to be processed to obtain log data, send the log data to a memory node, so that the memory node performs log playback on the log data to obtain a data processing result, and store the data processing result and the log data, where a persistent storage medium is configured on the memory node; the memory node is used for receiving the log data sent by the computing node, carrying out log playback on the log data to obtain a data processing result, and storing the data processing result and the log data. By separating the computing node and the memory node of the architecture, the memory node serves as a separate node for abstracting the memory characteristic, provides data storage service for the computing node, provides good expansibility, and transmits log data to the memory node after finishing data processing on the computing node, so that the memory node plays back the log data, restores the data processing result, reduces the data quantity transmitted to the memory node, fully utilizes the limited bandwidth, reduces the blocking probability and improves the efficiency and stability of data storage.

The above is an exemplary scenario of a data storage system of the present embodiment. It should be noted that, the technical solution of the data storage system and the technical solution of the data storage method belong to the same conception, and details of the technical solution of the data storage system, which are not described in detail, can be referred to the description of the technical solution of the data storage method.

FIG. 5 illustrates an architecture diagram of a data storage system provided in one embodiment of the present description. As shown in fig. 5:

the data storage system includes a compute node, a memory node, and a data storage node. Wherein, the computing node is configured with sufficient CPU (central processing unit) resources and limited dynamic storage media (memory) for processing user queries. Each computing node contains a medium-scale local cache pool (LBP) to cache the hottest data in dynamic storage media. The persistent storage medium is configured with a plurality of persistent storage media due to a group of servers comprising a plurality of persistent storage media and exposing a memory interface to the computing node. An amount of persistent storage space may be provided to enhance performance for the database. The corresponding memory node typically includes two partitions: a remote cache pool (which is typically much larger in size than the local cache pool) and a smaller log area (for fast WAL persistence). The compute nodes may be connected to multiple memory nodes to ensure high availability in a distributed or log replication manner. Among the copies of a set of persistent storage media, there is one that will be designated as the primary copy, extracting log data for the application. The data storage nodes are provided with a plurality of solid state disks and mechanical hard disks, and each computing node is further connected to a group of storage nodes by using a distributed file system as persistent and low-cost storage, and the data storage nodes are provided with the distributed file system, and data pages are stored in the distributed file system.

And on the computing node, analyzing and processing the SQL request through the database engine, performing data processing on the data to be processed to obtain log data, and sending the log data in the local cache pool to the memory node. Calling a log playback thread on a memory node, executing a log mechanism (CDLog) driven by a computing node, performing log playback on log data to obtain a data processing result, storing the data processing result into a remote cache pool of the memory node, and acquiring the data processing result from the remote cache pool for analysis and processing again under the condition that the subsequent computing node needs to reprocess the data processing result.

FIG. 6 illustrates a data structure diagram of a compute node in a data storage system according to one embodiment of the present disclosure. As shown in fig. 6:

on the computing node, after receiving the SQL request sent by the user, the database engine is called to analyze and process the SQL request, the SQL request is converted into transaction semantics and executed, and the user request can firstly enter the module. And performing read-write operation on the data page of the local cache pool to obtain the data page of the local cache pool: log sequence number [0,1,2 … … ] and dirty bit [010101 … … ]. Determining a first storage address according to a metadata table of a data page, wherein the metadata table of the data page records: the method comprises the steps of storing a first storage address 1 corresponding to a data page in a local cache pool, storing a first storage address 2 corresponding to a dirty bit vector, storing a first storage address 3 corresponding to a local log serial number, and storing a first storage address 4 corresponding to the data page in a remote cache pool. And according to the data page of the local cache pool, implementing the writing log and then submitting the transaction. The metadata table of the data page is metadata data information maintaining the data page, and is realized by using a hash table. The mapping from the number of the data page to the first memory address of the data page metadata is implemented by an open addressing method. The metadata of each data page contains 4 fields: the first memory address 1 represents the address of the current data in the local dynamic storage medium, and the first memory address 2 corresponding to the dirty bit vector is used to track the mini data page whose tag is modified. The first storage address 3 corresponding to the local log sequence number records the latest modified log sequence number of the data page, and the first storage address 4e corresponding to the data page in the remote cache pool is the storage address of the current page on the remote dynamic storage medium. The specific workflow is that after a user request arrives at a computing node, SQL analysis is performed, the underlying storage is accessed through a transaction interface, a data page is modified, the data page is logically divided into a plurality of mini data pages in advance, in the process of modifying the data page, a transaction engine marks the mini data pages modified by data on a dirty bit vector, a corresponding log serial number is allocated to each modified mini data page, and the corresponding log serial number is stored in a first storage address 3 corresponding to a local log serial number. At transaction commit, we send all modified mini data pages to the remote persistent storage node using RDMA.

FIG. 7 is a diagram illustrating a memory structure of a memory node in a data storage system according to one embodiment of the present disclosure. As shown in fig. 7:

the storage of CDLog includes two ring buffers: one is a log cache for storing data processing records and the other is an address cache for storing first memory addresses. The log cache is used to directly store data processing records, such as the log cache of MySQL database. Each first memory address mainly contains three fields: the first field is a first storage address a, records the log serial number of the data processing result sub-page at the current playback, the second field is a first storage address b, the record corresponding content is the data processing result sub-page, and the last field is the log serial number. When the computing node sends the data processing record to the memory node, corresponding contents are additionally written into the log cache and the address cache at the same time, and the writing operation is completed through single-side RDMA. And the memory node is provided with a corresponding log playback thread, and corresponding data processing records are played back to the data cache of the memory node by reading the contents of the log cache and the address cache, so that corresponding data processing result sub-pages are updated. The calculation node avoids overwriting the effective data on the memory node by reading the value of the log serial number which is already played back on the current persistent storage node, and the log playback thread is a background thread and is used for playing back the log.

FIG. 8 is a diagram illustrating a data structure of a memory node in a data storage system according to one embodiment of the present disclosure. As shown in fig. 8:

in a remote cache pool of the log cache node, the data page is divided into 64 mini pages by the corresponding pair for storing data processing result sub pages: data processing results sub page 0 … … data processing results sub page 63, wherein any data processing results sub page contains a corresponding first storage address, namely a current log sequence number (890, 960) for current playback and a target log sequence number (1000, 1063) for playback.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of a data storage device, and fig. 9 shows a schematic structural diagram of a data storage device according to one embodiment of the present disclosure. As shown in fig. 9, the apparatus is applied to a computing node, and the apparatus includes:

an acquisition module 902 configured to acquire data to be processed;

the processing module 904 is configured to perform data processing on the data to be processed to obtain log data;

the first sending module 906 is configured to send the log data to the memory node, so that the memory node performs log playback on the log data to obtain a data processing result, and stores the data processing result and the log data, where the memory node is configured with a persistent storage medium.

Optionally, the processing module 904 is further configured to:

Optionally, the data to be processed includes a data page to be processed, the data processing result includes at least one data processing result sub-page, and the log data includes at least one data processing record corresponding to the data processing result sub-page;

correspondingly, the processing module 904 is further configured to:

dividing a data page to be processed into a plurality of data sub-pages to be processed; and carrying out data processing on at least one data sub-page to be processed to obtain at least one data processing result sub-page and a corresponding data processing record.

Optionally, the first sending module 906 is further configured to:

configuring a corresponding first storage address for at least one data processing result sub-page; and sending at least one data processing record and a corresponding first storage address to the memory node so that the memory node performs log playback on each data processing record to obtain a corresponding data processing result sub-page, and storing each data processing result sub-page into the corresponding first storage address, wherein the first storage address is a storage address on a persistent storage medium.

Optionally, the computing node is configured with a dynamic storage medium;

correspondingly, the device further comprises:

and the second storage module is configured to store the data processing result into the dynamic storage medium.

Optionally, the apparatus further comprises:

and the second sending module is configured to send the data processing result to the data storage node so that the data storage node stores the data processing result.

Optionally, the apparatus further comprises:

a restoration module configured to acquire a history data processing result from the memory node in the case of performing data processing on the covered data processing result; judging whether the historical data processing result is a covered data processing result or not; if not, acquiring the history log data corresponding to the history data processing result from the memory node, and performing log playback on the history log data to obtain the data processing result.

The above is an exemplary scheme of a data storage device of the present embodiment. It should be noted that, the technical solution of the data storage device and the technical solution of the data storage method belong to the same conception, and details of the technical solution of the data storage device, which are not described in detail, can be referred to the description of the technical solution of the data storage method.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of a data storage device, and fig. 10 shows a schematic structural diagram of another data storage device provided in one embodiment of the present disclosure. As shown in fig. 10, the apparatus is applied to a memory node, and the memory node is configured with a persistent storage medium, and the apparatus includes:

a receiving module 1002 configured to receive log data sent by a computing node;

the playback module 1004 is configured to perform log playback on the log data to obtain a data processing result;

the first storage module 1006 is configured to store the data processing result and log data.

Optionally, the log data includes at least one data processing record corresponding to a sub page of the data processing result;

correspondingly, the playback module 1004 is further configured to:

Optionally, the receiving module 1002 is further configured to:

correspondingly, the first storage module 1006 is further configured to:

Optionally, the memory node is configured with a log cache and an address cache;

correspondingly, the device further comprises:

and the writing module is configured to write each data processing record into the log cache and write the first storage address into the address cache.

FIG. 11 illustrates a block diagram of a computing device provided in accordance with one embodiment of the present description. The components of computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. Processor 1120 is coupled to memory 1110 via bus 1130, and database 1150 is used to hold data.

The computing device 1100 also includes an access device 1140, the access device 1140 enabling the computing device 1100 to communicate via one or more networks 1160. Examples of such networks include a public switched telephone Network (PSTN, publicSwitchedTelephone Network), a local area Network (LAN, localAreaNetwork), a wide area Network (WAN, wideAreaNetwork), a personal area Network (PAN, personalAreaNetwork), or a combination of communication networks such as the internet. The access device 1140 may include one or more of any type of network interface, wired or wireless, such as a network interface card (NIC, networkInterfaceController), such as an IEEE802.11 wireless local area network (WLAN, wirelessLocalAreaNetwork) wireless interface, a worldwide interoperability for microwave access (Wi-MAX, worldwideInteroperabilityforMicrowaveAccess) interface, an ethernet interface, a universal serial bus (USB, universalSerialBus) interface, a cellular network interface, a bluetooth interface, a near field communication (NFC, nearFieldCommunication).

In one embodiment of the present description, the above components of computing device 1100, as well as other components not shown in FIG. 11, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device illustrated in FIG. 11 is for exemplary purposes only and is not intended to limit the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or personal computer (PC, personalComputer). Computing device 1100 may also be a mobile or stationary server.

Wherein the processor 1120 is configured to execute computer-executable instructions that, when executed by the processor, perform the steps of the data processing method described above. The foregoing is a schematic illustration of a computing device of this embodiment. It should be noted that, the technical solution of the computing device and the technical solution of the data storage method belong to the same concept, and details of the technical solution of the computing device, which are not described in detail, can be referred to the description of the technical solution of the data storage method.

An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data storage method described above.

The above is an exemplary version of a computer-readable storage medium of the present embodiment. It should be noted that, the technical solution of the storage medium and the technical solution of the data storage method belong to the same concept, and details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the data storage method.

An embodiment of the present disclosure also provides a computer program, where the computer program, when executed in a computer, causes the computer to perform the steps of the data storage method described above.

The above is an exemplary version of a computer program of the present embodiment. It should be noted that, the technical solution of the computer program and the technical solution of the data storage method belong to the same conception, and details of the technical solution of the computer program, which are not described in detail, can be referred to the description of the technical solution of the data storage method.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read-only memory (ROM), a random access memory (RAM, randomAccessMemory), an electrical carrier signal, a telecommunication signal, a software distribution medium, and so forth.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the embodiments are not limited by the order of actions described, as some steps may be performed in other order or simultaneously according to the embodiments of the present disclosure. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all required for the embodiments described in the specification.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are merely used to help clarify the present specification. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the teaching of the embodiments. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. This specification is to be limited only by the claims and the full scope and equivalents thereof.

Claims

1. A data storage method applied to a computing node in a data storage system, the data storage system further comprising an independently configured memory node, comprising:

acquiring data to be processed;

carrying out data processing on the data to be processed to obtain log data;

2. The method of claim 1, wherein the performing data processing on the data to be processed to obtain log data includes:

3. The method of claim 2, wherein the data to be processed comprises a data page to be processed, the data processing result comprises at least one data processing result sub-page, and the log data comprises a data processing record corresponding to the at least one data processing result sub-page;

the data processing is carried out on the data to be processed to obtain a data processing result and log data, and the method comprises the following steps:

dividing the data page to be processed into a plurality of data sub-pages to be processed;

4. The method of claim 3, the sending the log data to the memory node comprising:

configuring a corresponding first storage address for the at least one data processing result sub-page;

and sending the at least one data processing record and the corresponding first storage address to the memory node so that the memory node performs log playback on each data processing record to obtain corresponding data processing result sub-pages, and storing each data processing result sub-page into the corresponding first storage address, wherein the first storage address is a storage address on the persistent storage medium.

5. The method of claim 2, wherein the computing node has a dynamic storage medium configured thereon;

after the data processing is performed on the data to be processed to obtain a data processing result and log data, the method further comprises the following steps:

and storing the data processing result into the dynamic storage medium.

6. The method of claim 2, further comprising, after the performing data processing on the data to be processed to obtain a data processing result and log data:

and sending the data processing result to a data storage node so that the data storage node stores the data processing result.

7. The method of claim 1, after sending the log data to a memory node, further comprising:

under the condition of carrying out data processing on the covered data processing results, acquiring historical data processing results from the memory node;

judging whether the historical data processing result is the covered data processing result or not;

8. A data storage method applied to an independently configured memory node in a data storage system, the memory node configured with a persistent storage medium, comprising:

receiving log data sent by a computing node;

performing log playback on the log data to obtain a data processing result;

and storing the data processing result and the log data.

9. The method of claim 8, wherein the log data includes a data processing record corresponding to at least one data processing result sub-page;

the log playback is carried out on the log data to obtain a data processing result, which comprises the following steps:

and performing log playback on each data processing record to obtain a corresponding data processing result sub-page, wherein each data processing record and the corresponding data processing result sub-page are obtained by dividing a data page to be processed into a plurality of data sub-pages to be processed by the computing node and performing data processing on at least one data sub-page to be processed.

10. The method of claim 9, the receiving log data sent by a computing node, comprising:

receiving at least one data processing record and a corresponding first storage address sent by the computing node, wherein the first storage address is a storage address on the persistent storage medium and is allocated to the at least one data processing result sub-page by the computing node;

The storing the data processing result comprises the following steps:

11. The method of claim 10, wherein the memory node has a log cache and an address cache configured thereon;

after receiving the at least one data processing record and the corresponding first storage address sent by the computing node, the method further comprises:

12. A data storage system, comprising: a computing node and a memory node;

the computing node being configured to implement the data storage method of any one of claims 1 to 7;

the memory node being configured to implement the data storage method of any one of claims 8 to 11.

13. A computing device, comprising:

a memory and a processor;

the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which when executed by the processor, implement the steps of the data storage method of any one of claims 1 to 11.

14. A computer readable storage medium storing computer executable instructions which when executed by a processor implement the steps of the data storage method of any one of claims 1 to 11.