WO2020034729A1

WO2020034729A1 - Data processing method, related device, and computer storage medium

Info

Publication number: WO2020034729A1
Application number: PCT/CN2019/089582
Authority: WO
Inventors: 饶蓉; 魏明昌
Original assignee: 华为技术有限公司
Priority date: 2018-08-17
Filing date: 2019-05-31
Publication date: 2020-02-20
Also published as: CN110837479B; CN110837479A

Abstract

Disclosed is a data processing method, comprising: an object storage device receiving a data read or write IO request, wherein the data read or write IO request comprises a data integrity field (DIF), the DIF is used for bearing attribute information of data to be operated, and specifically, the attribute information may be attribute information of data to be read or attribute information of data to be written; furthermore, the object storage device processing the data to be operated according to the attribute information of the data to be operated. By using the embodiments of the present invention, the performance of a disk can be improved, and the waste of storage resources can be avoided.

Description

Data processing method, related equipment and computer storage medium

Technical field

The present invention relates to the field of storage technology, and in particular, to a data processing method, a related device, and a computer storage medium.

Background technique

With the development and progress of the Internet, the amount of global data is growing at an extremely fast rate every day. Faced with such a huge amount of data, the requirements for data storage systems are particularly important. At present, in order to meet the storage requirements of massive data, distributed storage systems are usually used for data storage.

However, in practice, it is found that the current distributed storage system uses a unified storage strategy and storage layout for data storage, which will cause loss of disk performance and waste of storage resources.

Summary of the Invention

The embodiment of the invention discloses a data processing method, related equipment and a computer storage medium, which reduces the loss of disk performance and the waste of storage resources.

According to a first aspect, an embodiment of the present invention provides a data processing method. The method includes: an object storage device receives a data read IO request sent by a host, the data read IO request includes a data integrity field DIF, and the DIF bearer The attribute information of the data to be read includes, for example, the size of the data to be read, the read granularity of the data to be read, and the read information of the address to be read corresponding to the data to be read. Accordingly, the object storage device may process the data to be read according to the attribute information of the data to be read.

By implementing the embodiments of the present invention, it is possible to perform storage management on the read data according to the attribute information of the data to be read, and to use the data characteristics for personalized storage to avoid problems such as low disk performance and waste of storage resources.

In some possible implementation manners, the attribute information of the data to be read includes a read granularity of the data to be read, and the read granularity is used to indicate a size of the pre-read data including the data to be read. Accordingly, if the size of the read-ahead data is greater than or equal to the first threshold value, the object storage device may read the read-ahead data including the data to be read, and send the read-ahead data to the client to cache it Memory on the client. It is convenient for the host to read the corresponding data to be read directly from the client's cache the next time the host sends the same data read IO request to save data reading time and improve data processing efficiency.

If the size of the read-ahead data is less than the first threshold, the object storage device may directly read the data to be read and send the data to be read to the host.

By implementing the embodiments of the present invention, it is possible to decide whether to extract and obtain the pre-read data according to the size of the pre-read data, which is convenient for reading data directly from the memory of the client next time, thereby saving data reading time and improving data processing efficiency.

In some possible embodiments, the attribute information of the data to be read includes the read information of the data to be read corresponding to the address to be read. The read information may include, but is not limited to, a read frequency or a read type. The read information is used to indicate how often data is read from the address to be read in a unit time. Accordingly, when the read information is the first read information, the object storage device writes the data to be read into a non-volatile cache of the object storage device. When the read information is the second read information, the object storage device writes the data to be read to a non-volatile cache of the object storage device, and sets an expiration time in the non-volatile cache to clear the non-volatile memory. The volatile cache stores data that exceeds the expiration time. The frequency indicated by the first read information is greater than the frequency indicated by the second read information.

In some possible embodiments, the attribute information of the data to be read includes the read information of the data to be read corresponding to the address to be read. The read information is used to indicate how often data is read from the address to be read in a unit time. When the frequency indicated by the read information is greater than or equal to the second threshold, the data to be read is sent to the client for caching in the client's memory. It is convenient to directly read the data to be read from the memory of the client next time, thereby improving the data reading efficiency. Correspondingly, when the frequency indicated by the read information is less than the second threshold, the data to be read is sent to the client, and the read data is cached in the hard disk of the client.

In a second aspect, an embodiment of the present invention provides a data processing method. The method includes: the object storage device receives a data write IO request forwarded by a client, and the data write IO request includes a data integrity field DIF, and the DIF It carries attribute information of the data to be written, such as the size of the data to be written, the granularity of the data to be written, and the write information of the address to be written corresponding to the data to be written. Accordingly, the object storage device processes the data to be written according to the attribute information of the data to be written. By implementing the embodiments of the present invention, the attribute information of the data to be written can be reasonably used to perform storage management on the data to be written, so as to solve problems such as disk performance loss and waste of storage resources in the existing distributed storage system.

In some possible implementation manners, the attribute information of the data to be written includes write information of an address to be written corresponding to the data to be written, such as a write frequency or a write type. The write information is used to indicate how often data is written to the address to be written in a unit time. Accordingly, when the write information is the first write information, the object storage device may store the data to be written into a non-volatile cache of the object storage device. When the write information is the second write information, the object storage device may store the data to be written into a hard disk of the object storage device. The frequency indicated by the first write information is greater than the frequency indicated by the second write information. The read-write rate of the non-volatile cache is greater than the read-write rate of the hard disk.

By implementing the above steps, the data to be written can be stored in the corresponding cache or hard disk according to the attribute characteristics of the data to be written, which facilitates improving the efficiency of data processing.

In some possible implementation manners, the attribute information of the data to be written includes a writing granularity of the data to be written, and the writing granularity is used to indicate a storage granularity used when storing the data to be written. After the client receives the data write IO request sent by the host, it can allocate corresponding stripe information to the data to be written according to the attribute information of the data to be written. The stripe information includes one or more of the data used to store the data to be written. Stripe units and the physical address of each stripe unit. Each stripe unit is configured to store a small part of the data to be written, and may also be referred to as data to be stored. Further, the client sends a stripe unit used when data is to be written to the corresponding object storage device according to the stripe information, and the stripe unit carries its own physical address and the data to be stored to be stored. Accordingly, the object storage device receives the stripe unit sent by the client. The object storage device writes part of the data to be written (data to be stored) into the hard disk of the object storage device according to the physical address of the stripe unit.

By implementing the above steps, the client can configure the corresponding stripe according to the attribute information of the data to be written. The stripe is composed of one or more stripe units, and each stripe unit is mapped to the hard disk in the object storage device. . Correspondingly, after receiving the stripe unit, the object storage device writes or stores the data to be written to the hard disk according to the stripe unit. Conducive to improving hard disk performance and hard disk utilization.

In some possible implementation manners, after configuring the stripe information, the client may create a corresponding storage mapping relationship for the data to be written according to the writing granularity of the data to be written, that is, a storage mapping relationship for the data to be written. Correspondingly, the object storage device receives the storage mapping relationship of the data to be written sent by the client, and the storage mapping relationship of the data to be written includes the address to be written when the data to be written is stored Mapping to physical addresses.

In some possible implementation manners, the object storage device may further create a storage mapping relationship of the data to be stored according to a writing granularity of the data to be written, and the storage mapping relationship includes a relationship between a logical address and a physical address when the data to be stored is stored. Mapping relationship, the logical address is related to the address to be written of the data to be written. Specifically, the logical address may be determined and obtained according to an address to be written and a writing granularity of data to be written.

According to a third aspect, an embodiment of the present invention provides a data processing method. The method includes: the host obtains attribute information of data to be operated, and generates a data operation request according to the attribute information of the data to be operated. Further, the host sends a data operation request to the client, where the data operation request includes a data integrity field DIF. The DIF is used to carry attribute information of data to be operated.

By implementing the embodiments of the present invention, the host can obtain the attribute information of the data to be operated according to actual needs, and then generate a data operation request and send it to the client or forward it to the object storage device through the client. It is convenient for the client or the object storage device to perform storage management on the operation data according to the attribute information of the data to be operated, so as to implement storage according to the attribute characteristics of the data, reduce the performance loss of the disk, and avoid waste of storage resources.

In some possible implementations, the data operation request may be a data read IO request. Accordingly, the attribute information of the data to be operated may include the read granularity of the data to be read and / or the read information of the read address corresponding to the data to be read, and the read information may specifically be the read frequency or read type . The read granularity is used to indicate the size of the read-ahead data including the data to be read. The read information is used to indicate how often data is read from the address to be read in a unit time.

In some possible implementations, the data operation request may be a data write IO request. Correspondingly, the attribute information of the data to be operated includes the writing granularity of the data to be written and / or the writing information of the address to be written corresponding to the data to be written. The writing new may specifically be a writing frequency or a writing type. . The write granularity is used to indicate a storage granularity used when data to be written is stored. The write information is used to indicate how often data is written to the address to be written in a unit time.

According to a fourth aspect, an embodiment of the present invention provides a data processing method. The method includes: a client receives a data write IO request sent by a host, and the data write IO request includes a data integrity field DIF, and the DIF bears pending data. The attribute information of the written data. Correspondingly, the client configures corresponding stripe information for the data to be written according to the attribute information of the data to be written. The stripe information includes at least one stripe unit used for the data to be written and each stripe. The physical address of the unit. The client sends the stripe unit carrying the partial data to be written and the physical address to the corresponding object storage device according to the stripe information.

In some possible embodiments, the attribute information of the data to be written includes a writing granularity of the data to be written. Correspondingly, the client can configure a stripe whose size of the stripe unit matches the write granularity for the data to be written. The information of the stripe (that is, the stripe information) includes at least one stripe when the data to be written is stored. The unit and the physical address of each stripe unit. Each stripe unit is used to store part of the data to be written. In other words, the client selects the stripe unit with the same write granularity to store the data to be written, which is beneficial to improving the utilization of the disk.

In some possible implementation manners, the attribute information of the data to be written includes write information of an address to be written corresponding to the data to be written, and is used to indicate how often data is written to the address to be written in a unit time. Accordingly, the client can configure the data to be written with stripes that match the frequency indicated by the write information. The writing type may specifically be a writing frequency or a writing type. In other words, the client can store the data to be written in the same write type or in the same write frequency range into the same stripe.

In a fifth aspect, an embodiment of the present invention provides a data processing apparatus (specifically, an object storage device), including a communication unit and a processing unit, where:

The communication unit is configured to receive a data read IO request, where the data read IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of data to be read;

The processing unit is configured to process the data to be read according to attribute information of the data to be read.

In some possible implementation manners, the attribute information of the data to be read includes a read granularity of the data to be read, and is used to indicate a size of the pre-read data including the data to be read. The processing unit is specifically configured to read pre-read data including the data to be read when the size of the pre-read data is greater than or equal to a first threshold, and send the pre-read data to a client To cache into the memory of the client; or when the size of the preset data is less than a first threshold, read the data to be read and send the data to be read to the host.

In some possible implementation manners, the attribute information of the data to be read includes read information of a to-be-read address corresponding to the data to be read, and the read information is used to indicate the unit of time to be read The frequency at which data is read from the address. The processing unit is specifically configured to write the data to be read into a non-volatile cache of the object storage device when the read information is first read information; or, when the read information is When the information is the second read information, the data to be read is written to a non-volatile cache of the object storage device, and an expiration period is set in the non-volatile cache of the object storage device To clear data stored in the non-volatile cache with a duration exceeding the expiration duration; wherein the frequency indicated by the first read information is greater than the frequency indicated by the second read information.

For the content that is not shown or described in the embodiment of the present invention, reference may be made to the related description in the embodiment described in the first aspect, and details are not described herein.

According to a sixth aspect, an embodiment of the present invention provides another data processing apparatus (which may specifically be an object storage device), including a communication unit and a processing unit, where:

The communication unit is configured to receive a data write IO request, where the data write IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of data to be written;

The processing unit is configured to process the data to be written according to attribute information of the data to be written.

In some possible implementation manners, the attribute information of the data to be written includes write information of an address to be written corresponding to the data to be written, and the write information is used to indicate the unit of time to be written The frequency at which data is written to the incoming address. The processing unit is specifically configured to: when the write information is first write information, the object storage device stores the data to be written into a non-volatile cache of the object storage device; or, when The write information is second write information, and the object storage device stores the data to be written in a hard disk of the object storage device; wherein the frequency indicated by the first write information is greater than The frequency indicated by the second write information is described.

In some possible implementation manners, the attribute information of the data to be written includes a writing granularity of the data to be written, and is used to indicate a storage granularity used when the data to be written is stored. The client receives the data write IO request, and configures corresponding stripe information for the data to be written according to the attribute information of the data to be written. The stripe information includes the data to be used when the data to be written is stored. The stripe unit and the physical address of the stripe unit; the client sends the stripe unit carrying the data to be written and the physical address to the communication according to the stripe information A unit; the communication unit receives the stripe unit carrying the data to be written and the physical address sent by the client. Accordingly, the processing unit is specifically configured to store the data to be written into a hard disk of the object storage device according to a physical address of the stripe unit.

In some possible implementation manners, the processing unit is further configured to create a storage mapping relationship of the data to be written according to a writing granularity of the data to be written, where the storage mapping relationship includes the data to be written The mapping relationship between the address to be written and the physical address during data storage.

Regarding the content that is not shown or described in the embodiment of the present invention, reference may be made to the related description in the embodiment described in the second aspect, and details are not described herein.

In a seventh aspect, an embodiment of the present invention provides another data processing device (which may specifically be a host), including a communication unit and a processing unit, where:

The processing unit is configured to obtain attribute information of data to be operated;

The communication unit is configured to send a data operation request to a client, where the data operation request includes a data integrity field DIF, and the DIF is used to carry attribute information of the data to be operated.

In some possible implementation manners, when the data operation request is a data read IO request, the attribute information of the data to be operated includes the read granularity of the data to be read and / or the data to be read corresponding to the data to be read Address read information. The read granularity of the data to be read is used to indicate the size of the pre-read data including the data to be read; the read information of the address to be read is used to indicate the unit of time to be read. The frequency at which data is read from the read address.

In some possible implementation manners, when the data operation request is a data write IO request, the attribute information of the data to be operated includes a writing granularity of data to be written and / or a data to be written corresponding to the data to be written Address write information. The write granularity of the data to be written is used to indicate the storage granularity used when the data to be written is stored; the write information of the address to be written is used to indicate the address to be written in a unit time. How often data is written.

For the content that is not shown or described in the embodiment of the present invention, reference may be made to related descriptions in the embodiment described in the third aspect, and details are not described herein.

In an eighth aspect, an embodiment of the present invention provides an object storage device, including: a processor, a memory, a communication interface, and a bus; the processor, the communication interface, and the memory communicate with each other through the bus; a communication interface for receiving and sending data; A memory for storing instructions; a processor for calling instructions in the memory to execute the method described in the first aspect or any possible implementation manner of the first aspect. In a ninth aspect, an embodiment of the present invention provides another object storage device, including: a processor, a memory, a communication interface, and a bus; the processor, the communication interface, and the memory communicate with each other through the bus; and the communication interface is used to receive and send data. A memory for storing instructions; a processor for calling instructions in the memory to execute the method described in the second aspect or any possible implementation manner of the second aspect.

According to a tenth aspect, an embodiment of the present invention provides a host, including: a processor, a memory, a communication interface, and a bus; the processor, the communication interface, and the memory communicate with each other through the bus; a communication interface for receiving and sending data; a memory, For storing instructions; a processor for calling instructions in the memory to execute the method described in the third aspect or any possible implementation manner of the third aspect. According to an eleventh aspect, an embodiment of the present invention provides a client, including: a processor, a memory, a communication interface, and a bus; the processor, the communication interface, and the memory communicate with each other through the bus; a communication interface for receiving and sending data; The memory is configured to store instructions; the processor is configured to call the instructions in the memory and execute the method described in the fourth aspect or any possible implementation manner of the fourth aspect.

In a twelfth aspect, a computer non-transitory storage medium is provided. The computer non-transitory storage medium stores program code for data processing. The program code includes instructions for performing the first aspect described above or the method described in any possible implementation of the first aspect.

In a thirteenth aspect, a computer non-transitory storage medium is provided, where the computer non-transitory storage medium stores program code for data processing. The program code includes instructions for performing the second aspect described above or the method described in any possible implementation of the second aspect.

In a fourteenth aspect, a computer non-transitory storage medium is provided, where the computer non-transitory storage medium stores program code for data processing. The program code includes instructions for performing the third aspect or the method described in any possible implementation manner of the third aspect.

In a fifteenth aspect, a computer non-transitory storage medium is provided, where the computer non-transitory storage medium stores program code for data processing. The program code includes instructions for performing the fourth aspect or the method described in any possible implementation of the fourth aspect.

The above storage medium may be non-volatile.

In a sixteenth aspect, a chip product is provided to execute the first aspect or the method in any possible implementation manner of the first aspect.

In a seventeenth aspect, a chip product is provided to execute the second aspect or the method in any possible implementation manner of the second aspect.

In an eighteenth aspect, a chip product is provided to execute the third aspect or the method in any possible implementation manner of the third aspect.

In a nineteenth aspect, a chip product is provided to execute the fourth aspect or the method in any possible implementation manner of the fourth aspect.

By implementing the embodiments of the present invention, the problems of resource waste and performance loss in the existing distributed storage system can be solved.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present invention or the prior art more clearly, the drawings used in the embodiments or the description of the prior art will be briefly introduced below.

FIG. 1 is a schematic diagram of a network framework of a data processing system according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart of a data processing method according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a format of a data operation request according to an embodiment of the present invention.

FIG. 4 is a schematic flowchart of another data processing method according to an embodiment of the present invention.

FIG. 5 is a schematic flowchart of another data processing method according to an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention.

FIG. 7 is a schematic structural diagram of another data processing device according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings of the present invention.

First, some technical terms related to this application are introduced.

Logical block address (LBA), also called logical address or relative address. The address that a block of data stored on disk or tape has for retrieval or rewriting. It can also mean that in a computing device with an address conversion function, the address (or operand) given by the access instruction is called a logical address.

Physical block address (LBA), also known as physical address. Refers to the address from the central processing unit (CPU) address bus for addressing. Among them, the physical address is usually mapped to memory or storage. In a computing device having an address conversion function, a logical address can be converted into an actual effective address (ie, a physical address) in a memory through calculation or conversion in an addressing manner.

Stripe is a method of dividing continuous data into data blocks of the same size and distributing each piece of data to different disks in the array. Among them, there are two parameters that affect the band effect: band depth and band size. Stripe depth refers to the number of stripes that can be read or written in parallel at the same time. The stripe size refers to the size of the data block written to each disk. This application will be exemplified below.

In order to solve the problems of performance loss and resource waste in the existing data storage solutions, this application proposes a data processing method, a network framework to which the method is applicable, and related equipment. First, FIG. 1 is a schematic diagram of a network framework of a data processing system according to an embodiment of the present invention. The data processing system 100 shown in FIG. 1 includes application software 102, a virtual disk interface 104, a client 106, a metadata controller 108 (metadata controller, MDC), and an object-based storage device 110 (OSD). . among them,

The application software 102 may send a data read or write input / output (IO) request to the virtual disk according to its actual needs. The data read / write IO request carries the size of the data to be read / written and the data. Stored logical block address (logical block address, LBA), hereinafter referred to as logical address and other information.

The virtual disk interface 104 may be an interface provided by a virtual block storage (VBS) management component for accessing a virtual disk. Specifically, the application software 102 sends a data read / write IO request to the corresponding virtual disk through the virtual disk interface 104 to read or write data from the corresponding disk. In practical applications, the virtual disk interface 104 and the application software 102 are usually deployed on the same physical device (for example, a host or a server). The metadata controller 108 is responsible for managing the object storage devices 110. The number of the object storage devices 110 may be one or more. The figure takes n as an example, and n is a positive integer. In a distributed data storage system, the number of object storage devices 110 is usually multiple.

The data processing system 100 can also be divided into two parts. The application software 102 and the virtual disk interface 104 are deployed on the client device. The client 106, the metadata controller 108, and the object storage device 110 are deployed on the data of the cloud service provider. In the center. The cloud service provider provides the object storage service to the customer through the client 106, the metadata controller 108, and the object storage device 110. The client accesses the object storage service through the application software 102 and the virtual disk interface 104.

Specifically, the metadata controller 108 may be responsible for maintaining a connection state between the object storage device 110 and the client 106, and the connection state includes an online state and an offline state. The online status indicates that the object storage device 110 and the client 106 can communicate normally, and a communication connection relationship is established. The offline state means that there is no communication connection between the object storage device 110 and the client 106, that is, the client cannot store data to the object storage device in the offline state. Optionally, the metadata controller 108 may also deploy the object storage device 110 according to a certain deployment strategy, such as a partition deployment strategy, a load balancing deployment strategy, and the like, which are not limited herein. Correspondingly, the metadata controller 108 can obtain information about the OSD of each object storage device currently communicating with the client 106, such as the OSD's Internet protocol address (IP), the ID of the OSD, and so on. Optionally, the metadata controller 108 may send the information of the object storage device 110 to the client 106 in advance, so that the client 106 can calculate the corresponding information according to the LBA logical address in the request after receiving the data read / write IO request The local object storage device 110 is not described in detail here. Further, the client may establish communication between the client 106 and the object storage device 110 according to the OSD information (for example, identification, IP address, etc.). For example, the client 106 forwards data read / write IO requests to the object storage. Equipment, etc.

In practical applications, the metadata controller 108 may be deployed on a certain physical device separately, or distributed to one or more physical devices without limitation.

The client 106 may be a software module for receiving a data read / write IO request issued by the application software 102 through the virtual disk interface 104. Further, the client 106 may configure a corresponding data strip for the data to be read / written according to the data read / write IO request to write or store the data into the data strip. In other words, the data in this application is stored in a stripe manner. Optionally, the client 106 may also use a preset algorithm to calculate an error correction code (EC) corresponding to the data. The preset algorithm is customized by a user or a system, such as an EC check code algorithm. How to calculate the EC check code is not described in detail in this application, and how to configure the data stripe will be described in detail later in this application.

Optionally, the client 106 may also send the data strip to be stored, the EC verification code, and the received data read / write IO request together or separately to the corresponding object storage device OSD. Exemplarily, after the client 106 obtains the data stripe to be stored and the EC verification code, the client 106 may newly add these two pieces of information to the data read / write IO request for re-encapsulation, and re-encapsulate the data read / The write IO request is sent to the object storage device, which is not limited in this application.

Accordingly, the object storage device 110 may receive a data read / write IO request and read or write corresponding data according to the request. When the object storage device 110 receives a data read IO request, it reads data blocks and metadata in the corresponding virtual disk according to the LBA logical address in the request. The metadata here may refer to data used to describe the attributes of the data block, such as one or more of a logical address, a physical address, and a size of the data block. Correspondingly, after receiving the data write IO request, the object storage device 110 may write the data to the corresponding virtual disk according to the LBA logical address in the request. How the object storage device performs corresponding data operations according to the data read / write IO request will be detailed in the following in this application.

In practical applications, the application software 102, the virtual disk interface 104, and the client 106 are usually deployed on a physical device. The object storage device 110 is deployed on another physical device. Optionally, the object storage device 110 may also be deployed on the same physical device as the application software 102, the client 106, and the like. The metadata controller 108 is separately deployed on another physical device. Regarding the deployment of each part in FIG. 1, this application is not limited.

In an alternative embodiment, the object storage device 110 includes a cache and a disk. There are no restrictions on the number of caches and disks. The illustration uses one cache and one disk as an example. The disk includes, but is not limited to, a virtual disk or a physical disk. In order to avoid the problem that multiple processes access the same disk and cause disk conflicts, this application uses striping technology to balance the IO load (data) to the disks of each object storage device. The following uses the n object storage devices as examples for data storage related description. That is, each disk of each of the n object storage devices provides a corresponding storage space (strip unit) for storing a small portion of the data. As shown in the figure, the data can be divided into n parts to obtain n data blocks, each data block includes a small part of the data. Correspondingly, each object storage unit in the n object storage devices may provide a stripe unit to store one data block, and n stripe units may be obtained to correspondingly store n data blocks. As shown in the figure, the object storage device 1 can provide a stripe unit 1 to store a first data block (data block 1), and the object storage device 2 can provide a stripe unit 2 to store a second data block (data block 2). . By analogy, the object storage device n may provide a stripe unit n to store the nth data block (data block n). In other words, the stripe used to store the entire data here is composed of n stripe units from n object storage devices, that is, one stripe unit provided by each of the n disks. composition.

Correspondingly, when an object storage device learns a stripe unit carrying a data block, it can store the data block on the stripe unit according to the physical address of the stripe unit, that is, the data block is stored in the stripe. The physical address of the tape unit is written to or stored on its own disk. For example, taking the object storage device 1 as an example, when it learns that the stripe unit 1 needs to store the data block 1, it can write the data block 1 to the stripe unit 1 according to the physical address of the stripe unit 1, that is, write To its own disk.

Similarly, when each of the n object storage devices knows the data block to be stored in its own stripe unit, it can store the corresponding data block according to the physical address of the corresponding stripe unit. When the n object storage devices have completed their respective data block storage, the entire data can be stored.

The cache in the object storage device is used to temporarily or temporarily store data. Optionally, a timer may be set in the cache, and when the data stored in the cache exceeds a preset expiration time, the data in the cache may be processed, such as clearing the data in the cache or deleting the data in the cache. Write data to hard disk and so on.

Next, please refer to FIG. 2, which is a schematic flowchart of a data processing method according to an embodiment of the present invention. The data processing method shown in FIG. 2 includes the following implementation steps:

Step S202: The host obtains attribute information of the data to be operated, and generates a data operation request according to the attribute information of the data to be operated.

Step S204: The host sends a data operation request to the client. Accordingly, the client receives the data operation request and forwards it to the object storage device.

In this application, the host refers to a physical device running application software. Specifically, the application software in the host can obtain attribute information of the data to be operated according to actual requirements. Further, the host may generate a data operation request, and encapsulate the attribute information of the data to be operated in a data integrity field (DIF) in the data operation request. The data operation request is further sent to the client for forwarding to the object storage device through the client.

The data to be operated here may include, but is not limited to, data to be read or data to be written. Accordingly, the attribute information of the data to be operated may specifically refer to the attribute information of the data to be written or the attribute information of the data to be read. The attribute information about the data to be operated is explained in detail below.

Step S206: The object storage device obtains a data operation request, where the data operation request includes attribute information of the data to be operated, and the attribute information is used to describe an attribute of the data to be operated.

Step S208: The object storage device processes the data to be operated according to the attribute information of the data to be operated.

In this application, the data operation request may specifically be a data read IO request or a data write IO request. When the data operation request is a data read IO request, the data to be operated may specifically be data to be read. When the data operation request is a data write IO request, the data to be operated may specifically be data to be written.

The data operation request includes a flag field set by the user or the system, and the flag field is used to indicate attribute information of the data to be operated, such as the size of the data to be operated, the granularity of the data IO, the logical address of the data to be operated, Any one or more of the read information (for example, read frequency or read type) and write information (for example, write frequency or write type) involved in the logical address are described in detail below.

Specifically, FIG. 3 shows a schematic diagram of a format of a data operation request. The data operation request includes a data domain and a data integrity field (DIF). The data field is used to carry data to be transmitted, which may specifically be data to be operated (data to be read or data to be written). The DIF field is used to check the integrity of the data to be transmitted. As shown in the figure, the DIF field includes a check area, a version area, an application software area, and an address area. The respective sizes of the data field and the DIF field can be customized by the user or the system. For example, several storage methods such as 512 + 8 bytes, 4096 + 8, and 4096 + 64 are used. No restrictions. among them,

The check area is used to carry check data, such as cyclic redundancy check (CRC) data. The size of the bytes occupied by the check area can be set by the user or the system, such as 2 bytes.

The version area is used to indicate the version number of the DIF. The size occupied by the version area can be set by the user or the system, such as 1 byte.

The address area is used to indicate the logical address LBA where the data to be operated corresponds to. The size it occupies can be customized by the user or the system, such as 4 bytes.

The application software area is composed of a reserved field and an LBA effective indication field. The size occupied by the application software area can be specifically set by the user or the system. For example, the figure takes 1 byte as an example. The LBA valid indication field is used to indicate whether the LBA logical address carried in the address area is valid. Specifically, when the LBA valid indication field is the first preset character (such as 1), it means that the LBA logical address carried in the address area is valid; when the LBA valid indication field is the second preset character (such as 0), then Indicates that the LBA logical address carried in the address area is invalid. The size occupied by the LBA effective indication field may be specifically set by a user or a system, for example, 1 bit.

The reserved field is a field to be defined, and its occupied size can also be customized by the user or the system, such as 7bit. This application redefines the reserved fields, that is, the attribute information of the data to be operated is carried in the reserved fields. Specifically, the reserved fields include, but are not limited to, any one or more of a data IO granularity field, a data write field, and a data read field. The size and position of each of the reserved fields can be defined according to actual requirements, which is not limited in this application. . among them,

Data IO granularity field, which means different meanings in different application scenarios. Specifically, in a data write application scenario (that is, the data operation request is a data write IO request), the data IO granularity field may specifically be a write granularity field, which is used to indicate a minimum unit size used when data to be operated is stored. In other words, the minimum unit granularity of the data to be stored is divided during storage. The size and location occupied by the storage granularity field may be specifically set by a user or a system. Exemplarily, the storage granularity field occupies 3 bits, and the corresponding writing granularity has the following 8 types: 512byte, 1Kbyte, 4Kbyte, 8Kbyte, 16Kbyte, 32Kbyte, 64Kbyte, 12Kbyte, and so on.

In a data read application scenario (that is, the data operation request is a data read IO request), the data IO granularity field may specifically be a read granularity field, which is used to indicate the size of the read-ahead data, and the read-ahead data includes the pending operation. data. The pre-read data is data to be read. Exemplarily, suppose that the upper-layer application software needs to read 1M data (that is, the size of the pre-read data is 1M), and the size of the data read IO request sent by the application software each time is 1K, that is, the data issued by each application software The size of the data requested by the read IO request is 1K. Correspondingly, in order to complete the reading of 1M data, the application software needs to issue 1000 data read IO requests. However, in this application, the size of the read-ahead data can be indicated by reading the granularity field, and the read-ahead data can be read in advance by using a data read IO request, saving time and improving the efficiency of data reading.

A data write field is used to indicate write information of an address to be written. The address to be written is an address when data to be operated is written, and it may specifically be an LBA logical address. The writing information includes, but is not limited to, a writing type or a writing frequency. In other words, the data write field is used to indicate the write type or write frequency of the address to be written. Specifically, the data write field may be used to reflect a writing frequency of data written in the address to be written in a unit time, that is, a frequency of writing data in the address to be written in a unit time.

Optionally, the device (specifically, the client or the object storage device) in this application may also obtain the write type of the address to be written according to the write frequency of the address to be written. Exemplarily, when the writing frequency of the address to be written is in the first threshold range, the corresponding writing type may be considered or determined to be the first writing type, for example, it is quickly overwritten or often overwritten. Wait. When the writing frequency of the address to be written is in the second threshold range, the corresponding writing type may be considered or determined to be the second writing type, for example, it is rarely overwritten. The upper and lower limits of the first threshold range and the second threshold range may be specifically set by a user or a system. The lower limit of the first threshold range is greater than or equal to the second threshold range. The upper limit.

The size occupied by the data write field may be specifically set by a user or a system according to actual requirements, for example, 2 bits and the like. Data is overwritten when there is multiple data writes to the same address to be written, that is, data written later will overwrite data previously written. Therefore, the writing frequency or type of the address to be written also indicates the writing frequency or type of data overwriting in the address to be written.

The data read field is used to indicate the read information of the address to be read. The address to be read is an address when data to be operated is read, and it may specifically be an LBA logical address. The read information includes, but is not limited to, a read type or a read frequency of an address to be read. Specifically, the data read field may be used to reflect a read frequency of data read occurring at the address to be read in a unit time.

Optionally, the device (specifically, a client or an object storage device) of the present application may also obtain the read type of the address to be read according to the read frequency of the address to be read. Exemplarily, when the read frequency of the address to be read is within a third threshold range, the corresponding read type may be considered or determined to be the first read type, such as frequent reads. When the read frequency of the address to be read is within the fourth threshold range, the corresponding read type may be considered or determined to be the second read type, such as to be read multiple times or sequentially. When the read frequency of the address to be read is in the fifth threshold range, the corresponding read type may be considered or determined to be the third type, such as rarely read or read only once. The third threshold range, the fourth threshold range, and the fifth threshold range may be specifically set by a user or a system. The lower limit of the third threshold range is greater than or equal to the upper limit of the fourth threshold range, and the lower limit of the fourth threshold range is greater than or equal to the upper limit of the fifth threshold range.

The size occupied by the data read field may be specifically set by a user or a system, such as 2 bits and the like.

The following describes related embodiments related to the data storage method provided by the present invention in combination with two application scenarios applicable to the present application. First, in an application scenario of writing data, referring to FIG. 4 is a schematic flowchart of another data processing method according to an embodiment of the present invention. The method shown in FIG. 4 includes the following implementation steps:

Step S402: The application software obtains attribute information of data to be written.

Step S404: The application software sends a data write IO request to the client, where the data write IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of the data to be written. Accordingly, the client receives the data write IO request.

In this application, application software (specifically, a host computer running the application software) may send a data write IO request to a client according to actual requirements, and the data write IO request includes attribute information of the data to be written. Specifically, the attribute information of the data to be written is carried in a DIF field in the data write IO request, and the attribute information of the information to be written may include, but is not limited to, any one or more of the following combinations: the Information such as an address to be written (which may be an LBA logical address) corresponding to the data to be written, a size of the data to be written, writing information of the address to be written, and a data IO granularity. Accordingly, the client receives the data write IO request.

Step S406: The client configures corresponding stripe information for the data to be written according to the attribute information of the data to be written. The stripe information includes at least one stripe unit used when the data to be written is stored and the at least one stripe unit. The physical address of each stripe unit in the stripe unit.

Step S408: The client sends the at least one stripe unit and the physical address of the stripe unit to each object storage device correspondingly according to the stripe information. Each stripe unit is configured to store a small part of the data to be written. Correspondingly, the object storage device receives the stripe unit sent by the client and the physical address of the stripe unit. The stripe unit carries data to be stored, and the data to be stored is part of the data to be written. Optionally, the stripe unit also carries a physical address of the stripe unit.

Step S410: The object storage device processes the data to be stored in the data to be written according to the attribute information of the data to be written.

In step S406, after receiving the data write IO request, the client may determine whether to configure a corresponding data strip for the data to be written according to the attribute information of the data to be written in the data write IO request to store The data to be written. Specifically, when the attribute information of the data to be written includes write information of an address to be written corresponding to the data to be written. The writing information may specifically be a writing frequency or a writing type, which may be used to indicate a frequency or a frequency range in which data is written to the address to be written in a unit time. When the writing information is the first writing information, it is used to indicate that a frequency of writing data to the address to be written within a unit time is within a first threshold range, or a writing type indicated by the first writing information In order to be quickly overwritten or frequently overwritten, etc., the client may abandon the corresponding stripe information for the data to be written. Conversely, when the writing information is the second writing information, it is used to indicate that the frequency of data writing to the address to be written is within a second threshold range within a unit time, or the writing indicated by the first writing information The input type is rarely overwritten, etc. The client can configure corresponding stripe information for the data to be written, which will be described in detail below. The lower limit of the first threshold range is greater than or equal to the upper limit of the second threshold range.

Optionally, after receiving the data write IO request, the client may also directly configure the stripe information for the data to be written according to the attribute information of the data to be written in the data write IO request, and download the data. Issued to the corresponding object storage device. The object storage device side again determines whether to store data according to the physical address of the corresponding stripe unit in the stripe information according to the attribute information of the data to be written, which is not limited in this application.

The client configures the stripe information for the data to be written according to the attribute information of the data to be written. The following specific implementations exist:

First, when the attribute information of the data to be written includes the writing granularity of the data to be written, the client may select a size and the size of the data to be written according to the writing granularity. A granularity-matched stripe unit is written for storing the data to be written. Specifically, in order to reduce the data storage amount, generally, a stripe unit with the same stripe size and the write granularity is selected to form a stripe for storing the data to be written. After each stripe unit in the stripe is obtained, a physical address corresponding to each stripe unit can be obtained, and the stripe unit corresponds to the physical address where the stripe unit corresponds.

For example, suppose the data write IO request size is 32K and the write granularity is 8K. Correspondingly, when the client selects a configuration strip for the data to be written, the client may select four strips of 8K size to form a strip that stores the data to be written. In other words, the stripe here includes 4 strips of 8K size. Correspondingly, during data storage, 32K of data to be written can also be divided into 4 pieces of data to be stored, and each piece of data to be stored is 8K in size, and is stored in one stripe unit. These four stripe units can come from four object storage devices, which are not discussed here.

Secondly, when the attribute information of the data to be written includes writing information of the address to be written corresponding to the data to be written, the writing information may specifically be a writing frequency or a writing type. Correspondingly, the client may allocate corresponding stripe information to the data to be written according to the write information, so as to store the data to be written. The stripe unit includes one or more stripe units and a physical address of each stripe unit, and each stripe unit is configured to store a part of data in the data to be written (that is, data to be stored).

Specifically, when the writing information is a writing frequency, the client may write or store data to be written in the same frequency range to the same stripe or stripe unit. The same frequency range can be specifically set by the user or the system according to actual needs, such as 10-15 times / minute. When the write information is a write type, the object storage device may store the data to be written of the same write type into the same stripe or stripe unit to implement classified storage of the data.

Third, the attribute information of the data to be written includes the writing granularity of the data to be written and the writing information of the address to be written corresponding to the data to be written. Correspondingly, the client may configure corresponding stripe information for the data to be written according to the write granularity and write information of the address to be written, and the stripe information includes information for storing the data to be written. One or more stripe units and the respective physical address of each stripe unit. Each stripe unit is used to store part of the data to be written (that is, data to be stored). Exemplarily, a stripe unit having the same stripe size as the write granularity is selected, and the stripe unit supports carrying or storing data that matches the write information of the address to be written. In other words, the client will comprehensively consider the write type of the data to be written and the write information (write frequency or write type) of the address to be written to configure the corresponding stripe for the data to be written to store the data. Data to be written.

In an optional embodiment, after receiving the data read IO request, the client may also calculate a corresponding EC check code for the data to be written, for example, a preset algorithm is used to calculate an EC check code. Correspondingly, the client may also send the calculated EC verification code and the data to be written to the object storage device, so that the object storage device stores the data to be written and the EC verification code together.

Optionally, in the process of storing data in a stripe manner in this application, for an EC check code that is not full stripe, the write type corresponding to the address to be written in the stripe may be marked as a preset type, For example, is about to be overwritten. It is convenient for the system to recalculate the new EC check code in the background, and combine the data to be written to form a full strip, covering the EC check code obtained in the previous calculation, which is helpful to improve the utilization of disk storage space.

After the client configures the stripe information for the data to be written, it can correspondingly distribute one or more stripe units in the stripe information to the corresponding object storage device, and the stripe unit can carry the physical address of the stripe unit. And data to be stored, which is part of the data to be written. Regarding the mapping relationship between the stripe unit and the object storage device, reference may be made to the related description in the foregoing embodiment, and details are not described herein again. In other words, after the client knows the stripe unit used to store the data to be stored, it can learn the physical address of the stripe unit, and then can know the object storage device or the hard disk of the object storage device to which the stripe unit corresponds. (Disk), which is not limited or detailed here.

Accordingly, the object storage device may subsequently write the data to be stored to the hard disk of the object storage device according to the received stripe unit according to the storage address of the stripe unit.

In step S410, after receiving the data write IO request sent by the client and the stripe unit carrying the data to be stored and the physical address, the object storage device may perform a write operation on the data to be stored in the data write IO request according to attribute information of the data to be written. Data is stored.

Specifically, the attribute information of the data to be written includes write information (such as a write frequency or a write type) of an address to be written. When the writing information is first writing information, the first writing information is used to indicate a frequency of data writing to the address to be written. Specifically, the first write information may be used to indicate that a write frequency of the address to be written is within a first threshold range, or a write type of the address to be written is a first type (for example, often overwritten, Write quickly, etc.). Correspondingly, the object storage device may write or store the data to be written into the first non-volatile cache (generally set inside the object storage device). Optionally, after a period of time, the object storage device may write the data stored in the first non-volatile cache to the non-volatile hard disk of the object storage device, which is convenient for reasonably arranging the data based on the characteristics of the data to be written. The data storage medium is written to improve the efficiency of data reading and writing. Because the writing frequency of the address to be written is high, storing the data to be written in the cache for a period of time and then writing it to the hard disk can make the object storage device receive other New write IO request. The data to be written carried in the new write IO request can overwrite the old data to be written stored in the cache, so that after this period of time, only the latest write IO request for the address to be written is pending. The written data is written to the hard disk. It avoids the inefficiency caused by the data to be written carried by each write IO request for the address to be written, which needs to be written to the hard disk.

When the write information is second write information, the second write information is used to indicate that a write frequency of the address to be written is in a second threshold range, or a write type of the address to be written is Two types (such as rarely overwritten). It can be used to indicate how often data is written to the data to be written. Accordingly, the object storage device may directly write or store the data to be written into a non-volatile hard disk of the object storage device. Specifically, the object storage device will write or store the data to be stored in the data to be written to the non-volatile hard disk of the object storage device according to the physical address of the stripe unit.

The first threshold range and the second threshold range are preset by the user or the system, and the lower limit value of the first threshold range is greater than or equal to the upper limit value of the second threshold range. The read-write rate of the non-volatile cache is greater than the read-write rate of the non-volatile hard disk.

In other words, the object storage device can store or write data to be written to different non-volatile memories at different writing frequencies or types. Exemplarily, if the write type of the address to be written corresponding to the data to be written is overwritten soon, the object storage device may write the data to be written into a non-volatile cache, instead of writing to Disk or hard drive. Optionally, the object storage device may also separately store the data to be written whose write types are frequently overwritten and rarely overwritten to reduce data access interference.

In an optional embodiment, the attribute information of the data to be written includes a writing granularity of the data to be written. After the object storage device receives the data write IO request and the stripe unit carrying the data to be stored, the object storage device may construct a storage mapping relationship of the data to be stored according to the writing granularity of the data to be written. The storage mapping relationship includes a mapping relationship between a logical address and a physical address when the data to be stored is stored. The logical address is determined according to the address to be written and the write granularity.

Specifically, the client receives a data write IO request carrying a data to be written address (usually an LBA logical address). Correspondingly, after the client configures stripe information (one or more stripe units) for the data to be written, the address to be written can be divided according to the write granularity of the data to be written (or the number of stripe units). To obtain the logical address corresponding to each stripe unit. Further, the logical address, the physical address of each stripe unit, and the data to be stored that needs to be stored in the stripe unit are sent to the corresponding object storage device. Accordingly, after receiving the foregoing information, the object storage device may create a corresponding storage mapping relationship for the data to be stored, that is, a storage mapping relationship for the data to be stored. It includes the mapping relationship between the logical address and the physical address when the data to be stored (striped unit) is stored.

Alternatively, after the client sends the stripe unit and the data write IO request to the object storage device, the object storage device can obtain the stripe according to the address to be written in the data write IO request and the granularity of the data to be written. The logical address corresponding to the unit is the logical address when the data to be stored in the stripe unit is stored. How to obtain the logical address of the stripe unit is not limited in this application. Exemplarily, it is assumed that the client is configured with 4 stripe units for data to be written, which are stripe unit 1-stripe unit 4 respectively. They are respectively located in the object storage device 1-the object storage device 4. Accordingly, when the object storage device 4 receives the stripe unit 4 sent by the client, it carries its own physical address and data to be stored. Further, the logical address of the stripe unit 4 is determined in combination with the address to be written in the received data write IO request (assuming 0-40), which is 30-40 here. Finally, a storage mapping relationship is created for the data to be stored, here is the mapping relationship between the logical address and the physical address of the data to be stored stored by the stripe unit 4.

In an optional embodiment, after configuring the stripe information for the data to be written, the client can also create a corresponding storage mapping relationship for the data to be written according to the write granularity of the data to be written, that is, the data to be written Storage mapping. The storage mapping relationship includes a mapping relationship between a logical address (that is, an address to be written) and a physical address when the data to be written is stored.

Specifically, the data write IO request includes the LBA logical address of the data to be written, but the logical address is not related or associated with the physical address of the stripe unit where the data to be written is actually stored. Correspondingly, after the data to be written is stored, the system cannot determine the physical address where the data is actually stored according to the logical address of the data to be written, and thus cannot query the data to be written. Therefore, the object storage device also needs to establish and save the storage mapping relationship of the data to be written. The storage mapping relationship is specifically divided or customized according to the writing granularity, and the logical address and physical address of the data to be written are stored. The address is obtained by association binding.

For example, suppose the size of the data write IO request (specifically the size of the data to be written in the data write IO request) is 32Kbytes, the request includes the LBA logical address of the data to be written is 0-200, and the The write granularity of the data to be written is 8K. The stripe configured by the client for the combination of data to be written includes 4 stripe units, each with a physical address of 0X00-0X04. Accordingly, the object storage device creates a storage mapping relationship table shown in Table 1 below to reflect the storage mapping relationship of the data to be written.

Table 1

As can be seen from Table 1 above, the stripe to be written into the data storage is composed of 4 stripe units, and the size of each stripe unit is 8K. Each stripe unit has its own logical address and physical address. Correspondingly, after the object storage device receives the data read IO request, the data read IO request includes a logical address (for example, 0-50) of the data to be read, and the object storage device may store the storage mapping relationship according to the above table 1. Table, find the physical address corresponding to the logical address (here, the physical address 0X01 of the stripe unit 1), and then read the data of the stripe unit 1 from 0X01 is the data to be read.

In an optional embodiment, the client can also send the storage mapping relationship of the data to be written to each object storage device, so that the object storage device can learn the mapping relationship between the logical address and the physical address of the stripe unit of the data storage. . After the object storage device receives the logical address of the data to be read carried in the data read IO request, it obtains the data to be read from the corresponding physical address according to the logical address.

By implementing the embodiments of the present invention, by redefining the fields in the DIF domain and optimizing the storage layout of data, compared with the prior art, the storage performance of the memory can be improved and the utilization rate of the memory can be improved.

Secondly, in an application scenario of reading data, please refer to FIG. 5, which is a schematic flowchart of another data processing method according to an embodiment of the present invention. The method shown in FIG. 5 includes the following implementation steps:

Step S502: The application software (specifically, a host computer running the application software) obtains attribute information of the data to be read.

Step S504: The application software sends a data read IO request to the client, where the data read IO request includes a DIF field, and the DIF is used to carry attribute information of the data to be read. Accordingly, the client reads the data read IO request. For the data read IO request, reference may be made to the related description of the data write IO request, and details are not described herein again.

Step S506: The client sends a data read IO request to the object storage device. Accordingly, the object storage device receives the data read IO request.

In this application, after the client receives the data read IO request, if the attribute information of the data to be read in the data read IO request includes the size of the read-ahead data, when the size of the read-ahead data is greater than the first threshold, The data read IO request is issued to the object storage device in advance, so that the read-ahead data can be read and obtained more quickly. There are various specific implementation modes issued in advance here, which are not limited in this application. For example, the client can allocate more transmission resources to the data read IO request to increase the delivery rate of the data read IO request, thereby extracting and sending the data read IO request to the object storage device. When the size of the read-ahead data is less than or equal to the first threshold, the data read IO request is sent to the target storage device at a normal delivery rate.

The pre-read data includes the data to be read, which can be understood as the total size of the data that the application software needs to read in a period of time. Understandably, due to the performance of the application software, there is a limit on the size of the data read IO request sent by the application software each time. When the read-ahead data is large, the application software can issue data read IO requests multiple times to complete the read of the read-ahead data of the corresponding size. For example, the size of the read-ahead data is 1M, and the size of the data read IO request is 1K (that is, the size of the data to be read in the data read IO request is 1K). Read data read.

Step S508: The object storage device reads the data to be read, and processes the data to be read according to attribute information of the data to be read.

Step S508 includes the following specific implementations: In the first type, the attribute information of the data to be read includes the size of the pre-read data including the data to be read. If the size of the read-ahead data is greater than the first threshold, the object storage device may first read the read-ahead data, then send the read-ahead data to the client, and cache it to the client side. It is convenient for subsequent application software to issue the same data read IO request, and directly read the corresponding to-be-read data in the pre-read data from the client, saving data reading time.

Specifically, the data read IO request includes the LBA logical address of the read-ahead data, and the object storage device can obtain the physical address corresponding to the logical address of the read-ahead data from the storage mapping relationship stored by itself according to the logical address of the read-ahead data. The address is then read according to the physical address of the read-ahead data to obtain the read-ahead data. Further, the object storage device may send the read-ahead data to the client. Correspondingly, the client can write the read-ahead data into the memory, or the client can store the read-ahead data according to the attribute information of the data to be read in the data read IO request. For details, refer to the following details.

If the size of the pre-read data is less than or equal to the first threshold, the object storage device may read the data to be read, and then store the data to be read according to attribute information of the data to be read. Optionally, after the object storage device reads the data to be read, the data to be read can also be sent to other object storage devices, so that other object storage devices can store the corresponding data according to the attribute information of the data to be read. The data to be read is not limited in this application. In this case, the data to be read need not be stored in the client's memory.

Specifically, when the size of the read-ahead data is less than or equal to the first threshold, the data read IO request includes the LBA logical address of the data to be read. Correspondingly, the object storage device obtains the physical address corresponding to the logical address of the data to be read from the stored storage mapping relationship according to the logical address of the data to be read. Then, the data to be read is obtained by reading from the physical address. Further, the data to be read is sent to an upper-layer application software (that is, a host where the application software is deployed).

Secondly, when the attribute information of the data to be read includes the read information of the address to be read corresponding to the data to be read, the object storage device may The data to be read is processed.

Specifically, the attribute information of the data to be read includes read information of an address to be read corresponding to the data to be read, and the read information includes, but is not limited to, a read type or a read frequency. When the read information is used to indicate the frequency of data reading at the address to be read in a unit time, when the frequency is greater than or equal to the second threshold, the object storage device may send the data to be read to the client for caching. Into the client's memory. It is convenient to directly read the data to be read from the memory of the client next time, and improve the data reading rate. Accordingly, when the frequency is less than the second threshold, the object storage device may store the data to be read into a hard disk of the object storage device. Optionally, the object storage device may also send the data to be read to the client for storage on the client's hard disk, etc., which is not limited in this application.

Optionally, when the read information is first read information, the first read information is used to indicate that a read frequency of an address to be read is within a third threshold range, or indicates a read type of the address to be read The third type (for example, read often). Correspondingly, the object storage device may write the data to be read into the second non-volatile cache of the object storage device, so as to facilitate reading the cached to-be-read directly from the second non-volatile cache next time. Get the data.

When the read information is second read information, the second read information is used to indicate that the read frequency of the address to be read corresponding to the data to be read is in a fourth threshold range, or to indicate the read of the address to be read The fetch type is the fourth type (for example, multiple reads are about to be read). Accordingly, the object storage device may store the data to be read into a third non-volatile cache of the object storage device. Optionally, the object storage device may further set a corresponding expiration period in the third non-volatile cache, which is convenient for clearing data in the third non-volatile cache that has a storage period exceeding the expiration period. The expiration period may be specifically set by a user or a system, for example, one day.

When the read information is third read information, the third read information is used to indicate that the read frequency of the address to be read corresponding to the data to be read is within a fifth threshold range, or to indicate the read of the address to be read The fetch type is the fifth type (for example, rarely read or read only, etc.). Accordingly, the object storage device may write or store the data to be read into a hard disk of the object storage device, and / or the object storage device may clear the data stored in the preset storage address. Specifically, the object storage device may determine, according to the data write IO request, a preset storage address (or a non-volatile memory) where the data to be cleared is indicated, and the preset storage address may be an LBA logical address. Correspondingly, the object storage device may clear the data corresponding to the preset storage address to release the corresponding data stored in the non-volatile memory; or directly release all the data stored in the non-volatile memory.

The third threshold range, the fourth threshold range, and the fifth threshold range can be customized by the user or the system according to actual needs. The lower limit of the third threshold range is greater than or equal to the upper limit of the fourth threshold range. The upper limit of the fourth threshold range is greater than or equal to the lower limit of the fifth threshold range. The read-write rate of the non-volatile cache is greater than the read-write rate of the non-volatile hard disk. The first non-volatile cache and the second non-volatile cache involved in this application may be the same cache or different caches deployed in the object storage device, and may be specifically determined according to actual needs, and are not limited.

In other words, the object storage device can store or write data to be read corresponding to different read frequencies or read types of addresses to be read into different non-volatile memories. Exemplarily, when the read type of the address to be read corresponding to the data to be read is frequent read, the object storage device may write the data to be read into the cache. When the read type of the address to be read corresponding to the data to be read is to be read multiple times (sequential read), the object storage may write the data to be read into the same memory (for example, memory, cache, or disk). To improve the performance and efficiency of data reading. Optionally, an overdue elimination duration (that is, an expiration duration) may be set in the memory to limit the storage duration of the data stored in the memory. When the read type of the data to be read is read once, the object storage device may directly write the data to be read to a disk without caching the data to be read. When the read type of the address to be read corresponding to the data to be read is a type used to indicate the release of the cache, the object storage device may obtain a preset storage address (such as a logical address) of the data to be cleared from the data write IO request. ) To clear the data stored in the preset storage address to release the cache and so on.

By implementing the embodiments of the present invention, by redefining the fields of the DIF domain and optimizing the storage layout of the data, compared with the prior art, the storage performance of the storage system can be improved and the working efficiency of the storage system can be improved.

Please refer to FIG. 6, which is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention. The apparatus 600 shown in FIG. 6 includes a communication module 602 and a processing module 604. among them,

In a possible implementation manner, the data processing apparatus 600 is a host. Specifically, the processing module 604 may be used to control and manage the actions of the data processing apparatus 600. Exemplarily, the processing module 604 is used to support the host to perform step S202 in FIG. 2, step S402 in FIG. 4, step S502 in FIG. 5, and / or other steps for performing the techniques described herein. The communication module 602 is used to support the communication between the data processing device 600 and other devices. For example, the communication module 602 is used by the host to perform step S204 in FIG. 2, step S404 in FIG. 4, step S504 in FIG. 5, and / or to perform the text Other steps of the described technique.

In another possible implementation manner, the data processing apparatus 600 is an object storage device. Specifically, the processing module 604 may be used to control and manage the actions of the data processing apparatus 600. Exemplarily, the processing module 604 is used to support the object storage device to perform steps S206 and S208 in FIG. 2, step S410 in FIG. 4, step S508 in FIG. 5, and / or other steps for performing the technology described herein. The communication module 602 is configured to support the communication between the data processing apparatus 600 and other devices. For example, the communication module 602 is configured to receive the data read IO request or data write IO request sent by the client from the object storage device, and / or perform the operations described herein. Other steps of the technique.

In another possible implementation manner, the data processing apparatus 600 is a client. Specifically, the processing module 604 may be used to control and manage the actions of the data processing apparatus 600. Exemplarily, the processing module 604 is configured to support the client to perform step S406 in FIG. 4 and / or to perform other steps of the technology described herein. The communication module 602 is configured to support communication between the data processing apparatus 600 and other devices. For example, the communication module 602 is configured to perform step S408 in FIG. 4, step S506 in FIG. 5, and / or other steps for performing the techniques described herein. .

Optionally, the data processing apparatus 600 may further include a storage module 606 for storing program code and data of the data processing apparatus 600.

The processing module 604 may be a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), and an application-specific integrated circuit (Application-Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logical blocks, modules, and circuits described in connection with the present disclosure. The processor may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, and so on. The communication module 602 may be a communication interface, a transceiver, a transceiver circuit, etc., where the communication interface is collectively referred to and may include one or more interfaces, such as an interface between a communication module and a processing module, an object storage device, and other devices such as a host Or client). The storage module 606 may be a memory, or other services or modules for providing storage functions.

When the processing module 604 is a processor, the communication module 602 is a communication interface, and the storage module 606 is a memory, the data processing apparatus involved in this embodiment of the present invention may be a data processing device shown in FIG. 7. The processing module 604, the communication module 602, and the storage module 606 may also be implemented by software.

Referring to FIG. 7, the data processing device 700 includes one or more processors 701, a communication interface 701, and a memory 703. Optionally, the data processing device 700 may further include a bus 704. The communication interface 703, the processor 702, and the memory 701 can be connected to each other through a bus 704. The bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (Abbreviated Architecture). EISA) bus and so on. The bus 704 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG. 7, but it does not mean that there is only one bus or one type of bus. among them:

The processor 701 may be composed of one or more general-purpose processors, such as a central processing unit (Central Processing Unit). The processor 701 may be configured to run a program of a processing function in a related program code. That is, the processor 701 executes the program code to implement the functions of the processing module. For details about the processing module, refer to related descriptions in the foregoing embodiments.

In a possible implementation manner, when the data processing device 700 is a host, the processor 701 of the host is configured to run related program codes to implement the functions of the processing modules described in this application. Or to implement step S202 in FIG. 2, step S402 in FIG. 4, step S502 in FIG. 5, and / or other steps for implementing the technology described herein, which are not limited in this application.

In another possible implementation manner, when the data processing device 700 is an object storage device, the processor 701 of the object storage device is configured to run related program code, so as to implement the functions of the foregoing processing module in this application. Alternatively, steps S206 and S208 in FIG. 2, step S410 in FIG. 4, step S508 in FIG. 5, and / or other steps for implementing the technology described herein, etc., are not limited in this application.

In another possible implementation manner, when the data processing device 700 is a client, the processor 701 of the client is configured to run related program code to implement the functions of the foregoing processing module in this application. Alternatively, step S406 in FIG. 4 of the present application, and / or other steps for implementing the technology described herein, etc. are not limited in this application. The communication interface 602 may be a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or using a wireless local area network interface) for communicating with other modules / devices. For example, the communication interface 602 in the embodiment of the present application may be specifically configured to obtain attribute information of data to be operated, such as attribute information of data to be read or data to be written.

The memory 603 may include volatile memory (Volatile Memory), such as Random Access Memory (RAM); the memory may also include non-volatile memory (Non-Volatile Memory), such as Read-Only Memory (ROM), flash memory (Flash), hard disk (HDD), or solid-state drive (SSD); memory 603 may also include a combination of the above types of memory. The memory 603 may be configured to store a set of program code, so that the processor 601 calls the program code stored in the memory 603 to implement the functions of the communication module and / or the processing module involved in the embodiment of the present invention.

It should be noted that FIG. 7 is only one possible implementation manner of the embodiment of the present application. In practical applications, the data processing device may further include more or fewer components, which is not limited herein. For the content that is not shown or described in the embodiments of the present application, reference may be made to related descriptions in any one of the foregoing embodiments in FIGS. 2 to 5, and details are not described herein again.

An embodiment of the present invention also provides a computer non-transitory storage medium. The computer non-transitory storage medium has instructions stored therein. When the computer non-transitory storage medium runs an instruction, it is shown in any one of the embodiments in FIGS. The method flow is realized.

An embodiment of the present invention also provides a computer program product. When the computer program product runs on a processor, the method flow shown in any one of the embodiments in FIG. 2 to FIG. 5 is implemented.

The steps of the method or algorithm described in connection with the disclosure of the embodiments of the present invention may be implemented in a hardware manner, or may be implemented in a manner that a processor executes software instructions. Software instructions can be composed of corresponding software modules. Software modules can be stored in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), erasable programmable read-only memory (ROM Erasable (Programmable ROM, EPROM), electrically erasable programmable read-only memory (EPROM), registers, hard disks, removable hard disks, read-only optical disks (CD-ROMs), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may reside in an ASIC. In addition, the ASIC may reside in a computing device. Of course, the processor and the storage medium may also exist as discrete components in a computing device.

A person of ordinary skill in the art may understand that all or part of the processes in the method of the foregoing embodiment may be implemented by using a computer program to instruct related hardware. The program may be stored in a computer-readable storage medium. When executed, the processes of the embodiments of the methods described above may be included. The foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disc.

Claims

A data processing method, characterized in that the method includes:

The object storage device receives a data read IO request, where the data read IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of the data to be read;

And processing the data to be read according to the attribute information of the data to be read.
The method according to claim 1, wherein the attribute information of the data to be read includes a read granularity of the data to be read, and is used to indicate a size of the pre-read data including the data to be read ;

The processing the data to be read according to attribute information of the data to be read includes:

When the size of the read-ahead data is greater than or equal to a first threshold, read the read-ahead data including the data to be read, and send the read-ahead data to the client for caching to the client Local memory; or,

When the size of the preset data is less than the first threshold, the data to be read is read, and the data to be read is sent to the host.
The method according to claim 1 or 2, wherein the attribute information of the data to be read includes read information of an address to be read corresponding to the data to be read, and the read information is used to indicate The frequency of data reading at the address to be read in a unit time,

The processing the data to be read according to attribute information of the data to be read includes:

When the read information is the first read information, write the data to be read into a cache of the object storage device; or,

When the read information is the second read information, write the data to be read into a cache of the object storage device, and set an expiration period in the cache of the object storage device to clear Data stored in the cache with a duration exceeding the expiration duration;

The frequency indicated by the first read information is greater than the frequency indicated by the second read information.
A data processing method, characterized in that the method includes:

The object storage device receives a data write IO request, where the data write IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of data to be written;

And processing the data to be written according to the attribute information of the data to be written.
The method according to claim 4, wherein the attribute information of the data to be written comprises write information of an address to be written corresponding to the data to be written, and the write information is used to indicate a unit time The frequency of data writing to the address to be written,

The processing the data to be written according to the attribute information of the data to be written includes:

When the write information is the first write information, the object storage device stores the data to be written into a cache of the object storage device; or,

When the write information is the second write information, the object storage device stores the data to be written in a hard disk of the object storage device;

The frequency indicated by the first write information is greater than the frequency indicated by the second write information.
The method according to claim 4 or 5, wherein the attribute information of the data to be written includes a writing granularity of the data to be written, and is used to indicate a storage granularity used when the data to be written is stored; The method further includes:

The client receives the data write IO request, and configures corresponding stripe information for the data to be written according to the attribute information of the data to be written. The stripe information includes the data to be used when the data to be written is stored. The stripe unit and the physical address of the stripe unit;

The client sends a stripe unit carrying data to be stored and the physical address to the object storage device according to the stripe information, and the data to be stored is data in the data to be written ;

Receiving, by the object storage device, the stripe unit carrying the data to be stored and the physical address sent by the client;

The processing the data to be written according to the attribute information of the data to be written includes:

The object storage device stores the data to be stored in a hard disk of the object storage device according to a physical address of the stripe unit.
The method according to claim 6, further comprising: the object storage device creating a storage mapping relationship of the data to be stored according to a write granularity of the data to be written, the storage The mapping relationship includes a mapping relationship between a logical address and the physical address when the data to be stored is stored, and the logical address is associated with the address to be written.
A data processing method, characterized in that the method includes:

The host acquires attribute information of the data to be operated, and generates a data operation request according to the attribute information of the data to be operated;

Sending the data operation request to a client, where the data operation request includes a data integrity field DIF, and the DIF is used to carry attribute information of the data to be operated.
The method according to claim 8, characterized in that, when the data operation request is a data read IO request, the attribute information of the data to be operated includes the read granularity of the data to be read and / or the data to be read Read information of the corresponding address to be read;

The read granularity of the data to be read is used to indicate the size of the pre-read data including the data to be read;

The read information of the address to be read is used to indicate how often data is read from the address to be read in a unit time.
The method according to claim 8, characterized in that when the data operation request is a data write IO request, the attribute information of the data to be operated includes a write granularity of data to be written and / or data to be written Write information of the corresponding address to be written;

The write granularity of the data to be written is used to indicate a storage granularity used when the data to be written is stored;

The writing information of the address to be written is used to indicate a frequency of data writing to the address to be written in a unit time.
A data processing device, comprising a communication module and a processing module; wherein,

The communication module is configured to receive a data read IO request, where the data read IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of data to be read;

The processing module is configured to process the data to be read according to attribute information of the data to be read.
A data processing device, comprising a communication module and a processing module; wherein,

The communication module is configured to receive a data write IO request, where the data write IO request includes a data integrity field DIF, and the DIF is used to carry attribute information of data to be written;

The processing module is configured to process the data to be written according to attribute information of the data to be written.
A data processing device, comprising a communication unit and a processing unit; wherein,

The processing unit is configured to obtain attribute information of data to be operated;

The communication unit is configured to send a data operation request to a client, where the data operation request includes a data integrity field DIF, and the DIF is used to carry attribute information of the data to be operated.
An object storage device, comprising: a processor, a memory; the processor and the memory establish communication; wherein the memory is used to store instructions; the processor is used to call instructions in the memory, The method according to any one of claims 1-3 is performed.
An object storage device, comprising: a processor, a memory; the processor and the memory establish communication; wherein the memory is used to store instructions; the processor is used to call instructions in the memory, Perform the method according to any one of claims 4-7.
A host is characterized in that it includes a processor, a memory, the processor, and the memory establishing communication; wherein the memory is used to store instructions; the processor is used to call the instructions in the memory to execute A method as claimed in any one of claims 8-10 above.
A computer non-transitory storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 3 when executed by a computing device.
A computer non-transitory storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 4 to 7 when executed by a computing device.
A computer non-transitory storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 8 to 10 when executed by a computing device.