WO2025145672A1

WO2025145672A1 - Methods, systems, and storage mediums for storing data

Info

Publication number: WO2025145672A1
Application number: PCT/CN2024/119790
Authority: WO
Inventors: Wenlong JIANG; Xiaolong NIU; Yunjun GAO; Mingwei Zhou; Tao Xu; Xin Luo; Bofang ZHENG
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2024-01-04
Filing date: 2024-09-19
Publication date: 2025-07-10
Anticipated expiration: 2026-07-04
Also published as: CN117539407B; CN117539407A

Abstract

A storing data method is provided. The method includes obtaining a candidate storage space corresponding to data to be stored. The candidate storage space is divided into a plurality of stripes, and each stripe of the plurality of strips includes a preset count of storage regions. The preset count of storage regions are sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes are a storage unit. The method includes determining at least one target stripe corresponding to the data to be stored from the plurality of stripes and generating storage instructions corresponding to the data to be stored. Each storage unit in the at least one target stripe corresponds to one storage instruction. The method further includes writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions.

Description

METHODS, SYSTEMS, AND STORAGE MEDIUMS FOR STORING DATA

CROSS-REFERENCE RELATED TO APPLICATIONS

This application claims priority to Chinese application No. 202410012614.5, filed January 4, 2024, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of data storage, and in particular, to methods, systems, and storage mediums for storing data.

BACKGROUND

With the development of big data, distributed storage has become a key technology to address the explosive growth of data. When distributed storage is performed, data is written to a cache. The cache is divided into stripes. Each stripe consists of a plurality of storage regions. In the process of data storage, each storage region corresponds to a storage instruction (i.e., input/output (I/O) instruction) for storing and writing data, which leads to a large number of storage instructions in the process of distributed storage and affects the efficiency of data writing and storage performance.

Therefore, it is desirable to provide methods, systems, and storage mediums for storing data to improve the efficiency of data writing and the storage performance.

SUMMARY

One or more embodiments of the present disclosure provide a method for storing data. The method may be performed by a computing device having at least one processor and at least one storage device. The method may include obtaining a candidate storage space corresponding to data to be stored from the at least one storage device. The candidate storage space may be divided into a plurality of stripes. Each stripe of the plurality of stripes may include a preset count of storage regions. The preset count of storage regions may be sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes may be a storage unit. The method may include determining at least one target stripe corresponding to the data to be stored from the plurality of stripes. The method may include generating storage instructions corresponding to the data to be stored. Each storage unit in the at least one target stripe corresponds to one storage instruction. The method may include writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions.

In some embodiments, the method may further include determining whether a data volume of the data to be stored exceeds a preset data volume threshold, in response to a determination that the data volume does not exceed the preset data volume threshold, determining that the data to be stored corresponds to a plurality of data objects, and in response to a determination that the data volume exceeds the preset data volume threshold, determining that the data to be stored corresponds to a same data object.

In some embodiments, the data to be stored may correspond to the plurality of data objects, and the generating storage instructions corresponding to the data to be stored may include obtaining an initial offset of the at least one target stripe, the initial offset being a position, in the data to be stored, corresponding to initial data stored in each target stripe of the at least one target stripe; determining a sorting result by sorting, based on the initial offset, the at least one target stripe, and generating the storage instructions corresponding to the data to be stored based on the sorting result.

In some embodiments, the data to be stored may correspond to the same data object, and storage addresses of a plurality of storage regions in each storage unit in the at least one target stripe may be consecutive.

In some embodiments, the writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions may include for each storage instruction of the storage instructions, serializing, based on a preset algorithm, the data to be stored corresponding to the storage instruction, and writing, based on the storage instruction, the serialized data to be stored to a storage unit corresponding to the storage instruction.

In some embodiments, the data to be stored may correspond to the plurality of data objects, and the serializing the data to be stored corresponding to the storage instruction may include for each storage region of a plurality of storage regions corresponding to the storage instruction, obtaining an initial storage address, in the data to be stored, of data stored in the storage region, and serializing the data to be stored corresponding to the storage instruction based on initial storage addresses and lengths of data corresponding to the plurality of storage regions using the preset algorithm.

In some embodiments, the initial storage address and the length of data may be obtained through a struct I/O vector (iovec) .

In some embodiments, the data to be stored may correspond to the same data object, and the serializing the data to be stored corresponding to the storage instruction may include obtaining an initial address of consecutive storage addresses of a plurality of storage regions corresponding to the storage instruction, obtaining an address length of the data to be stored corresponding to the storage instruction, and serializing the data to be stored corresponding to the storage instruction based on the initial address and the address length using the preset algorithm.

In some embodiments, storage regions of the plurality of stripes may be dispersed on one or more storage nodes, and storage regions in a same storage unit may be located in a same storage node.

In some embodiments, the method may further include dividing the candidate storage space into the plurality of stripes using a target division manner. The target division manner may be determined by determining at least one candidate division manner of the candidate storage space based on the data to be stored, determining, through a prediction model, a cache success rate and a cache speed corresponding to each candidate division manner of the at least one candidate division manner based on a size of the data to be stored, a storage capacity of the at least one storage device, a storage capacity of the candidate storage space, and the at least one candidate division manner, and determining, based on the at least one candidate division manner, the cache success rate, and the cache speed, the target division manner.

One or more embodiments of the present disclosure provide a system for storing data. The system may include an obtaining module, a determination module, a generation module, and a storage module. The obtaining module may be configured to obtain a candidate storage space corresponding to data to be stored from at least one storage device. The candidate storage space is divided into a plurality of stripes, each stripe of the plurality of stripes includes a preset count of storage regions. The preset count of storage regions are sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes are a storage unit. The determination module may be configured to determine at least one target stripe corresponding to the data to be stored from the plurality of stripes. The generation module may be configured to storage instructions corresponding to the data to be stored. Each storage unit in the at least one target stripe corresponds to one storage instruction. The storage module may be configured to write, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions.

One or more embodiments of the present disclosure provide a system. The system may include at least one processor and at least one storage device storing computer instructions. When executed by the at least one processor, the computer instructions may cause the at least one processor to perform the method described in embodiments of the present disclosure.

One or more embodiments of the present disclosure provide a computer-readable storage medium storing computer instructions. When reading the computer instructions in the storage medium, a computer may perform the method described in embodiments of the present disclosure.

Additional features will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the examples. The features of the present disclosure may be realized and attained by practice or use of various aspects of the methodologies, instrumentalities, and combinations set forth in the detailed examples discussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is further described in terms of exemplary embodiments. These exemplary embodiments are described in detail with reference to the drawings. The drawings are not to scale. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent similar structures throughout the several views of the drawings, and wherein:

FIG. 1 is a schematic diagram illustrating an exemplary data storage system according to some embodiments of the present disclosure;

FIG. 2 is a schematic diagram illustrating an exemplary processing device according to some embodiments of the present disclosure;

FIG. 3 is a flowchart illustrating an exemplary process for storing data according to some embodiments of the present disclosure;

FIG. 4 is a schematic diagram illustrating exemplary divisions of data to be stored and stripes according to some embodiments of the present disclosure;

FIG. 5 is a schematic diagram illustrating an exemplary process for determining a division manner of data to be stored and a candidate storage space according to some embodiments of the present disclosure;

FIG. 6 is a schematic diagram illustrating an exemplary process for generating a storage instruction according to some embodiments of the present disclosure;

FIG. 7 is a schematic diagram illustrating sorting target stripes based on corresponding initial offsets according to some embodiments of the present disclosure;

FIG. 8 is a schematic diagram illustrating consecutive storage addresses of a plurality of storage regions according to some embodiments of the present disclosure;

FIG. 9A is a schematic diagram illustrating generating storage instructions according to some embodiments of the present disclosure;

FIG. 9B is a schematic diagram illustrating generating storage instructions according to other embodiments of the present disclosure; and

FIG. 10 is a flowchart illustrating an exemplary process for writing data to be stored according to some embodiments of the present disclosure.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled in the art to make and use the present disclosure and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms “a, ” “an, ” and “the” may be intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprise, ” “comprises, ” and/or “comprising, ” “include, ” “includes, ” and/or “including” when used in this disclosure, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It will be understood that the term “system, ” “engine, ” “unit, ” “module, ” and/or “block” used herein are one method to distinguish different components, elements, parts, sections or assembly of different levels in ascending order. However, the terms may be displaced by another expression if they achieve the same purpose.

It will be understood that, although the terms “first, ” “second, ” “third, ” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of exemplary embodiments of the present disclosure.

These and other features, and features of the present disclosure, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, may become more apparent upon consideration of the following description with reference to the accompanying drawings, all of which form a part of this disclosure. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended to limit the scope of the present disclosure. It is understood that the drawings are not to scale.

The flowcharts used in the present disclosure illustrate operations that systems implement according to some embodiments in the present disclosure. It is to be expressly understood, the operations of the flowchart may be implemented not in order. Conversely, the operations may be implemented in an inverted order, or simultaneously. Moreover, one or more other operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

When the distributed cloud storage of data is performed, the data may be first stored temporarily in the cache layer. The cache layer may be usually divided into a plurality of stripes. Each stripe of the plurality of stripes may consist of a plurality of storage regions that are dispersed on a plurality of storage nodes. During data writing, data writing of each storage region may trigger a corresponding storage instruction. Therefore, a large number of storage instructions may be generated when distributed storage is performed, which results in low data writing efficiency and storage performance.

The present disclosure provides methods, systems, and storage mediums for storing data. The method may include obtaining a candidate storage space corresponding to data to be stored from the at least one storage device. The candidate storage space may be divided into a plurality of stripes. Each stripe of the plurality of stripes may include a preset count of storage regions. The preset count of storage regions may be sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes may be a storage unit. The method may include determining at least one target stripe corresponding to the data to be stored from the plurality of stripes. The method may further include generating storage instructions corresponding to the data to be stored and writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions. Each storage unit in the at least one target stripe may correspond to one storage instruction.

In the embodiments of the present disclosure, the at least one target stripe corresponding to the data to be stored may be obtained, and storage regions having a same sorting position in the at least one target stripe may correspond to one storage instruction, so that when the data is written, the storage regions having the same sorting position in all the target stripes may be written to using only one storage instruction instead of a case where data writing of each storage region triggers the corresponding storage instruction, which avoids the generation of too many storage instructions, thereby improving the data writing efficiency and storage performance.

FIG. 1 is a schematic diagram illustrating an exemplary data storage system 100 according to some embodiments of the present disclosure. As shown in FIG. 1, the data storage system 100 may include a storage device 110, a processing device 120, a terminal 130, and a network 140.

The storage device 110 may store data and/or instructions. The data and/or instructions may be obtained from, for example, the processing device 120, and/or any other component of the data storage system 100. In some embodiments, the storage device 110 may store data and/or instructions that the processing device 120 may execute or use to perform exemplary methods described in the present disclosure. In some embodiments, the storage device 110 may include a hard disk device, a mass storage device, a removable storage device, a volatile read-and-write memory, a read-only memory (ROM) , or the like, or any combination thereof. In some embodiments, the storage device 110 may be implemented on a cloud platform. In some embodiments, the storage device 110 may be part of the processing device 120.

The processing device 120 may process information and/or data relating to the data storage system 100 to perform one or more functions described in the present disclosure. For example, the processing device 120 may obtain a candidate storage space corresponding to data to be stored from the storage device 110. The candidate storage space may be divided into a plurality of stripes. Each stripe of the plurality of stripes may include a preset count of storage regions. The preset count of storage regions may be sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes may be a storage unit. The processing device 120 may determine at least one target stripe corresponding to the data to be stored from the plurality of stripes. Furthermore, the processing device 120 may generate storage instructions corresponding to the data to be stored, and write, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions. Each storage unit in the at least one target stripe may correspond to one storage instruction.

The processing device 120 may be a single server or a server group. The server group may be centralized, or distributed (e.g., the processing device 120 may be a distributed system) . In some embodiments, the processing device 120 may be local or remote. For example, the processing device 120 may access information and/or data stored in the storage device 110 via the network 140. As another example, the processing device 120 may be directly connected to the storage device 110 to access stored information and/or data. In some embodiments, the processing device 120 may be implemented on a cloud platform. Merely by way of example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud, a multi-cloud, or the like, or any combination thereof.

In some embodiments, the processing device 120 may include one or more processors (e.g., single-core processor (s) or multi-core processor (s) ) . Merely by way of example, the processing device 120 may include a central processing unit (CPU) , an application-specific integrated circuit (ASIC) , an application-specific instruction-set processor (ASIP) , a graphics processing unit (GPU) , a physics processing unit (PPU) , a digital signal processor (DSP) , a field-programmable gate array (FPGA) , a programmable logic device (PLD) , a controller, a microcontroller unit, a reduced instruction-set computer (RISC) , a microprocessor, or the like, or any combination thereof.

The terminal 130 may interact with a user. The user may issue an operation instruction to the processing device 120 via the terminal 130 to cause the processing device 120 to complete a specified operation, for example, to write the data to be stored to corresponding storage units. In some embodiments, the terminal 130 may cause the processing device 120 to perform exemplary methods described in the present disclosure through instructions. In some embodiments, the terminal 130 may include other devices with input and/or output functions such as a mobile device 130-1, a tablet computer 130-2, a laptop computer 130-3, a desktop computer, or the like, or any combination thereof.

The network 140 may include any suitable network that can facilitate the exchange of information and/or data for the data storage system 100. In some embodiments, one or more components (e.g., the processing device 120) of the data storage system 100 may communicate information and/or data with one or more other components of the data storage system 100 via the network 140. For example, the processing device 120 may write data to the storage device 110 or read the data from the storage device 110 via the network 150. In some embodiments, the network 150 may be or include a wired network, a wireless network (e.g., an 802.11 network, a Wi-Fi network) , etc.

It should be noted that the above description is merely provided for the purposes of illustration, and not intended to limit the scope of the present disclosure. For persons having ordinary skills in the art, multiple variations and modifications may be made under the teachings of the present disclosure. In some embodiments, the data storage system 100 may include one or more additional components and/or one or more components of the data storage system 100 described above may be omitted. Additionally or alternatively, two or more components of the data storage system 100 may be integrated into a single component. A component of the image processing system 100 may be implemented on two or more sub-components. However, those variations and modifications do not depart from the scope of the present disclosure.

FIG. 2 is a schematic diagram illustrating an exemplary processing device 120 according to some embodiments of the present disclosure. In some embodiments, the processing device 120 may include an obtaining module 210, a determination module 220, a generation module 230, and a storage module 240.

The obtaining module 210 may be configured to obtain a candidate storage space corresponding to data to be stored from at least one storage device (e.g., the storage device 110) . The candidate storage space may be divided into a plurality of stripes. Each stripe of the plurality of stripes may include a preset count of storage regions. The preset count of storage regions may be sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes may be a storage unit. More descriptions of the obtaining a candidate storage space may be found elsewhere in the present disclosure (e.g., operations 310 and the descriptions thereof) .

The determination module 220 may be configured to determine at least one target stripe corresponding to the data to be stored from the plurality of stripes. More descriptions of the determining at least one target stripe may be found elsewhere in the present disclosure (e.g., operations 320 and the descriptions thereof) .

The generation module 230 may be configured to generate storage instructions corresponding to the data to be stored. Each storage unit in the at least one target stripe may correspond to one storage instruction. More descriptions of the generating storage instructions may be found elsewhere in the present disclosure (e.g., operations 330 and the descriptions thereof) .

The storage module 240 may be configured to write, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions. More descriptions of the writing the data to be stored may be found elsewhere in the present disclosure (e.g., operations 340 and the descriptions thereof) .

In some embodiments, the processing device 120 may further include a division module 250 configured to divide the candidate storage space into the plurality of stripes using a target division manner or divide the data to be stored into a plurality of data blocks and check blocks. More descriptions of the dividing the candidate storage space and the data to be stored may be found elsewhere in the present disclosure (e.g., FIG. 4 and FIG. 5 and the descriptions thereof) .

It should be noted that the above descriptions of the processing device 120 are provided for the purposes of illustration, and are not intended to limit the scope of the present disclosure. For persons having ordinary skills in the art, various modifications and changes in the forms and details of the application of the above system may occur without departing from the principles of the present disclosure. In some embodiments, two or more modules of the processing device 120 may be integrated into a single module and/or a module may be divided into two or more units. For example, the determination module 220 and the generation module 230 may be combined into a single module that determines the at least one target stripe and generates the storage instructions corresponding to the data to be stored. In some embodiments, the processing device 120 may include one or more other modules and/or one or more modules described above may be omitted. For example, the division module 250 may be omitted, and the function of the division module 250 may be performed by other modules, for example, the obtaining module 210. However, those modifications and changes do not depart from the scope of the present disclosure.

FIG. 3 is a flowchart illustrating an exemplary process for storing data according to some embodiments of the present disclosure. In some embodiments, process 300 may be executed by the data storage system 100. For example, the process 300 may be implemented as a set of instructions (e.g., an application) stored in a storage device (e.g., the storage device 110 illustrated in FIG. 1) . The processing device 120 of the data storage system 100 and/or one or more modules of the processing device 120 illustrated in FIG. 2 may execute the set of instructions and may accordingly be directed to perform the process 300.

In 310, a candidate storage space corresponding to data to be stored may be obtained from the at least one storage device (e.g., the storage device 110) . In some embodiments, operation 310 may be performed by the obtaining module 210.

The data to be stored refers to data that needs to be stored, for example, images, text, etc. that need to be stored. In some embodiments, the processing device 120 may obtain the data to be stored in various ways. For example, the processing device 120 may obtain the data to be stored by obtaining user input information. As another example, the processing device 120 may obtain the data to be stored from a storage device inside or outside the data storage system 100.

The candidate storage space refers to a storage space configured to store data to be stored, for example, a data cache space, etc. In some embodiments, the processing device 120 may directly request a space of a preset storage capacity in the storage device 110 of the data storage system 100 as the candidate storage space. The preset storage capacity may be determined based on experience or demand.

In some embodiments, the processing device 120 may divide the data to be stored based on an Erasure Correcting Code (EC) technology. For example, the processing device 120 may divide the data to be stored into N data blocks and M check blocks. The data blocks refer to real data blocks corresponding to the data to be stored. The check blocks refer to simulated data blocks calculated by an EC algorithm based on the data to be stored for data recovery. It is understandable that when there is data corruption in the data blocks, data of the data blocks can be inversed according to the check blocks through the EC algorithm, thereby ensuring that the data is not lost. N data blocks and M check blocks may correspond to at least one division group, and different division groups may include a same count of data blocks and a same count of check blocks. In each division group, the data blocks may have a same data volume, the check blocks may have a same data volume, and a maximum data volume of the data blocks may be same as that of the check blocks. An actual data volume stored in the data blocks of different division groups may be the same or different, and a data volume of the check blocks of the different division groups may be the same.

In some embodiments, the processing device 120 may divide the data to be stored based on a count of data blocks and a count of check blocks included in each division group, and a maximum data volume of the data blocks and a maximum data volume of the check blocks. Specifically, the processing device 120 may determine whether the data to be stored is greater than a total maximum data volume of the data blocks in each division group, and divide the data to be stored based on a determination result. FIG. 4 is a schematic diagram illustrating exemplary divisions of data to be stored and stripes according to some embodiments of the present disclosure. For example, as shown in FIG. 4, each division group may include 3 data blocks and 1 check block, and the maximum data volume of each data block and the check block may be 1 M. When the data volume of the data to be stored is 3M, the processing device 120 may determine that the data to be stored is smaller than or equal to the total maximum data volume of 3 M of the 3 data blocks, and divide the data to be stored into 3 data blocks and 1 check block, and the data volume of each data block and the check block may be 1 M. As another example, when the data volume of the data to be stored is 4.8 M, the processing device 120 may determine that the data to be stored is greater than the total maximum data volume of 3 M of the 3 data blocks. The processing device 120 may first divide the data to be stored into 3 data blocks of 1 M, correspondingly determine 1 check block of 1 M (i.e., the 3 data blocks of 1 M and the 1 check block of 1 M are a division group) . Further, the processing device 120 may divide the remaining 1.8 M into 3 new data blocks, the data volume of each data block of the 3 new data blocks being 1.8/3 M, and correspondingly determine 1 check block of 1 M (i.e., the 3 new data blocks of 1.8/3 M and the 1 check block of 1 M are another division group) . That is, the final division result may be 1 M (data block) + 1 M (data block) + 1 M (data block) + 1 M (check block) and 0.6 M (data block) + 0.6 M (data block) + 0.6 M (data block) + 1 M (check block) .

In some embodiments, the candidate storage space may be divided into a plurality of stripes. The stripe refers to a data read/write unit consisting of a single or a plurality of consecutive regions in a hard disk. A count of the plurality of stripes may be preset based on experience or demand. In some embodiments, each of the plurality of stripes may include a preset count of storage regions. The storage regions may be data regions and check regions. The data region may be configured to store the data to be stored, and the check region may be configured to store check data. The check data refers to simulated data calculated by the EC algorithm based on the data to be stored and configured for data recovery. A count of data regions and a count of check regions included in each stripe may be determined based on the division result of the data to be stored. For example, the count of data regions may be equal to the count of data blocks, and the count of check regions may be equal to the count of check blocks. In some embodiments, a maximum storage capacity of each storage region may be predetermined. For example, the maximum storage capacity of each storage region may be 32 kb. More descriptions of the division manner of the data to be stored and the candidate storage space may be found in FIG. 5 and the related descriptions thereof.

In some embodiments, the preset count of storage regions may be sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes may be a single storage unit. As shown in FIG. 4, a candidate storage space may be divided into n stripes. Each stripe of the stripe 1 to stripe n may include four storage regions sequentially arranged, including piece 1, piece 2, piece 3, and piece 4. Piece 1, piece 2, and piece 3 may be the data regions, and piece 4 may be the check region. Piece 1, piece 2, piece 3, and piece 4 may be sorted at positions 1, 2, 3, and 4, respectively. Pieces 1 of stripe 1 to stripe n may be a storage unit. Pieces 2 of stripe 1 to stripe n may be a storage unit. Pieces 3 of stripe 1 to stripe n may be a storage unit. Pieces 4 of stripe 1 to stripe n may be a storage unit.

In 320, at least one target stripe corresponding to the data to be stored may be determined from the plurality of stripes. In some embodiments, operation 320 may be performed by the determination module 220.

Understandably, after the data to be stored is written to the plurality of stripes in the candidate storage space, there may be a stripe where no data is written, a stripe where data is partially written, a stripe where data is fully written, etc. In the embodiments of the present disclosure, the target stripe may be determined from the plurality of stripes. In some embodiments, the target stripe may be a stripe that satisfies a first preset rule. The first preset rule may be set based on experience or demand. For example, the first preset rule may include, in response to a fact that the data to be stored corresponds to a same data object (e.g., a big picture) , if the data to be stored is not finished being stored and all the stripes are fully-written, determining that the target stripe is the fully-written stripes, or if the data to be stored is finished being stored, determining that the target stripe is a data-stored stripe (including the fully-written stripes and the non-fully-written stripes) ; and in response to a fact that the data to be stored corresponds to a plurality of data objects (e.g., a plurality of small figures) , if the data to be stored is finished being stored and a count of stripes with data exceeds a preset count threshold, determining that the target stripe is the data-stored stripes (including the fully-written stripes and the non-fully-written stripes) .

In some embodiments, the preset count threshold may be dynamically adjusted. Specifically, the preset count threshold may be determined based on remaining computing resources of a current system. Remaining computing resources refer to computing resources available for use at the current moment. In some embodiments, the more the remaining computing resources are, the greater the preset count threshold set by the processing device 120 may be. Understandably, when the remaining computing resources are relatively sufficient, the system may process more data at the same time. When the remaining computing resources are relatively tight, the data should be distributed for processing as much as possible to avoid overloading of the system caused by too much data processing at one time. In the embodiments of the present disclosure, by setting a dynamically adjusted preset count threshold, a more reasonable data processing mode may be selected based on the actual operation of the system to reduce the burden of the system.

In some embodiments, the processing device 120 may traverse the plurality of stripes, and determine the stripe that satisfies the first preset rule as the target stripe.

In 330, storage instructions corresponding to the data to be stored may be generated. In some embodiments, operation 330 may be performed by the generation module 230.

The storage instructions refer to instructions that perform data storage. For example, an IO instruction for data storage. In some embodiments, each storage unit in the at least one target stripe may correspond to one storage instruction. Storage regions having a same sorting position in the at least one target stripe may be a storage unit. The storage unit may correspond to one storage instruction. For example, as shown in FIG. 4, if stripe 1-stripe n are target stripes, pieces 1 of stripe 1 to stripe n may be one storage unit that corresponds to one storage instruction, pieces 2 of stripe 1 to stripe n may be one storage unit that corresponds to one storage instruction, pieces 3 of stripe 1 to stripe n may be one storage unit that corresponds to one storage instruction, and pieces 4 of stripe 1 to stripe n may be one storage unit that corresponds to one storage instruction.

In some embodiments, the processing device 120 may determine whether a data volume of the data to be stored exceeds a preset data volume threshold. In response to a determination that the data volume does not exceed the preset data volume threshold, the processing device 120 may determine that the data to be stored corresponds to a plurality of data objects. In response to a determination that the data volume exceeds the preset data volume threshold, the processing device 120 may determine that the data to be stored corresponds to a same data object. When the data to be stored corresponds to the plurality of data objects, the processing device 120 may obtain an initial offset of the at least one target stripe, determine a sorting result by sorting, based on the initial offset, the at least one target stripe, and generate the storage instructions corresponding to the data to be stored based on the sorting result. When the data to be stored corresponds to the same data object, the processing device 120 may determine that storage addresses of a plurality of storage regions in each storage unit in the at least one target stripe are consecutive, and determine the storage instructions corresponding to the data to be stored. More description of the determining storage instructions may be found in FIG. 6 and the related descriptions thereof.

In 340, based on the storage instructions, the data to be stored may be written to storage units corresponding to the storage instructions. In some embodiments, operation 340 may be performed by the storage module 240.

In some embodiments, the processing device 120 may directly write, based on the storage instructions, the data to be stored to the storage units corresponding to the storage instructions.

In some embodiments, for each storage instruction of the storage instructions, the processing device 120 may serialize, based on a preset algorithm, the data to be stored corresponding to the storage instruction, and, write, based on the storage instruction, the serialized data to be stored to a storage unit corresponding to the storage instruction. More descriptions of the writing the data to be stored to a storage unit corresponding to the storage instruction may be found in FIG. 10 and the related descriptions thereof.

In the embodiments of the present disclosure, each storage unit in at least one target stripe may correspond to one storage instruction. When the data is written, the storage regions having the same sorting position in all the target stripes may be written to using only one storage instruction instead of a case where data writing of each storage region triggers the corresponding storage instruction, which is equivalent to merging the storage instructions of the storage regions having the same sorting position in all the target stripes, and avoids generation of too many storage instructions, thereby improving the data writing efficiency and storage performance.

FIG. 5 is a schematic diagram illustrating an exemplary process 500 for determining a division manner of data to be stored and a candidate storage space according to some embodiments of the present disclosure.

As shown in FIG. 5, the processing device 120 may determine at least one candidate division manner 520 based on the data to be stored 510. The candidate division manner 520 refers to a candidate division manner of the data to be stored 510 and the candidate storage space. In some embodiments, the candidate division manner 520 may include a count of division groups, a count of data blocks and a count of check blocks in each division group, and a count of stripes. In some embodiments, the processing device 120 may obtain the candidate division manner 520 in various ways. For example, the processing device 120 may determine the candidate division manner 520 based on user input information. Exemplarily, the processing device 120 may randomly generate, based on a range of the count of data blocks, a range of the count of check blocks, and a range of the count of stripes input by the user, the count of data blocks and the count of check blocks in each division group, and the count of stripes in the above ranges. As another example, the processing device 120 may obtain historical data, and determine a historical division manner in the historical data as the candidate division manner 520.

Furthermore, the processing device 120 may determine, through a prediction model 540, a cache success rate 551 and a cache speed 552 corresponding to each candidate division manner 520 based on a size of the data to be stored 531, a storage capacity of the at least one storage device 532, a storage capacity of the candidate storage space 533, and the candidate division manner 520. The prediction model 540 may be a trained machine learning model. Merely by way of example, the prediction model 540 may include a neural network (NN) model, a deep neural network (DNN) model, or the like, or any combination thereof. An input of the prediction model 540 may include the size of the data to be stored 531, the storage capacity of the at least one storage device 532, the storage capacity of the candidate storage space 533, and the candidate division manner 520. An output of the prediction model may include the cache success rate 551 and the cache speed 552 corresponding to each candidate division manner 520. The cache success rate 551 refers to a parameter characterizing a degree of difficulty of caching. In some embodiments, the cache success rate 551 may be a percentage of a volume of data that failed to be cached to the data volume of the data to be stored 510 during the process of storing the data to be stored 510 into the candidate storage space. The cache speed 552 refers to a parameter characterizing a time to store a unit of data. In some embodiments, the cache speed 552 may be a ratio of a total time for the data to be stored 510 to be stored in the candidate storage space to the data volume of the data to be stored 510. In some embodiments, the processing device 120 may directly obtain the relevant data (e.g., the volume of data that failed to be cached, the total time for the data to be stored 510 to be stored in the candidate storage space, etc. ) , and calculate to determine the cache success rate (the cache success rate = the volume of data that failed to be cached /the data volume of data to be stored x 100%) and the cache speed (the cache speed= the total time for the data to be stored to be stored in the candidate storage space /the data volume of data to be stored.

In some embodiments, the input of the prediction model 540 may also include a software type 534 that generates the data to be stored 510. For example, the software type 534 may include an office software (e.g., Microsoft Office, WPS Office, etc. ) , an image processing software (e.g., Adobe Photoshop, GIMP, etc. ) , an audio/video playback software (e.g., VLC, Windows Media Player, etc. ) , a game software, etc. Understandably, when the system needs to store data constantly, different types of software may have different features when generating data, which may affect the cache success rate 551 and the cache speed 552 of data.

Usually, during the data writing process, data writing of each storage region in each stripe triggers a corresponding storage instruction. According to the embodiment of the present disclosure, when the data is written, the storage regions having the same sorting position in all the target stripes may be written into using only a single storage instruction instead of the case where data writing of each storage region triggers the corresponding storage instruction, which is equivalent to merging the storage instructions of the storage regions having the same sorting position of all the target stripes. In some embodiments, the output of the prediction model 540 may also include a merging rate 553 corresponding to each candidate division manner 520. The merging rate 553 refers to a parameter reflecting a degree of reduction of the storage instructions caused by embodiments of the present disclosure. In some embodiments, the merging rate 553 may be a percentage of a difference between a count of storage instructions before merging and a count of storage instructions after merging to the count of storage instructions before merging. The count of storage instructions before merging refers to a count of storage instructions required when data writing of each storage region triggers the corresponding storage instruction. The count of storage instructions after merging refers to a count of storage instructions required when the storage regions having the same sorting position in all the target stripes may be written into using only one storage instruction according to embodiments of the present disclosure.

In some embodiments, the prediction model 540 may be obtained by training. For example, the processing device 120 may obtain a training sample and obtain a trained prediction model 540 by training an initial prediction model based on the training sample. The training sample may include training data and a label corresponding to the training data. The training data may include a size of sample data to be stored, a storage capacity of a sample storage device, a storage capacity of a sample candidate storage space, and a sample candidate division manner. The label may include a cache success rate and a cache speed corresponding to the sample candidate division manner. In some embodiments, the processing device 120 may obtain the training sample in various ways. For example, the processing device 120 may obtain the training data in the training sample through historical data. As another example, the processor may obtain the label in the training sample through manual labeling.

Furthermore, the processing device 120 may determine a target division manner 560 based on the candidate division manner 520, the cache success rate 551, and the cache speed 552. The target division manner 560 refers to a final division manner of the data to be stored 510 and the candidate storage space. In some embodiments, the processing device 120 may determine, through a third preset rule, the target division manner 560 based on the candidate division manner 520, the cache success rate 551, and the cache speed 552. The third preset rule may be determined based on experience or demand. For example, the third preset rule may be to screen the candidate division manners 520 in which the cache success rate 551 is greater than a preset success rate threshold, and further screen a candidate division manner in which the cache speed 552 is greater than a preset speed threshold and the cache speed is greatest from the screened candidate division manners as the target division manner 560. As another example, the third preset rule may be to further screen the candidate division manner with a highest merging rate 553 as the target division manner based on the preceding embodiment. As yet another example, the third preset rule may be to determine, through a weighted calculation manner, weighted results based on the cache success rate 551 and the cache speed 552, and determine a candidate division manner with a largest weighted result as the target division manner 560 by sorting the weighted results from large to small. The weights may be preset based on experience or demand. In some embodiments, if there is no candidate division manner that satisfies the third preset rule, the processing device 120 may regenerate different candidate division manners, repeat the relevant steps of determining the target division manner 560 until the target division manner 560 that satisfies the third preset rule is determined.

In some embodiments, different data to be stored may correspond to different candidate division manners 520. For example, different data to be stored 510 may correspond to a different count of division groups, a count of data blocks and a count of check blocks in each division group, and a count of stripes.

In the embodiment of the present disclosure, the cache success rate 551 and the cache speed 552 corresponding to the candidate division manner 520 may be determined through the prediction model 540, and the target division manner 560 may be further determined, so that a number of influencing factors may be comprehensively referred to during the determination process of the target division manner 560, thereby determining a more accurate and reasonable division manner of the data to be stored and the candidate storage space. By setting the input of the prediction model 540 including the software type 534 that generates the data to be stored, the relevant features of data generated and transmitted by different types of software can be taken into account when the division manner is determined, thereby improving the accuracy of determining the division manner. By setting the output of the prediction model 540 including the merging rate 553 corresponding to the candidate division manner 520, a division manner with a better optimization effect may be determined to consider an instruction simplification effect achieved by the division manner when the target division manner 560 is determined in a subsequent process. By setting the different candidate division manners corresponding to the different data to be stored, different candidate division manners may be flexibly determined based on the relevant features of the different data to be stored, thereby resulting in better storage effects.

FIG. 6 is a schematic diagram illustrating an exemplary process for generating a storage instruction according to some embodiments of the present disclosure. In some embodiments, at least part of the process 600 may be performed to achieve at least part of operation 330 as described in connection with FIG. 3. For example, the processing device 120 or the generation module 230 may generate the storage instruction 660 by executing at least a portion of the process 600.

As shown in FIG. 6, the processing device 120 may determine whether a data volume of data to be stored 610 exceeds a preset data volume threshold 620. The preset data volume threshold 620 may be preset based on experience or demand. In response to a determination that the data volume of data to be stored 610 does not exceed the preset data volume threshold 620, the processing device 120 may determine that the data to be stored corresponds to a plurality of data objects. The data object refers to a specific object of a source of the data to be stored. For example, the data object may include an image 1, an image n, a text 1, a text n, etc. In response to a determination that the data volume of data to be stored 610 exceeds the preset data volume threshold 620, the processing device 120 may determine that the data to be stored corresponds to a same data object. Exemplarily, the data volume of data to be stored 610 may be a M, and the preset data volume threshold 620 may be b M. If a ≤ b, the processing device 120 may determine that the data to be stored corresponds to the plurality of data objects (e.g., the data to be stored corresponds to image 1-image n) . If a>b, the processing device 120 may determine that the data to be stored corresponds to the same data object (e.g., the data to be stored corresponds to image 1) .

Understandably, if the data volume of data to be stored 610 is different, the corresponding data object may be different, and the way of subsequently determining the storage instruction 660 may be also different. For example, if the data object is an image, and when a large image with a large data volume of data to be stored 610 is written, all the stripes of the candidate storage space may be filled with the large image. Usually, each storage region corresponding to each stripe may be written corresponding to one storage instruction. In order to ameliorate the problem that the storage instructions are numerous and occupy computing resources, in the embodiments of the present disclosure, all the stripes may be taken as the target stripes 640, and the storage instructions of the storage regions having the same sorting position in all the target stripes 640 may be merged into one storage instruction 660 to complete data writing. When a small image with small data volume of data to be stored 610 is written, the small image may not be fully written with all the stripes of the candidate storage space. Therefore, in the embodiments of the present disclosure, in order to avoid an increase of the storage instructions, instead of writing the data in the stripes to the storage unit immediately after the small picture is written, a next small picture may be waited to be written to the stripes, and when a count of fully-written stripes in the memory reaches a preset count threshold, all the fully-written stripes may be taken as the target stripes 640. The storage instructions of the storage regions having the same sorting position in all the target stripes 640 may be merged into one single storage instruction 660 to complete the data writing.

In the embodiments of the present disclosure, the different data objects corresponding to the different data to be stored may be distinguished, and different ways for determining the target stripe 640 may be selected according to the different features, so that the system can flexibly cope with different storage scenarios, thereby improving system adaptability and versatility of the system.

In some embodiments, when the data to be stored may correspond to the plurality of data objects, and the processing device 120 may obtain an initial offset 630 of at least one of the target stripes 640. The initial offset 630 refers to a position in the data to be stored corresponding to initial data stored in each stripe of the at least one target stripe 640. For example, if the initial data stored in stripe 2 in the target stripe 640 is 98^th-kb data of the data to be stored, the 98^th-kb may be the initial offset of stripe 2. Understandably, all stripes may be initially free of data offsets.

Furthermore, the processing device 120 may determine a sorting result 650 by sorting, based on the initial offset 630, the at least one target stripe 640. It is understood that when the data to be stored corresponds to the plurality of data objects, for the target stripes 640, an order of occupied target stripes 640 during the writing of the plurality of data objects thereof may be inconsistent with an initial order of the stripes in the candidate storage space since the data volume of the data objects is relatively small. Therefore, it is necessary to sort the target stripes 640 in the order in which the data is written. FIG. 7 is a schematic diagram illustrating sorting target stripes based on corresponding initial offsets according to some embodiments of the present disclosure. For example, as shown in FIG. 7, when data to be stored corresponds to a plurality of data objects, the target stripes in stripe 1 to stripe n in the candidate storage space may be stripe 1, stripe 3, and stripe 6. A size of a storage region "piece" of each stripe may be 32 kb. An initial offset of stripe 1 may be 0 kb. An initial offset of stripe 3 may be 96+1 kb. An initial offset of stripe 6 may be 2×96+1 kb. The processing device 120 may determine that the sorting result is stripe 1 -stripe 3 -stripe 6 by sorting the target stripes based on the initial offsets of stripe 1, stripe 3, and stripe 6.

Furthermore, the processing device 120 may generate the storage instructions 660 corresponding to the data to be stored based on the sorting result 650. For example, the processing device 120 may correspond, based on the sorting result 650, storage regions having a same sorting position in all target stripes 640 to one same storage instruction 660. For example, as shown in FIG. 7, the storage regions “pieces 1” (i.e., storage unit 1) of the target stripes 1, 3, and 6 may correspond to the same storage instruction. The storage regions “pieces 2” (i.e., storage unit 2) of the target stripes 1, 3, and 6 may correspond to the same storage instruction. The storage regions “pieces 3” (i.e., storage unit 3) of the target stripes 1, 3, and 6 may correspond to the same storage instruction. The storage regions “pieces 4” (i.e., storage unit 4) of the target stripes 1, 3, and 6 may correspond to the same storage instruction.

In the embodiments of the present disclosure, when the data to be stored corresponds to the plurality of data objects, the initial offset of the at least one target stripe may be obtained, the sorting result may be determined by sorting, based on the initial offset, the at least one target stripe; and the storage instructions corresponding to the data to be stored may be generated based on the sorting result, which ensures that the data in the target stripe is continuous, thereby improving the accuracy and effectiveness of the determined storage instructions.

In some embodiments, when the data to be stored corresponds to the same data object, storage addresses of a plurality of storage regions in each storage unit in the at least one target stripe may be consecutive. That is, the storage addresses of the storage regions having the same sorting position in the at least one target stripe 640 may be consecutive. The storage address refers to a memory address of the storage region in the system. Understandably, when the data to be stored corresponds to the same data object, since the data volume of data to be stored 610 is relatively large, all or most of the stripes in the candidate storage space may be occupied when written each time. Therefore, the order of the occupied target stripes 640 may be consistent with the initial order of the stripes in the candidate storage space. Accordingly, the storage addresses of the storage regions having the same sorting position in the target stripes 640 may be consecutive. FIG. 8 is a schematic diagram illustrating consecutive storage addresses of a plurality of storage regions according to some embodiments of the present disclosure. As shown in FIG. 8, when the data to be stored corresponds to the same data object, stripe 1 to stripe n in the candidate storage space may be the target stripes. The storage addresses of the storage regions “pieces 1” of stripe 1 to stripe n may be consecutive. The storage addresses of the storage region “piece 1” of stripe n and the storage region “piece 2” of stripe 1 may be consecutive. The storage addresses of the storage regions “pieces 2” of stripe 1 to stripe n may be consecutive. The storage addresses of the storage regions “piece 2” of stripe n and the storage region “piece 3” of stripe 1 may be consecutive. The storage addresses of the storage regions “pieces 3” of stripe 1 to stripe n may be consecutive. The storage addresses of the storage region “piece 3” of stripe n and the storage region “piece 4’ of stripe 1 may be consecutive. The storage addresses of the storage regions “pieces 4” of stripe 1 to stripe n may be consecutive. It should be noted that the storage addresses of storage regions “piece 1 to piece n” in the same stripe are not consecutive.

Furthermore, the processing device 120 may generate the storage instructions 660 corresponding to the data to be stored. For example, the processing device 120 may correspond the storage regions having the same sorting position in all the target stripes 640 to the same storage instruction 660. As shown in FIG. 8, the storage regions “pieces 1” (i.e., storage unit 1) of the target stripes 1 to n may correspond to the same storage instruction. The storage regions “pieces 2” (i.e., storage unit 2) of the target stripes 1 to n may correspond to the same storage instruction. The storage regions “pieces 3” (i.e., storage unit 3) of the target stripes 1 to n may correspond to the same storage instruction. The storage regions “piece 4” (i.e., storage unit 4) of the target stripes 1 to n may correspond to the same storage instruction.

In some embodiments, the processing device 120 may determine an object to which the target stripe 640 belongs based on the storage address, and determine the storage instruction 660 corresponding to the data to be stored. The object refers to a unit of data divided in a file.

In some embodiments, when the data to be stored corresponds to the same data object, the data to be stored is not finished being stored, and all the stripes in the candidate storage space are fully written, the target stripe 640 may be all the fully-written stripes, and the storage addresses of all the target stripes 640 may be consecutive. In this case, the processing device 120 may designate all the target stripes 640 as one object. FIG. 9A is a schematic diagram illustrating generating storage instructions according to some embodiments of the present disclosure. FIG. 9B is a schematic diagram illustrating generating storage instructions according to some embodiments of the present disclosure. As shown in FIG. 9A, it is assumed that the data to be stored is finished being stored, the data to be stored is not finished being stored, and stripe 1 to stripe n are fully written, stripe 1 to stripe n may be target stripes, and stripe 1 to stripe n may correspond to an object.

Furthermore, the processing device 120 may merge the target stripes 640 belonging to the same object and having consecutive addresses and merge the storage instructions corresponding to the storage units in the target stripes 640 into one target storage instruction. For example, as shown in FIG. 9A, the processing device 120 may merge the storage instructions of the storage units 1-4 of stripe 1 to stripe n into one target storage instruction. Understandably, before the merging, one storage instruction may need to be executed for each storage unit, i.e., a total of four storage instructions may need to be executed. After the merging, only a single storage instruction may need to be executed for the storage units 1-4, which further reduces the generation of storage instructions and further improves data writing efficiency and storage performance.

In some embodiments, when the data to be stored corresponds to the same data object, the data to be stored is finished being stored, and there are fully-written or non-fully-written stripes in the candidate storage space, the target stripes 640 may be stripes (including both the fully-written and non-fully-written stripes) in which the data to be stored is stored, and the storage addresses of all the target stripes 640 may be consecutive. In this case, the processing device 120 may make each storage unit in the target stripes 640 correspond to a storage instruction. As shown in FIG. 9B, assuming that the data to be stored is finished being stored, stripe 1 to stripe m are fully written, and stripe m to stripe n are not fully written, stripe 1 to stripe m may be the target stripes, and storage addresses of storage regions, having the same sorting position, of stripe 1 to stripe m may be consecutive. The processing device 120 may make storage unit 1 correspond to a storage instruction, storage unit 2 correspond to a storage instruction, storage unit 3 correspond to a storage instruction, and storage unit 4 correspond to a storage instruction.

FIG. 10 is a flowchart illustrating an exemplary process for writing data to be stored according to some embodiments of the present disclosure. In some embodiments, at least part of the process 1000 may be performed to achieve at least part of operation 340 as described in connection with FIG. 3. For example, the processing device 120 or the storage module 240 may write the data to be stored to a storage unit corresponding to the data to be stored by performing at least part of the process 1000.

In some embodiments, for each storage instruction of storage instructions, the processing device 120 may write the data to be stored to the storage unit corresponding to the storage instruction by performing operations 1010-1020.

In 1010, the processing device 120 may serialize, based on a preset algorithm, the data to be stored corresponding to the storage instruction.

The preset algorithm may be set based on experience or demand. For example, the preset algorithm may include an object serialization algorithm, a JSON serialization algorithm, an XML serialization algorithm, etc.

Serialization refers to a process of converting the data to be stored into a form that can be stored or transmitted. Understandably, after the data to be stored is written to a temporary or persistent storage region during the serialization process, the data to be stored may be recreated by being read or deserialized from the storage region.

As described in FIG. 6, when the data to be stored corresponds to the same data object, since the data volume of the data to be stored is relatively large, all or most of the stripes in the candidate storage space may be occupied when written each time. Therefore, the order of the occupied target stripes may be consistent with the initial order of the stripes in the candidate storage space. Accordingly, the storage addresses of the storage regions having the same sorting position in the target stripes may be consecutive. As a result, in some embodiments, when the data to be stored corresponds to the same data object, the processing device 120 may obtain an initial address of the consecutive storage addresses of the plurality of storage regions corresponding to the storage instruction and obtain an address length of data to be stored corresponding to the storage instruction. In this case, the initial address refers to a storage address of a first storage region of a first target stripe of the at least one target stripe in the candidate storage space. The address length refers to a data volume that needs to be executed when the storage instruction is executed, i.e., the data volume of the data to be stored that needs to be stored in the plurality of storage regions corresponding to the storage instruction.

Furthermore, the processing device 120 may serialize the data to be stored corresponding to the storage instruction based on the initial address and the address length using the preset algorithm.

As described in FIG. 6, when the data to be stored corresponds to the plurality of data objects, for the target stripe, since the data volume of each data object is relatively small, during the process of writing the plurality of data objects, the order of the occupied target stripes may not be consistent with the initial order of the stripes in the candidate storage space. Therefore, the storage addresses of the storage regions in the target stripes in the candidate storage space may not necessarily be consecutive. As a result, in some embodiments, when the data to be stored corresponds to the plurality of data objects, for each storage region of the plurality of storage regions corresponding to the storage instruction, the processing device 120 may obtain an initial storage address, in the data to be stored, of data stored in the storage region and obtain a length of data stored in the storage region. In this case, the initial storage address refers to a position in the data to be stored of data first stored in the storage region. The length of data refers to a length of data stored in the storage region. In some embodiments, the processing device 120 may obtain the initial storage address and the length of data through a struct I/O vector (iovec) . The struct iovec may be configured to describe a data buffer and may be usually used with a readv system and a writev system to read or write a plurality of buffers in a single system call.

Furthermore, the processing device 120 may serialize the data to be stored corresponding to the storage instruction based on initial storage addresses and lengths of data corresponding to the plurality of storage regions using the preset algorithm.

In some embodiments, the processing device 120 may store and merge, through the struct iovec, the initial storage address and the address length of each data region corresponding to the storage instruction. Accordingly, the processing device 120 may traverse the struct iovec to serialize the data to be stored based on the initial storage address and the address length of each data region.

In the embodiments of the present disclosure, the data to be stored corresponding to the storage instruction may be serialized according to the situation according to the different data objects corresponding to the data to be stored, which improves the accuracy and reasonableness of the serialization of the data to be stored.

In 1020, the processing device 120 may write, based on the storage instruction, the serialized data to be stored to a storage unit corresponding to the storage instruction.

In some embodiments, the storage regions of the plurality of stripes may be dispersed on one or more storage nodes. It should be noted that storage regions in a same storage unit are located in a same storage node. The processing device 120 may send, through a network protocol, the serialized data to be stored to the corresponding storage node based on the storage instruction. The storage node may write the serialized data to a disk. In some embodiments, if there are a storage node and a plurality of disks corresponding to the storage node, the data to be stored corresponding to different storage units may be written to different disks of the current storage node. If there are a plurality of nodes, the data to be stored corresponding to different storage units may be written to different nodes. For each node, according to information such as loads or spaces of disks corresponding to the node, or whether or not the disks are used, a disk to which the data to be stored is written may be determined.

In the embodiment of the present disclosure, the data to be stored corresponding to each storage instruction may be serialized, and the serialized data to be stored may be written to the storage unit corresponding to the storage instruction, where the data to be stored may be accurately and efficiently written. In addition, the storage regions in the same storage unit may be located in the same storage node, so that the data to be stored corresponding to the same storage instruction may be written to the same memory space without other memory overhead, thereby ensuring data consecutiveness, integrity, and consistency.

In some embodiments, the processing device 120 may form a queue of stripes by initializing, based on the candidate storage space, the plurality of stripes, so that each storage region of each stripe may point to a space in the memory space of at least one storage device (e.g., the storage device 110) , and the data to be stored may be written to the stripes (i.e., the data to be stored may be written to the memory space) .

In some embodiments, for N data blocks obtained by dividing the data to be stored, the processing device 120 may write the N data blocks in the stripe simultaneously. For example, for the 3 data blocks obtained by dividing the data to be stored, the processing device 120 may simultaneously write the 3 data blocks to piece 1 -piece 3 of the stripe 1, and when the stripe 1 is fully written, the processing device 120 may write the 3 data blocks to piece 1-piece 3 of the stripe 2, etc. until the data to be stored is finished being stored.

In some embodiments, the processing device 120 may determine whether there is a stripe that meets a first preset condition in the queue of stripes in use. The queue of stripes in use refers to a queue consisting of stripes in which data is being stored. In response to a fact that there is the stripe that meets the first preset condition, the processing device 120 may write the data to be stored to the stripe. In response to a fact that there is no stripe that meets the first preset condition, the processing device 120 may call a stripe that meets a second preset condition from a queue of idle stripes to write the data to be stored to the stripe. The first preset condition and the second preset condition may be set based on experience or demand. For example, the first preset condition may be that a stripe to which the data is written is not fully written. As another example, the second preset condition may be that a range of address length of the stripe covers the initial offset of currently written data.

In response to the fact that there is no stripe that meets the second preset condition, the processing device 120 may select an idle stripe from the queue of idle stripes, determine the initial offset of the currently written data as the initial offset of the idle stripe, and update an end offset (i.e., a position of end data of the stripe in the data to be stored) of the idle stripe according to the address length. The queue of idle stripes refers to a queue consisting of idle stripes with no data stored. Understandably, after the initialization of the stripes is complete, all the stripes may be located in the queue of idle stripes.

Exemplarily, a maximum storage capacity of a stripe may be 96 kb. If a storage order is stripe 1 -stripe 3, the initial offset (i.e., a position of the initial data in the data to be stored) of stripe 1 may be a 0^th kb, the end offset (i.e., a position of the end data in the file) of stripe 1 may be the 96^th kb, and the initial offset of stripe 3 may be a (96 kb + 1 kb) ^a (i.e., the order of the end data of stripe 1 and the initial data of stripe 3 in the data to be stored may be consecutive) . If the processing device 120 needs to write 1 M data in the order of stripe 1-stripe n, and stripe 1 is fully written and half of stripe 2 is fully written, the remaining data may need to continue to be written to stripe 2 according to the requirement of data consecutiveness. At this time, the system may find stripe 2 from the queue of stripes in use according to the initial offset of the stripe. Since the end data of stripe 1 and the initial data of stripe 2 are in continuous positions in the data to be stored, stripe 2 may be determined by determining the initial offset of 96 kb+1 kb to realize the continuation of data writing.

One or more embodiments of the present disclosure further provide a computer-readable storage medium that stores computer instructions. When reading the computer instructions in the storage medium, a computer may perform the method as described in the embodiments of the present disclosure.

The operations of the illustrated processes 300, 500, 600 and 1000 presented above are intended to be illustrative. In some embodiments, a process may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. Additionally, the order in which the operations of a process described above is not intended to be limiting.

Having thus described the basic concepts, it may be rather apparent to those skilled in the art after reading this detailed disclosure that the foregoing detailed disclosure is intended to be presented by way of example only and is not limiting. Various alterations, improvements, and modifications may occur and are intended to those skilled in the art, though not expressly stated herein. These alterations, improvements, and modifications are intended to be suggested by this disclosure and are within the spirit and scope of the exemplary embodiments of this disclosure.

Moreover, certain terminology has been used to describe embodiments of the present disclosure. For example, the terms “one embodiment, ” “an embodiment, ” and/or “some embodiments” mean that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Therefore, it is emphasized and should be appreciated that two or more references to “an embodiment” or “one embodiment” or “an alternative embodiment” in various portions of this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined as suitable in one or more embodiments of the present disclosure.

Further, it will be appreciated by one skilled in the art, aspects of the present disclosure may be illustrated and described herein in any of a number of patentable classes or context including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Accordingly, aspects of the present disclosure may be implemented entirely hardware, entirely software (including firmware, resident software, micro-code, etc. ) or combining software and hardware implementation that may all generally be referred to herein as a “unit, ” “module, ” or “system. ” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer readable program code embodied thereon.

A non-transitory computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including electromagnetic, optical, or the like, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that may communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable signal medium may be transmitted using any appropriate medium, including wireless, wireline, optical fiber cable, RF, or the like, or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB. NET, Python or the like, conventional procedural programming languages, such as the "C" programming language, Visual Basic, Fortran, Perl, COBOL, PHP, ABAP, dynamic programming languages such as Python, Ruby, and Groovy, or other programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN) , or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider) or in a cloud computing environment or offered as a service such as a Software as a Service (SaaS) .

Furthermore, the recited order of processing elements or sequences, or the use of numbers, letters, or other designations, therefore, is not intended to limit the claimed processes and methods to any order except as may be specified in the claims. Although the above disclosure discusses through various examples what is currently considered to be a variety of useful embodiments of the disclosure, it is to be understood that such detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover modifications and equivalent arrangements that are within the spirit and scope of the disclosed embodiments. For example, although the implementation of various components described above may be embodied in a hardware device, it may also be implemented as a software-only solution, e.g., an installation on an existing server or mobile device.

Similarly, it should be appreciated that in the foregoing description of embodiments of the present disclosure, various features are sometimes grouped together in a single embodiment, figure, or description thereof to streamline the disclosure aiding in the understanding of one or more of the various inventive embodiments. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed object matter requires more features than are expressly recited in each claim. Rather, inventive embodiments lie in less than all features of a single foregoing disclosed embodiment.

In some embodiments, the numbers expressing quantities, properties, and so forth, used to describe and claim certain embodiments of the application are to be understood as being modified in some instances by the term “about, ” “approximate, ” or “substantially. ” For example, “about, ” “approximate” or “substantially” may indicate ±20%variation of the value it describes, unless otherwise stated. Accordingly, in some embodiments, the numerical parameters set forth in the written description and attached claims are approximations that may vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of some embodiments of the application are approximations, the numerical values set forth in the specific examples are reported as precisely as practicable.

Each of the patents, patent applications, publications of patent applications, and other material, such as articles, books, specifications, publications, documents, things, and/or the like, referenced herein is hereby incorporated herein by this reference in its entirety for all purposes, excepting any prosecution file history associated with same, any of same that is inconsistent with or in conflict with the present document, or any of same that may have a limiting affect as to the broadest scope of the claims now or later associated with the present document. By way of example, should there be any inconsistency or conflict between the description, definition, and/or the use of a term associated with any of the incorporated material and that associated with the present document, the description, definition, and/or the use of the term in the present document shall prevail.

In closing, it is to be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of the application. Other modifications that may be employed may be within the scope of the application. Thus, by way of example, but not of limitation, alternative configurations of the embodiments of the application may be utilized in accordance with the teachings herein. Accordingly, embodiments of the present application are not limited to that precisely as shown and described.

Claims

A method for storing data, performed by a computing device having at least one processor and at least one storage device, the method comprising:

obtaining a candidate storage space corresponding to data to be stored from the at least one storage device, the candidate storage space being divided into a plurality of stripes, each stripe of the plurality of stripes including a preset count of storage regions, the preset count of storage regions being sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes being a storage unit;

determining at least one target stripe corresponding to the data to be stored from the plurality of stripes;

generating storage instructions corresponding to the data to be stored, each storage unit in the at least one target stripe corresponding to one storage instruction; and

writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions.
The method of claim 1, further comprising:

determining whether a data volume of the data to be stored exceeds a preset data volume threshold;

in response to a determination that the data volume does not exceed the preset data volume threshold, determining that the data to be stored corresponds to a plurality of data objects; and

in response to a determination that the data volume exceeds the preset data volume threshold, determining that the data to be stored corresponds to a same data object.
The method of claim 2, wherein the data to be stored corresponds to the plurality of data objects, and the generating storage instructions corresponding to the data to be stored includes:

obtaining an initial offset of the at least one target stripe, the initial offset being a position, in the data to be stored, corresponding to initial data stored in each target stripe of the at least one target stripe;

determining a sorting result by sorting, based on the initial offset, the at least one target stripe; and

generating the storage instructions corresponding to the data to be stored based on the sorting result.
The method of claim 2, wherein

the data to be stored corresponds to the same data object, and

storage addresses of a plurality of storage regions in each storage unit in the at least one target stripe are consecutive.
The method of claim 3 or 4, wherein the writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions includes:

for each storage instruction of the storage instructions,

serializing, based on a preset algorithm, the data to be stored corresponding to the storage instruction; and

writing, based on the storage instruction, the serialized data to be stored to a storage unit corresponding to the storage instruction.
The method of claim 5, wherein the data to be stored corresponds to the plurality of data objects, and the serializing the data to be stored corresponding to the storage instruction includes:

for each storage region of a plurality of storage regions corresponding to the storage instruction,

obtaining an initial storage address, in the data to be stored, of data stored in the storage region; and

obtaining a length of data stored in the storage region; and

serializing the data to be stored corresponding to the storage instruction based on initial storage addresses and lengths of data corresponding to the plurality of storage regions using the preset algorithm.
The method of claim 6, wherein the initial storage address and the length of data are obtained through a struct I/O vector (iovec) .
The method of claim 5, wherein the data to be stored corresponds to the same data object, and the serializing the data to be stored corresponding to the storage instruction includes:

obtaining an initial address of consecutive storage addresses of a plurality of storage regions corresponding to the storage instruction;

obtaining an address length of the data to be stored corresponding to the storage instruction; and

serializing the data to be stored corresponding to the storage instruction based on the initial address and the address length using the preset algorithm.
The method of any one of claims 1-8, wherein storage regions of the plurality of stripes are dispersed on one or more storage nodes, and storage regions in a same storage unit are located in a same storage node.
The method of any one of claims 1-9, further comprising dividing the candidate storage space into the plurality of stripes using a target division manner, wherein the target division manner is determined by:

determining at least one candidate division manner of the candidate storage space based on the data to be stored;

determining, through a prediction model, a cache success rate and a cache speed corresponding to each candidate division manner of the at least one candidate division manner based on a size of the data to be stored, a storage capacity of the at least one storage device, a storage capacity of the candidate storage space, and the at least one candidate division manner; and

determining, based on the at least one candidate division, the cache success rate, and the cache speed, the target division manner.
A system for storing data, comprising:

an obtaining module configured to obtain a candidate storage space corresponding to data to be stored from at least one storage device, the candidate storage space being divided into a plurality of stripes, each stripe of the plurality of stripes including a preset count of storage regions, the preset count of storage regions being sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes being a storage unit;

a determination module configured to determine at least one target stripe corresponding to the data to be stored from the plurality of stripes;

a generation module configured to storage instructions corresponding to the data to be stored, each storage unit in the at least one target stripe corresponding to one storage instruction; and

a storage module configured to write, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions.
The system of claim 11, wherein the obtaining module is further configured to:

determine whether a data volume of the data to be stored exceeds a preset data volume threshold;

in response to a determination that the data volume does not exceed the preset data volume threshold, determine that the data to be stored corresponds to a plurality of data objects; and

in response to a determination that the data volume exceeds the preset data volume threshold, determine that the data to be stored corresponds to a same data object.
The system of claim 12, wherein the data to be stored corresponds to the plurality of data objects, and to generate the storage instructions corresponding to the data to be stored, the generation module is further configured to:

obtain an initial offset of the at least one target stripe, the initial offset being a position, in the data to be stored, corresponding to initial data stored in each target stripe of the at least one target stripe;

determine a sorting result by sorting, based on the initial offset, the at least one target stripe; and

generate the storage instructions corresponding to the data to be stored based on the sorting result.
The system of claim 12, wherein

the data to be stored corresponds to the same data object, and

storage addresses of a plurality of storage regions in each storage unit in the at least one target stripe are consecutive.
The system of claim 13 or 14, wherein to write, based on the storage instructions , the data to be stored to storage units corresponding to the storage instructions, the storage module is further configured to:

for each storage instruction of the storage instructions,

serialize, based on a preset algorithm, data to be stored corresponding to the storage instruction; and

write, based on the storage instruction, the serialized data to be stored to a storage unit corresponding to the storage instruction.
The system of claim 15, wherein the data to be stored corresponds to the plurality of data objects, and to serialize the data to be stored corresponding to the storage instruction, the storage module is further configured to:

for each storage region of a plurality of storage regions corresponding to the storage instruction,

obtain an initial storage address, in the data to be stored, of data stored in the storage region; and

obtain a length of data stored in the storage region; and

serialize the data to be stored corresponding to the storage instructions based on initial storage addresses and lengths of data corresponding to the plurality of storage regions using the preset algorithm.
The system of claim 16, wherein the initial storage address and the length of data are obtained through a struct I/O vector (iovec) .
The system of claim 15, wherein the data to be stored corresponds to the same data object, and to serialize the data to be stored corresponding to the storage instruction, the storage module is further configured to:

obtain an initial address of consecutive storage addresses of a plurality of storage regions corresponding to the storage instruction;

obtain an address length of the data to be stored corresponding to the storage instruction; and

serialize the data to be stored corresponding to the storage instruction based on the initial address and the address length using the preset algorithm.
The system of any one of claims 11-18, wherein storage regions of the plurality of stripes are dispersed on one or more storage nodes, and the storage regions in a same storage unit are located in a same storage node.
The system of any one of claims 11-19, further comprising a division module configured to divide the candidate storage space into the plurality of stripes using a target division manner, wherein to determine the target division manner, the division module is further configured to:

determine at least one candidate division manner of the candidate storage space based on the data to be stored;

determine, through a prediction model, a cache success rate and a cache speed corresponding to each candidate division manner of the at least one candidate division manner based on a size of the data to be stored, a storage capacity of the at least one storage device, a storage capacity of the candidate storage space, and the at least one candidate division manner; and

determine, based on the at least one candidate division manner, the cache success rate, and the cache speed, the target division manner.
A system, comprising at least one processor and at least one storage device storing computer instructions, wherein when executed by the at least one processor, the computer instructions cause the at least one processor to perform operations including:

obtaining a candidate storage space corresponding to data to be stored from the at least one storage device, the candidate storage space being divided into a plurality of stripes, each stripe of the plurality of stripes including a preset count of storage regions, the preset count of storage regions being sequentially arranged in the stripe and have corresponding sorting positions, and storage regions having a same sorting position in the plurality of stripes being a storage unit;

determining at least one target stripe corresponding to the data to be stored from the plurality of stripes;

generating storage instructions corresponding to the data to be stored, each storage unit in the at least one target stripe corresponding to one storage instruction; and

writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions.
The system of claim 21, wherein the operations further include:

determining whether a data volume of the data to be stored exceeds a preset data volume threshold;

in response to a determination that the data volume does not exceed the preset data volume threshold, determining that the data to be stored corresponds to a plurality of data objects; and

in response to a determination that the data volume exceeds the preset data volume threshold, determining that the data to be stored corresponds to a same data object.
The system of claim 22, wherein the data to be stored corresponds to the plurality of data objects, and the generating storage instructions corresponding to the data to be stored includes:

obtaining an initial offset of the at least one target stripe, the initial offset being a position, in the data to be stored, corresponding to initial data stored in each target stripe of the at least one target stripe;

determining a sorting result by sorting, based on the initial offset, the at least one target stripe; and

generating the storage instructions corresponding to the data to be stored based on the sorting result.
The system of claim 22, wherein

the data to be stored corresponds to the same data object, and

storage addresses of a plurality of storage regions in each storage unit in the at least one target stripe are consecutive.
The system of claim 23 or 24, wherein the writing, based on the storage instructions, the data to be stored to storage units corresponding to the storage instructions includes:

for each storage instruction of the storage instructions,

serializing, based on a preset algorithm, the data to be stored corresponding to the storage instruction; and

writing, based on the storage instruction, the serialized data to be stored to a storage unit corresponding to the storage instruction.
The system of claim 25, wherein the data to be stored corresponds to the plurality of data objects, and the serializing the data to be stored corresponding to the storage instruction includes:

for each storage region of a plurality of storage regions corresponding to the storage instructions,

obtaining an initial storage address, in the data to be stored, of data stored in the storage region; and

obtaining a length of data stored in the storage region; and

serializing the data to be stored corresponding to the storage instructions based on initial storage addresses and lengths of data corresponding to the plurality of storage regions using the preset algorithm.
The system of claim 26, wherein the initial storage address and the length of data are obtained through a struct I/O vector (iovec) .
The system of claim 25, wherein the data to be stored corresponds to the same data object, and the serializing the data to be stored corresponding to the storage instruction includes:

obtaining an initial address of consecutive storage addresses of a plurality of storage regions corresponding to the storage instruction;

obtaining an address length of the data to be stored corresponding to the storage instruction; and

serializing the data to be stored corresponding to the storage instruction based on the initial address and the address length using the preset algorithm.
The system of any one of claims 21-28, wherein storage regions of the plurality of stripes are dispersed on one or more storage nodes, and storage regions in a same storage unit are located in a same storage node.
The system of any one of claims 21-29, the operations further comprising dividing the candidate storage space into the plurality of stripes using a target division manner, wherein the target division manner is determined by:

determining at least one candidate division manner of the candidate storage space based on the data to be stored;

determining, through a prediction model, a cache success rate and a cache speed corresponding to each candidate division manner of the at least one candidate division manner based on a size of the data to be stored, a storage capacity of the at least one storage device, a storage capacity of the candidate storage space, and the at least one candidate division manner; and

determining, based on the at least one candidate division manner, the cache success rate, and the cache speed, the target division manner.
A computer-readable storage medium storing computer instructions, wherein when reading the computer instructions in the storage medium, a computer performs the method of any one of claims 1-10.