CN114546295B - Intelligent writing distribution method and device based on ZNS solid state disk - Google Patents
Intelligent writing distribution method and device based on ZNS solid state disk Download PDFInfo
- Publication number
- CN114546295B CN114546295B CN202210437084.XA CN202210437084A CN114546295B CN 114546295 B CN114546295 B CN 114546295B CN 202210437084 A CN202210437084 A CN 202210437084A CN 114546295 B CN114546295 B CN 114546295B
- Authority
- CN
- China
- Prior art keywords
- data
- writing
- written
- target
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an intelligent writing distribution method and device based on a ZNS solid state disk, wherein the method comprises the following steps: receiving data to be written which is issued by a storage application; distributing a target writing object for the data to be written; extracting multi-dimensional data attribute characteristics of the target writing object, and mapping the multi-dimensional data attribute characteristics into a writing partition type; and distributing a writing area for the data to be written according to the writing partition type. According to the intelligent write allocation method and device provided by the embodiment of the invention, the written data is classified and stored through object attribute feature extraction oriented to storage application, different clustering type objects can be kept to be separately stored, sequential and random objects are separately stored, and cold and hot data objects are separately stored, so that the performance loss caused by data mixed storage is reduced, the classification failure problem caused by loss of real data attributes based on an LBA data classification algorithm is avoided, meanwhile, the algorithm universality is kept, and meanwhile, the performance advantages of a flash memory medium are fully exerted.
Description
Technical Field
The invention relates to the field of ZNS solid state disks, in particular to an intelligent writing distribution method and device based on a ZNS solid state disk.
Background
In the prior art, a logical address is converted into an LBA address by storage application, the LBA logical address is converted into an NAND actual physical address by an FTL (fiber to the Home) inside the SSD, when data is written, data of different modes and heat are stored in a mixed mode, the performance advantage of a flash memory medium cannot be fully exerted, because the NAND needs ex-situ update, invalid data caused by logical data overwriting needs garbage recovery, and the garbage recovery performance can be seriously influenced when the data is mixed, so that the performance of the whole storage system can be optimized by introducing a write classification technology into the solid-state storage system.
The current writing classification technology is mainly divided into two types, one type is to classify and store data aiming at a certain specific storage application, and the classification technology has low universality and cannot be self-adapted to different storage applications; the other method is that data is classified and stored at an LBA level inside the SSD, because the LBA address received by the SSD loses application layer attribute information, the LBA after storage and application logic conversion is different from the original data attribute, two files with fixed sizes are written in the same sequence, two data streams are stored in a cross-cycle covering manner, the LBA written in the SSD is completely sequential and belongs to different files when the disk is empty, when the storage space is full, the old file needs to be deleted, new file writing is generated, the LBA occupied by the deleted file is not continuous, the subsequent LBA rule of the new file generated after recovery is disturbed, and therefore the data classification technology established on the LBA cannot effectively extract the real attribute of the data.
The ZNS solid state disk is used as a novel solid state disk and has the characteristic of an open flash memory operation interface, so that an intelligent writing distribution algorithm based on the ZNS solid state disk needs to be designed urgently, and classified storage is carried out according to the real attribute of data.
Disclosure of Invention
In view of this, an object of an embodiment of the present invention is to provide an intelligent write allocation method and apparatus based on a ZNS solid state disk, an electronic device, and a readable storage medium, which specifically include:
in a first aspect, an embodiment of the present invention provides an intelligent write allocation method based on a ZNS solid state disk, where the method includes:
receiving data to be written issued by a storage application, wherein the data to be written is new written data;
distributing a target writing object for the data to be written;
extracting multi-dimensional data attribute characteristics of the target writing object, and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the data attribute features include an aggregate feature to characterize the target write object access type, a sequential feature to characterize the target write object attribute ordering, and a frequency feature to characterize the target write object access frequency;
and distributing a writing area for the data to be written according to the writing partition type.
Optionally, the aggregative feature of the multi-dimensional data attribute features is extracted according to the following manner:
based on the data association degree of the sample write-in data, dividing the sample write-in data into multiple access type groups by adopting an unsupervised learning algorithm;
determining an access type of the target write object among the plurality of access types as the aggregative feature in the multi-dimensional data attribute features.
Optionally, the unsupervised learning algorithm is a gradient boosting decision tree, K-means, hierarchical cluster analysis, or DBSCAN.
Optionally, the sequential feature in the multidimensional data attribute feature is extracted according to the following manner:
if the storage index mode of the target writing object is key value oriented storage, performing dictionary sorting based on the name character string of the target writing object to obtain a sequence of the target writing object as the sequence feature in the multi-dimensional data attribute feature.
Optionally, the sequential feature in the multi-dimensional data attribute feature is extracted according to the following manner:
and if the storage index mode of the target writing object is based on address or logic number storage, binary value sequencing is carried out based on the address or logic number corresponding to the target writing object to obtain a sequence of the target writing object as the sequence feature in the multi-dimensional data attribute feature.
Optionally, the frequency feature in the multi-dimensional data attribute features is extracted according to the following manner:
setting a corresponding binary counter for the target writing object;
when data is written once to the target write object, the count of the binary counter is increased by 1;
shifting the count in the binary counter by one bit to the right at fixed time intervals;
determining the frequency feature in the multi-dimensional data attribute features from the counts in the binary counter.
Optionally, after allocating a write area for the data to be written, the method further includes:
marking the writing area as a writing completion writing area;
distributing partitions with the same multidimensional data attribute characteristics as the writing areas from an idle partition pool of a ZNS solid state disk storage space;
and when the number of the available partitions in the free partition pool of the ZNS solid state disk storage space is less than a preset value, starting a garbage recovery mechanism to generate a new available partition.
In a second aspect, an embodiment of the present invention provides an intelligent writing distribution apparatus based on a ZNS solid state disk, where the apparatus includes:
the write-in data receiving module is used for receiving data to be written, which is issued by the storage application, wherein the data to be written is new write-in data;
the object distribution module is used for distributing a target writing object for the data to be written;
the characteristic extraction module is used for extracting the multi-dimensional data attribute characteristics of the target writing object and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the data attribute features include an aggregate feature to characterize the target write object access type, a sequential feature to characterize the target write object attribute ordering, and a frequency feature to characterize the target write object access frequency;
and the partition allocation module is used for allocating a write-in area for the data to be written according to the write-in partition type.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
one or more processors;
a memory for storing one or more programs;
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of the first aspect.
In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium having stored thereon executable instructions, which when executed by a processor, cause the processor to perform the method according to the first aspect.
According to the intelligent write allocation method, device, electronic equipment and readable storage medium based on the ZNS solid state disk, the characteristics of open flash operation of the ZNS solid state disk are utilized, the object attribute characteristics oriented to storage application are extracted, the written data are stored in the ZNS solid state disk in a classified mode, different clustering type objects can be kept to be stored separately, sequential and random objects are stored separately, cold and hot data objects are stored separately, performance loss caused by data mixed storage is reduced, the problem of classification failure caused by loss of real data attributes based on an LBA data classification algorithm is solved, meanwhile algorithm universality is kept, and meanwhile performance advantages of the flash storage medium are fully played; and meanwhile, a dynamic allocation mode is adopted for the writing areas allocated according to the data classification, a free space is dynamically allocated for each type of the mapped writing areas, and objects with different characteristic attributes share the global ZNS solid state disk space.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative work. The foregoing and other objects, features and advantages of the application will be apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the drawings. The drawings are not intended to be to scale as practical, emphasis instead being placed upon illustrating the subject matter of the present application.
FIG. 1 is a schematic flow chart of an intelligent ZNS solid state disk-based write allocation method provided by the embodiment of the invention.
Fig. 2 is a flowchart illustrating a partition management method according to an embodiment of the present invention.
FIG. 3 is a schematic structural diagram of an intelligent ZNS solid state disk-based write allocation device provided by the embodiment of the invention.
Fig. 4 shows a schematic structural diagram of a physical electronic device provided in accordance with an embodiment of the present invention.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "a", "an" and "the", and the like, as used herein, are also intended to include the meaning of "a plurality" and "the" unless the context clearly indicates otherwise. Furthermore, the terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
In the prior art, a logical address is converted into an LBA address by storage application, the LBA logical address is converted into an NAND actual physical address by an FTL (fiber to the Home) inside the SSD, when data is written, data of different modes and heat are stored in a mixed mode, the performance advantage of a flash memory medium cannot be fully exerted, because the NAND needs ex-situ update, invalid data caused by logical data overwriting needs garbage recovery, and the garbage recovery performance can be seriously influenced when the data is mixed, so that the performance of the whole storage system can be optimized by introducing a write classification technology into the solid-state storage system.
The current writing classification technology is mainly divided into two types, one type is to classify and store data aiming at a certain specific storage application, and the classification technology has low universality and cannot be self-adapted to different storage applications; the other method is that data is classified and stored at an LBA level inside the SSD, because the LBA address received by the SSD loses application layer attribute information, the LBA after storage and application logic conversion is different from the original data attribute, two files with fixed sizes are written in the same sequence, two data streams are stored in a cross-cycle covering manner, the LBA written in the SSD is completely sequential and belongs to different files when the disk is empty, when the storage space is full, the old file needs to be deleted, new file writing is generated, the LBA occupied by the deleted file is not continuous, the subsequent LBA rule of the new file generated after recovery is disturbed, and therefore the data classification technology established on the LBA cannot effectively extract the real attribute of the data.
The ZNS solid state disk is a novel solid state disk with the characteristic of an open flash memory operation interface, so that an intelligent writing distribution algorithm based on the ZNS solid state disk needs to be designed urgently, and classified storage is carried out according to the real attribute of data.
In view of this, an object of the embodiments of the present disclosure is to provide an intelligent writing distribution method and apparatus based on a ZNS solid state disk, an electronic device, and a readable storage medium, and the following describes in detail the disclosure of the embodiments of the present disclosure with reference to the accompanying drawings.
FIG. 1 is a schematic flow chart of an intelligent write allocation method based on a ZNS solid state disk provided by an embodiment of the present invention, and the specific contents are as follows:
step S110, receiving data to be written issued by the storage application, where the data to be written is new written data.
The storage application in the embodiment of the invention refers to an external application program which has a data storage requirement and sends a data writing command to the ZNS solid state disk. The ZNS solid state disk can respond to the write command through the front-end interface and identify and receive data to be written, which is issued by the storage application.
The intelligent writing distribution method in the embodiment of the invention supports the data to be written as newly written data and also supports the data to be written as updating the written data. The two different forms of data to be written are both target data implemented by the intelligent write allocation method in the embodiment of the invention.
The method is different from the write allocation method in the prior art, and only aims at a certain specific storage application to store data in a classified manner. The data to be written in the embodiment of the invention can be issued by any storage application, and has better universality.
Step S120, distributing a target writing object for the data to be written.
In a ZNS solid state disk, a target write object matched with the data to be written is generally allocated to the data to be written. The target writing object refers to a file system allocation unit, a database operation unit, a block device operation unit and other various storage system operation units which are built on the ZNS solid state disk storage system. The ZNS solid state disk storage system can allocate different target write objects to data to be written according to the type of the data to be written, namely the target write objects are operation units for performing write operation on the data to be written. At this point, the management algorithm inside the storage application is responsible for indexing the target write object.
In step S110, the embodiment of the present invention supports that the data to be written is newly written data, and also supports that the data to be written is updated to the written data. Specifically, when the system detects that the data to be written is newly written data, a target write object needs to be allocated to the data to be written; when the system detects that the data to be written is the data which is updated, the target writing object distributed in the first writing can be directly used.
In actual processing, in this step, data attribute labeling of the write object may be performed, and attribute field assignment of the write object is performed according to the type of the storage application, and the labeling is also a basis and input for subsequent feature extraction operations. The specifically labeled content may include object type, object recommendation attribute classification, and the like.
Step S130, extracting multi-dimensional data attribute characteristics of the target writing object, and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the data attribute features include an aggregative feature for characterizing the target write object access type, a sequential feature for characterizing the target write object attribute ordering, and a frequency feature for characterizing the target write object access frequency.
In this step, the characteristics of the flash memory operation interface are opened based on the ZNS solid state disk, so that the object designed for the storage application contains the real attributes of the storage application, such as the logical context between data, the relevance in data access, the data type, the real access frequency of logical data, and the like. Therefore, the problem that in the prior art, data are classified and stored in an LBA level inside the SSD is solved, and because the LBA address received by the SSD loses the attribute information of the application layer, the LBA after the logic conversion of the storage application is different from the original data attribute, the classification and storage are inaccurate due to the loss of the real attribute.
Specifically, the real attributes of the storage application in the embodiment of the present invention are mainly embodied as multidimensional data attribute features of the target write object, where the multidimensional data attribute features include an aggregative feature for characterizing an access type of the target write object, a sequential feature for characterizing an attribute ordering of the target write object, and a frequency feature for characterizing an access frequency of the target write object. After the multidimensional data attribute characteristics of the target write object are obtained, the multidimensional data attribute characteristics can be mapped into the write partition type, so that the data to be written is determined to be suitable for being written into which ZONE partition in the ZNS solid state disk.
In the embodiment of the present invention, the aggregative feature used for characterizing the access type of the target write object in the multidimensional data attribute feature is extracted according to the following manner: based on the data association degree of the sample written data, adopting an unsupervised learning algorithm to divide the sample written data into access type groups; determining an access type of the target write object among the plurality of access types as the aggregative feature in the multi-dimensional data attribute features.
Specifically, the aggregation characteristic is one of real attributes of the storage application, and is used for reflecting relevance in data access, strongly correlated data are written into the same ZONE partition, the efficiency of data access can be improved, and the performance advantage of the flash memory medium can be fully exerted. The aggregate feature extraction may employ an unsupervised learning algorithm in machine learning to group target write objects into multiple access type groups. The access type grouping may be aggregated and grouped according to the types of file system allocation, database operation, and the like, or may be aggregated and grouped according to other types of indexes related to the write data type, which is not specifically limited in the embodiment of the present invention. Furthermore, the unsupervised learning algorithm is mainly based on the relevance of data, and can adopt algorithms such as a gradient lifting decision tree, K-means, Hierarchical Cluster Analysis (HCA), DBSCAN and the like to aggregate data into different groups. And dividing the target writing object into any group, and marking a label corresponding to the group as the aggregative characteristic of the target writing object.
In the embodiment of the present invention, the order feature for characterizing the attribute ordering of the target write object in the multidimensional data attribute feature may be extracted according to the following manner: if the storage index mode of the target writing object is key value oriented storage, performing dictionary sorting based on the name character string of the target writing object to obtain a sequence of the target writing object as the sequence feature in the multi-dimensional data attribute feature.
In the embodiment of the present invention, the sequential feature used for characterizing the target write object attribute ordering in the multidimensional data attribute feature may be further extracted according to the following manner: and if the storage index mode of the target writing object is based on address or logic number storage, binary value sequencing is carried out based on the address or logic number corresponding to the target writing object to obtain a sequence of the target writing object as the sequence feature in the multi-dimensional data attribute feature.
In particular, the ordering attribute is one of the real attributes of the storage application for embodying a general order of target write object accesses. The data to be written corresponding to the target write object with the access sequence close to each other is written into the same ZONE partition, so that the efficiency of data access can be improved, the performance advantages of the flash memory medium can be fully exerted, and the performance loss caused by frequent data access across the ZONE partitions is avoided. In addition, the sequentially accessed target write objects and the randomly accessed target write objects also need to be stored separately to further take advantage of the performance advantages of the flash media. The extraction process of the sequence characteristics can be designed into a plug-in mode, supports various different sequence detection methods, and carries out sequencing according to different target write object attributes.
Specifically, for example, for a target write object oriented to a key-value store, sequential detection based on an object name string dictionary sorting method may be performed; for storage objects to be based on address or logical number, sequential detection based on object name binary value ordering is performed. Target write objects with close access sequences can be endowed with the same sequence characteristic value, and target write objects with random access can be solely endowed with sequence characteristic values different from the target write objects with sequential access, so that the target write objects with random access are subsequently allocated with ZONE partitions specially suitable for random write data.
In the embodiment of the present invention, a frequency feature used for characterizing the access frequency of the target write object in the multidimensional data attribute feature may be extracted according to the following manner: setting a corresponding binary counter for the target writing object; when data is written once to the target write object, the count of the binary counter is increased by 1; shifting the count in the binary counter by one bit to the right every fixed time period; determining the frequency feature in the multi-dimensional data attribute features from the counts in the binary counter.
In particular, the ordering feature is one of the real attributes of the storage application for embodying the data heat of the target write object access. The adaptive ZONE partitions are distributed for the data with different heat degrees, so that the efficiency of data access can be improved, and the performance advantage of the flash memory medium can be fully exerted. In the step, the data heat is obtained not only by accumulating the count in a counter manner, but also by shifting the count periodically, so that the real heat of the target writing object can be fed back dynamically in real time by the count value.
According to the method, the cold and hot of the data can be judged according to the condition of writing the counter, and in order to reduce the space complexity of the writing counter, a multi-Hash function mode can be introduced to map a target writing object to the writing counter through the multiple Hash functions.
After the multidimensional data attribute features of the target writing object including the aggregation feature, the sequence feature and the frequency feature are obtained, the multidimensional data attribute features are mapped to the writing partition type. The embodiment of the invention can establish the mapping table of the multi-dimensional data attribute characteristics and the writing partition types in advance, and map the data with different attribute characteristics to different writing partition types by searching the mapping table.
The ZNS solid state disk opens the operation characteristic of the flash memory, and can be divided into a plurality of storage areas such as an SLC area, an MLC area, a TLC area, a QLC area, a random writing area capable of supporting random writing and the like, and combinations of the storage areas of different types according to the characteristic of the flash memory and different manufacturer strategies. Each type of storage area in the ZNS solid state disk comprises a plurality of free ZONE partitions for allocation and management by the method provided by the embodiment of the invention. Different storage areas all require different storage performance due to their physical characteristics, thereby adapting to the needs of different types of data. For example, a random write area that can support random writing is particularly suitable for write data whose sequential characteristics are marked as randomness; for another example, the SLC storage area is suitable for storing important data with high data heat and requiring frequent access because the data reading time is shortest; the QLC storage area has the slowest data access speed and can be used to store write data with low heat. Those skilled in the art can establish a mapping table from the multi-dimensional data attribute characteristics to the write partition types according to the different storage performances exhibited by the above exemplary different storage regions, so as to meet the requirements of separate storage of objects of different cluster types, separate storage of sequential and random objects, separate storage of cold and hot data objects, and the like, and fully exert the performance advantages of the flash memory medium.
Step S140, distributing a writing area for the data to be written according to the writing partition type.
After the mapping from the multi-dimensional data attribute characteristics to the writing partition type is determined, a corresponding space ZONE partition is allocated to the data to be written in the storage area of the partition type, and the writing operation is completed.
According to the ZNS solid state disk-based intelligent write allocation method provided by the embodiment of the invention, the characteristics of open flash operation of the ZNS solid state disk are utilized, the written data is classified and stored into the ZNS solid state disk through the object attribute characteristic extraction oriented to storage application, different clustering type objects can be kept to be stored separately, sequential and random objects are stored separately, and cold and hot data objects are stored separately, so that the performance loss caused by data mixed storage is reduced, the classification failure problem caused by loss of real data attributes based on an LBA data classification algorithm is avoided, the algorithm universality is kept, and the performance advantages of a flash memory medium are fully exerted.
Based on the foregoing embodiments, fig. 2 shows a flowchart of a partition management method provided by an embodiment of the present invention, which specifically includes the following contents.
Step S210, mark the writing area as a writing-completed writing area.
In the embodiment of the invention, based on the characteristic of the ZNS solid state disk open flash memory operation interface, the characteristic of the storage space which is globally shared by all data written into the ZNS solid state disk is realized. The free ZONE partitions in any type of storage area can jointly form a free partition pool in the ZNS solid state disk, and the ZONE partitions in any type of storage area, which are completed to write data, can jointly form a write completion ZONE partition pool in the ZNS solid state disk. Each ZONE partition can be individually marked as idle or write complete.
According to the foregoing embodiment, different types of free ZONE partitions are allocated to data to be written, and after the operation of writing the data is completed, the flag of the ZONE partition may be converted into a ZONE partition that is completed writing.
And S220, distributing the partitions with the same multidimensional data attribute characteristics as the writing areas from the free partition pool of the ZNS solid state disk storage space.
To avoid the inability to meet the lack of any type of idle ZONE partition, embodiments of the present invention require dynamic allocation of idle ZONE partitions. Since the ZNS solid state disk manages the storage space by means of storage region reservation or garbage collection, a certain idle ZONE partition needs to be allocated from the globally shared idle partition pool as an idle ZONE partition of a specified type, so as to perform supplementary allocation on the idle ZONE partition of the specified type in real time. The idle ZONE partitions which are allocated in a supplementing mode are generally the same as the ZONE partitions marked as write completion in type, and the dynamic balance of the whole intelligent write allocation system is kept, so that different data storage proportions are adjusted in a self-adaptive mode, reserved space is shared, and storage space is utilized to the maximum extent.
And step S230, when the number of the available partitions in the free partition pool of the ZNS solid state disk storage space is less than a preset value, starting a garbage collection mechanism to generate a new available partition.
When the number of the available partitions is smaller than a preset value due to continuous allocation of the idle partitions in the globally shared idle partition pool, a garbage recovery mechanism needs to be started to generate a new available partition, and the new available partition is added into the globally shared idle partition pool, so that the utilization rate of the storage space of the whole ZNS solid state disk is further maintained.
According to the partition management method provided by the embodiment of the invention, the writing areas distributed according to the data classification adopt a dynamic distribution mode, the free space is dynamically distributed to each type of the mapped writing areas, and objects with different characteristic attributes share the global ZNS solid state hard disk space.
Based on any one of the above embodiments, fig. 3 is a schematic structural diagram illustrating an intelligent writing distribution device based on a ZNS solid state disk according to an embodiment of the present invention, and the specific contents are as follows.
A write data receiving module 310, configured to receive data to be written issued by a storage application, where the data to be written is new write data;
an object allocation module 320, configured to allocate a target write object for the data to be written;
the feature extraction module 330 is configured to extract multidimensional data attribute features of the target write object, and map the multidimensional data attribute features into a write partition type; wherein the data attribute features comprise an aggregative feature for characterizing the target write object access type, a sequential feature for characterizing the target write object attribute ordering, and a frequency feature for characterizing the target write object access frequency;
and the partition allocation module 340 is configured to allocate a write area for the data to be written according to the write partition type.
According to the intelligent writing distribution device based on the ZNS solid state disk, provided by the embodiment of the invention, by utilizing the characteristic that the ZNS solid state disk is operated by opening a flash memory and extracting the object attribute characteristics facing the storage application, the written data is stored in the ZNS solid state disk in a classified manner, different clustering type objects can be kept to be stored separately, sequential objects and random objects can be stored separately, and cold and hot data objects can be stored separately, so that the performance loss caused by data mixed storage is reduced, the problem of classification failure caused by loss of real data attributes based on an LBA data classification algorithm is avoided, meanwhile, the universality of the algorithm is kept, and meanwhile, the performance advantages of a flash memory medium are fully exerted.
Based on any of the above embodiments, fig. 4 shows a schematic physical structure diagram of an electronic device provided in an embodiment of the present invention, and as shown in fig. 4, the electronic device may include: a processor (processor)410, a communication Interface 420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to perform the following method: receiving data to be written issued by a storage application, wherein the data to be written is new written data; distributing a target writing object for the data to be written; extracting multi-dimensional data attribute characteristics of the target writing object, and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the multi-dimensional data attribute features comprise an aggregative feature for characterizing the target write object access type, a sequential feature for characterizing the target write object attribute ordering, and a frequency feature for characterizing the target write object access frequency; and distributing a writing area for the data to be written according to the writing partition type.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented by a processor to perform the method provided by the foregoing embodiments, for example, including: receiving data to be written issued by a storage application, wherein the data to be written is new written data; distributing a target writing object for the data to be written; extracting multi-dimensional data attribute characteristics of the target writing object, and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the multi-dimensional data attribute features comprise an aggregative feature for characterizing the target write object access type, a sequential feature for characterizing the target write object attribute ordering, and a frequency feature for characterizing the target write object access frequency; and distributing a writing area for the data to be written according to the type of the writing area.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An intelligent writing distribution method based on a ZNS solid state disk is characterized by comprising the following steps:
receiving data to be written issued by a storage application, wherein the data to be written is new written data;
according to the type of data to be written, different target writing objects are allocated to the data to be written, and the target writing objects are a file system allocation unit, a database operation unit and a block device operation unit which are built on a ZNS solid state disk storage system;
extracting multi-dimensional data attribute characteristics of the target writing object, and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the multi-dimensional data attribute features comprise an aggregative feature for characterizing the target write object access type, a sequential feature for characterizing the target write object attribute ordering, and a frequency feature for characterizing the target write object access frequency;
and distributing a writing area for the data to be written according to the type of the writing area.
2. The intelligent write allocation method according to claim 1, wherein the aggregative features of the multidimensional data attribute features are extracted according to:
based on the data association degree of the sample written data, adopting an unsupervised learning algorithm to divide the sample written data into multiple access type groups;
determining an access type of the target write object among the plurality of access types as the aggregative feature in the multi-dimensional data attribute features.
3. The intelligent write allocation method of claim 2 wherein the unsupervised learning algorithm is a gradient boosting decision tree, K-means, hierarchical cluster analysis or DBSCAN.
4. The intelligent write allocation method according to claim 1, wherein the sequential features in the multi-dimensional data attribute features are extracted according to the following:
if the storage index mode of the target writing object is key value oriented storage, performing dictionary sorting based on the name character string of the target writing object to obtain a sequence of the target writing object as the sequence feature in the multi-dimensional data attribute feature.
5. The intelligent write allocation method according to claim 1, wherein the sequential features in the multi-dimensional data attribute features are extracted according to the following:
and if the storage index mode of the target writing object is based on address or logic number storage, binary value sequencing is carried out based on the address or logic number corresponding to the target writing object to obtain a sequence of the target writing object as the sequence feature in the multi-dimensional data attribute feature.
6. The intelligent write allocation method according to claim 1, wherein the frequency features of the multi-dimensional data attribute features are extracted according to the following:
setting a corresponding binary counter for the target writing object;
when data is written once to the target write object, the count of the binary counter is increased by 1;
shifting the count in the binary counter by one bit to the right every fixed time period;
determining the frequency feature in the multi-dimensional data attribute features from the counts in the binary counter.
7. The intelligent write allocation method according to claim 1, wherein after allocating a write area for the data to be written, the method further comprises:
marking the writing area as a writing completion writing area;
distributing partitions with the same multidimensional data attribute characteristics as the writing areas from an idle partition pool of a ZNS solid state disk storage space;
and when the number of the available partitions in the free partition pool of the ZNS solid state disk storage space is less than a preset value, starting a garbage recovery mechanism to generate a new available partition.
8. An intelligent writing distribution device based on a ZNS solid state disk, the device comprising:
the write-in data receiving module is used for receiving data to be written, which is issued by the storage application, wherein the data to be written is new write-in data;
the target writing object distribution module is used for distributing different target writing objects to the data to be written according to the type of the data to be written, wherein the target writing objects are a file system distribution unit, a database operation unit and a block device operation unit which are built on a ZNS solid state disk storage system;
the characteristic extraction module is used for extracting the multi-dimensional data attribute characteristics of the target writing object and mapping the multi-dimensional data attribute characteristics into a writing partition type; wherein the data attribute features comprise an aggregative feature for characterizing the target write object access type, a sequential feature for characterizing the target write object attribute ordering, and a frequency feature for characterizing the target write object access frequency;
and the partition allocation module is used for allocating a write-in area for the data to be written according to the write-in partition type.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.
10. A computer readable storage medium having executable instructions stored thereon, wherein the executable instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210437084.XA CN114546295B (en) | 2022-04-25 | 2022-04-25 | Intelligent writing distribution method and device based on ZNS solid state disk |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210437084.XA CN114546295B (en) | 2022-04-25 | 2022-04-25 | Intelligent writing distribution method and device based on ZNS solid state disk |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114546295A CN114546295A (en) | 2022-05-27 |
| CN114546295B true CN114546295B (en) | 2022-07-01 |
Family
ID=81666595
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210437084.XA Active CN114546295B (en) | 2022-04-25 | 2022-04-25 | Intelligent writing distribution method and device based on ZNS solid state disk |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114546295B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115758206B (en) * | 2022-11-07 | 2023-05-16 | 武汉麓谷科技有限公司 | Method for quickly searching last write end position of Norflash in ZNS solid state disk |
| CN115657972B (en) * | 2022-12-27 | 2023-06-06 | 北京特纳飞电子技术有限公司 | Solid state disk writing control method and device and solid state disk |
| CN116501266B (en) * | 2023-06-27 | 2023-09-12 | 苏州浪潮智能科技有限公司 | Message context processing method, device, computer equipment and storage medium |
| CN117873406B (en) * | 2024-03-11 | 2024-07-09 | 武汉麓谷科技有限公司 | Method for controlling wear balance of ZNS solid state disk |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109460186A (en) * | 2018-11-02 | 2019-03-12 | 深圳忆联信息系统有限公司 | A kind of method and its system promoting solid state hard disk reading performance |
| CN112463333A (en) * | 2020-12-03 | 2021-03-09 | 北京浪潮数据技术有限公司 | Data access method, device and medium based on multithreading concurrency |
| CN113515231A (en) * | 2020-04-09 | 2021-10-19 | 爱思开海力士有限公司 | Data storage device and operation method thereof |
| CN113778327A (en) * | 2020-06-10 | 2021-12-10 | 西部数据技术公司 | Zone allocation for data storage devices based on zone reset behavior |
| CN114138193A (en) * | 2021-11-25 | 2022-03-04 | 郑州云海信息技术有限公司 | Data writing method, device and equipment for solid state disk with partitioned name space |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200089407A1 (en) * | 2019-11-22 | 2020-03-19 | Intel Corporation | Inter zone write for zoned namespaces |
| US11455124B2 (en) * | 2020-10-09 | 2022-09-27 | Western Digital Technologies, Inc. | Command prioritization to reduce latencies of zone commands |
-
2022
- 2022-04-25 CN CN202210437084.XA patent/CN114546295B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109460186A (en) * | 2018-11-02 | 2019-03-12 | 深圳忆联信息系统有限公司 | A kind of method and its system promoting solid state hard disk reading performance |
| CN113515231A (en) * | 2020-04-09 | 2021-10-19 | 爱思开海力士有限公司 | Data storage device and operation method thereof |
| CN113778327A (en) * | 2020-06-10 | 2021-12-10 | 西部数据技术公司 | Zone allocation for data storage devices based on zone reset behavior |
| CN112463333A (en) * | 2020-12-03 | 2021-03-09 | 北京浪潮数据技术有限公司 | Data access method, device and medium based on multithreading concurrency |
| CN114138193A (en) * | 2021-11-25 | 2022-03-04 | 郑州云海信息技术有限公司 | Data writing method, device and equipment for solid state disk with partitioned name space |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114546295A (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114546295B (en) | Intelligent writing distribution method and device based on ZNS solid state disk | |
| US11681754B2 (en) | Technologies for managing connected data on persistent memory-based systems | |
| KR101994021B1 (en) | File manipulation method and apparatus | |
| CN110209490B (en) | Memory management method and related equipment | |
| US11288287B2 (en) | Methods and apparatus to partition a database | |
| US10712943B2 (en) | Database memory monitoring and defragmentation of database indexes | |
| CN114138193B (en) | Data writing method, device and equipment for partition naming space solid state disk | |
| CN110968269A (en) | SCM and SSD-based key value storage system and read-write request processing method | |
| CN100424699C (en) | Attribute extensible object file system | |
| US20210011634A1 (en) | Methods and systems for managing key-value solid state drives (kv ssds) | |
| CN110674052B (en) | Memory management method, server and readable storage medium | |
| WO2023143095A1 (en) | Method and system for data query | |
| CN112148736B (en) | Method, device and storage medium for caching data | |
| CN104142979B (en) | A kind of indexing means for realizing RFID tag storage management | |
| CN101063976B (en) | Method and equipment for fast deletion of physically clustered data | |
| CN106201918B (en) | A kind of method and system based on big data quantity and extensive caching quick release | |
| CN110008030A (en) | A kind of method of metadata access, system and equipment | |
| CN104133970A (en) | Data space management method and device | |
| CN104102735A (en) | File system optimizing method and file system optimizing device aiming at database files | |
| CN108664217B (en) | Caching method and system for reducing jitter of writing performance of solid-state disk storage system | |
| CA2415018C (en) | Adaptive parallel data clustering when loading a data structure containing data clustered along one or more dimensions | |
| CN110825953B (en) | Data query method, device and equipment | |
| US11977485B2 (en) | Method of cache management based on file attributes, and cache management device operating based on file attributes | |
| CN116302376A (en) | Process creation method, process creation device, electronic equipment and computer readable medium | |
| CN115586943A (en) | Hardware marking implementation method for dirty pages of virtual machine of intelligent network card |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |