CN114415967A - Multimedia intelligent storage disk design method based on ZNS solid state disk - Google Patents
Multimedia intelligent storage disk design method based on ZNS solid state disk Download PDFInfo
- Publication number
- CN114415967A CN114415967A CN202210088471.7A CN202210088471A CN114415967A CN 114415967 A CN114415967 A CN 114415967A CN 202210088471 A CN202210088471 A CN 202210088471A CN 114415967 A CN114415967 A CN 114415967A
- Authority
- CN
- China
- Prior art keywords
- data
- multimedia
- multimedia data
- ddr
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 99
- 239000007787 solid Substances 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title description 16
- 238000013461 design Methods 0.000 title description 6
- 238000012545 processing Methods 0.000 claims abstract description 73
- 238000007405 data analysis Methods 0.000 claims abstract description 21
- 230000006835 compression Effects 0.000 claims abstract description 10
- 238000007906 compression Methods 0.000 claims abstract description 10
- 238000012546 transfer Methods 0.000 claims abstract description 8
- 238000010801 machine learning Methods 0.000 claims abstract description 6
- 238000007726 management method Methods 0.000 claims description 61
- 238000009826 distribution Methods 0.000 claims description 21
- 238000005192 partition Methods 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 7
- 238000013468 resource allocation Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0868—Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides an intelligent storage disc that contains multimedia management system, its characterized in that: the intelligent storage disk with the multimedia management system comprises at least one standard ZNS solid state hard disk, an in-disk multimedia processing accelerator and a DDR data cache; the multimedia processing accelerator module comprises a media data object management module, a multimedia data processing module, a multimedia data analysis module and the like; the multimedia data object management module runs a multimedia management algorithm, realizes access control by adopting an object storage mode and organizes multimedia data into an object mode for management; the multimedia data processing module carries out transcoding, compression and other processing on multimedia data such as voice, pictures, videos and the like; the multimedia data analysis module adopts a machine learning algorithm to analyze and extract the characteristics of multimedia; the DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing and analyzing module.
Description
Technical Field
The application belongs to the technical field of solid state disk application, and particularly relates to an intelligent solid state disk for accessing multimedia data.
Background
Compared with the traditional magnetic disk, the solid state disk has low delay, high bandwidth, strong concurrency capability and low power consumption, also has the oscillation resistance which is not possessed by a mechanical hard disk, is used as a substitute of the magnetic disk, and is widely applied to the environment of a high-performance computer system. With the development of flash memory technology and bus technology, the traditional method for operating the solid state disk according to the magnetic disk increasingly restricts the development of the solid state disk technology, and the partitioned ZNS solid state disk based on the open flash memory operation interface can be directly exposed to an application program in a flash memory operation mode, so that the application program can be conveniently optimized for the flash memory.
ZNS SSD is a partition namespace solid state disk, ZNS is developed from OC (Open Channel) SSD, FTL (Flash Translation Layer) is moved to a Host end of an upper Layer from the inside of SSD, and the inside of SSD is opened to the Host end, so that a user can flexibly own a specific FTL according to the needs of the user.
Different zones and different physical positions can be mapped in the ZNS SSD according to different application scenes; config a is the configuration that is most beneficial for sequential read and write, and this method can ensure the best concurrency of the read and write process Die, and in general, standard SSD will be written in this way. The Config B is more beneficial to Latency sensitive scenes, and each user controls own Die, thereby reducing mutual influence.
The ZRWA allows for random writes and in-place overwrites of data in the SSD cache. This approach requires more resources on the SSD than the Zone Append command. The ZNS SSD requires that all the zones can only be written in sequence (or appended and the like), cannot be subjected to repeated writing operation, and a zone reset command and the like need to be executed on the zones before reuse, so that more applications and drives need to be matched with the ZNS SSD, but all large storage systems and open source tools are already or plan to complete the adaptation and support of the ZNS SSD currently, and a standardized interface realizes a powerful software and hardware ecosystem.
Compared with the common SSD product, ZNS SSD mainly has five advantages: WAF (Write Amplification) of a GC (Garbage Collection) process of the SSD is eliminated, traditional OP (Over provisioning) space is eliminated, due to a smaller mapping table, the DRAM is fewer, the throughput is higher, the delay is lower, and the service life of the SSD disk is greatly prolonged.
According to the existing multimedia data storage scheme, a traditional file system is used for managing data, data access is achieved through a disk mode to access a solid state disk, data processing is completed through an additional operation part, a host, a processing part and an SSD are required to conduct a large amount of data interaction for each data processing, IO paths are long, excessive data interaction severely occupies system resources, and storage space waste is caused due to invalid data storage. And different ZONEs and different physical positions can be mapped in the ZNS SSD according to different application scenes, and when large-capacity multimedia data is stored and taken, resource difference values and idle superblock resources can be subjected to difference processing in a partition storage mode, so that the resource efficiency of the system is improved.
Disclosure of Invention
The invention aims to provide a ZNS-based solid state disk design method capable of supporting a multimedia management function, which simplifies the design of a multimedia system, reduces the waste of system resources, improves the utilization rate of storage space and improves the throughput rate of the system.
The invention provides an intelligent storage disk with a multimedia management system, which is characterized in that: the intelligent storage disk with the multimedia management system comprises at least one standard ZNS solid state hard disk, an in-disk multimedia processing accelerator and a DDR data cache; the multimedia processing accelerator module comprises a media data object management module, a multimedia data processing module, a multimedia data analysis module and the like; the multimedia data object management module runs a multimedia management algorithm, realizes access control by adopting an object storage mode and organizes multimedia data into an object mode for management; the multimedia data processing module carries out transcoding, compression and other processing on multimedia data such as voice, pictures, videos and the like; the multimedia data analysis module adopts a machine learning algorithm to analyze and extract the characteristics of multimedia; the DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing and analyzing module.
Furthermore, the multimedia data analysis module analyzes the transmitted data into multimedia data and other data types, and the multimedia data analysis module analyzes the multimedia data according to the identifier corresponding to the multimedia data type, analyzes the coding and decoding format of the received multimedia data, and distinguishes the multimedia data through different identifiers.
Further, the multimedia data parsing module determines a corresponding encoding format according to the corresponding identifier, such as an audio format class corresponding to the identifier 11H, a video format class corresponding to the identifier 12H, a picture format class corresponding to the identifier 13H, a text format class corresponding to the identifier 14H, and the like. The multimedia data processing module carries out transcoding, compression and other processing on the analyzed multimedia data.
Further, DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing module and an analysis module; the data write-in control module is used for controlling intermediate data of the data processing module and the analysis module to be written and stored into the DDR data cache and the ZNS solid state disk; and the write-in control module comprises a judging execution unit, and the judging execution unit is used for inquiring and reading the multimedia data corresponding to the data instruction from the DDR data cache when receiving the data reading instruction.
Further, when the DDR data cache contains multimedia data corresponding to the data reading instruction, the data writing operation in a first storage unit of the DDR data cache is stopped, meanwhile, the generated multimedia data are continuously written into a second storage unit of the DDR data cache, the multimedia data corresponding to the data reading instruction are read from the first storage unit of the DDR data cache, and then the multimedia data missing from the first storage unit of the DDR data cache are recovered according to the multimedia data in the second storage unit of the DDR data cache; and when the DDR data cache comprises the multimedia data corresponding to the write-in data instruction, stopping the read data operation in the first storage unit of the DDR data cache, continuing to perform the generated read data operation in the second storage unit of the DDR data cache, writing the multimedia data corresponding to the data instruction in the first storage unit of the DDR data cache, and then recovering the missing multimedia data in the first storage unit of the DDR data cache according to the read multimedia data in the second storage unit of the DDR data cache.
Further, when the multimedia data are stored, firstly, writing object resource allocation is carried out, wherein the writing object resource allocation comprises storage space allocation, DDR data cache allocation, metadata allocation and the like, then the writing data are transmitted to the DDR cache, the data processing module carries out data processing in the ZNS solid state disk on the multimedia data, the data analyzing module carries out feature extraction and analysis in the ZNS solid state disk on the multimedia data, then an NVMe storage command is generated for the processed and analyzed object data, data falling is completed, and finally the object metadata are updated, and the storage type command operation is completed.
And further, storing the multimedia data into a partition of the ZNS solid state disk by adopting an active zone resource difference value and idle superblock resource difference value equalization processing. The intelligent storage disk further comprises a distribution management module, the distribution management module assigns a first set difference value and a second set difference value according to the efficiency value of system resource optimization calculation, compares whether the active zone resource difference value is larger than the first set difference value and whether the idle superblock resource difference value is larger than the second set difference value, and preferentially selects the loadable kernel module with the largest number of idle superblock resources to distribute and execute write operation when the active zone resource exceeds the set first set difference value. The first setting difference is 4-5 and the second setting difference is 6-8.
Further, for the query type command, the multimedia data object management module firstly analyzes object metadata, indexes to an actual storage position, then generates an NVMe command reading command for the object, scans and reads out part of classified data of the partitions from the ZNS solid state disk, then controls the data analysis module to perform in-partition disk analysis and data feature extraction corresponding to the command on the multimedia data, controls the data processing module to perform in-partition disk data processing, and finally returns data or/and other information required by Host through the interface module according to the analyzed and processed object data.
Compared with the background technology, the invention has the beneficial effects that: the invention manages multimedia data by adopting an object management mode based on the ZNS solid state disk, simplifies system design, improves IO efficiency, improves space utilization rate by adopting the solid state disk scheme without FTL, and reduces the influence of write amplification. The built-in multimedia data processing and analyzing module improves the multimedia data processing efficiency, reduces the bus data transmission quantity and the system overhead, and improves the utilization rate of the available space. And performing differentiated balance processing on the active zone resource difference value and the idle superblock resource, and writing and storing the multimedia data to the ZNS solid state disk in a partition storage mode, so that the resource efficiency of the system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or technical solutions in related arts, the drawings used in the description of the embodiments or prior arts will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
The structures, proportions, and dimensions shown in the drawings and described in the specification are for illustrative purposes only and are not intended to limit the scope of the present disclosure, which is defined by the claims, but rather by the claims, it is understood that these drawings and their equivalents are merely illustrative and not intended to limit the scope of the present disclosure.
FIG. 1 is a system architecture diagram of a multimedia smart storage disk;
FIG. 2 is a flow chart of multimedia data storage;
fig. 3 is a flow chart of multimedia data query.
Detailed Description
Embodiments of the present application will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the application are shown, and in which it is to be understood that the embodiments described are merely illustrative of some, but not all, of the embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The invention relates to a solid state disk design method, which is based on a ZNS solid state disk and is used for designing an intelligent storage disk containing a multimedia management system, wherein the multimedia system storage disk comprises at least one standard ZNS solid state disk, an in-disk multimedia processing accelerator and a DDR data cache; the DDR data cache is used for data transfer, and intermediate data temporary storage of the data processing and analyzing module; the multimedia processing accelerator module comprises three parts, wherein the multimedia data object management module runs a multimedia management algorithm, realizes access control by adopting an object storage mode and organizes multimedia data into an object mode for management; the multimedia data processing module carries out transcoding, compression and other processing on multimedia data such as voice, pictures, videos and the like; the multimedia data analysis module adopts a machine learning algorithm to analyze and extract the characteristics of the multimedia.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.
Example 1
The intelligent storage disk comprising the multimedia management system comprises at least one standard ZNS solid state hard disk, an in-disk multimedia processing accelerator and a DDR data cache; the multimedia processing accelerator module comprises a media data object management module, a multimedia data processing module, a multimedia data analysis module and the like; the multimedia data object management module runs a multimedia management algorithm, realizes access control by adopting an object storage mode and organizes multimedia data into an object mode for management; the multimedia data processing module carries out transcoding, compression and other processing on multimedia data such as voice, pictures, videos and the like; the multimedia data analysis module adopts a machine learning algorithm to analyze and extract the characteristics of multimedia; the DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing and analyzing module.
As shown in fig. 1, the intelligent storage disk with a multimedia management system includes at least one standard ZNS solid state hard disk, an on-disk multimedia processing accelerator, and a DDR data cache; the multimedia processing accelerator module comprises a media data object management module, a multimedia data processing module, a multimedia data analysis module and the like;
the multimedia data object management module runs a multimedia management algorithm, realizes access control by adopting an object storage mode and organizes multimedia data into an object mode for management;
the Host receives multimedia storage and query commands through the object operation interface module, and then forwards the multimedia storage and query commands to the multimedia data object management module; the multimedia data object management module manages multimedia data by using an object storage technology, which takes the multimedia data as a storage object, the ZNS solid state disk is used as a storage medium to complete all storage management work such as storage space allocation, random writing into a ZONE partition of the ZNS solid state disk, sequential writing conversion, garbage recovery, indexing and the like, controlling the data to be transmitted and received among the object interface module, the DDR data cache and the ZNS solid state disk according to the type of the received command, meanwhile, a control command is generated, the multimedia data processing module is operated to perform transcoding, compression, decompression, encryption, decryption and other operation processing on multimedia voice, picture and video data in the DDR data cache, the multimedia data analysis module is operated to perform analysis, feature extraction and other operations on the multimedia data in the DDR data cache, and the ZNS solid state disk is operated to perform actual data access;
the multimedia data analysis module analyzes multimedia data and other data types from the transmitted data, the multimedia data analysis module analyzes the multimedia data according to the identification corresponding to the multimedia data type, analyzes the coding and decoding formats of the received multimedia data, the coding formats of the multimedia data are also distinguished through different identifications, and the multimedia data analysis module determines the corresponding coding formats according to the corresponding identifications, such as an audio format class corresponding to the identification 11H, a video format class corresponding to the identification 12H, a picture format class corresponding to the identification 13H, a text format class corresponding to the identification 14H and the like. Of course, the above lists only major categories, and the categories are not detailed into specific subclasses, for example, the picture format category also includes JPG coding format, BMP coding format, GIF coding format, and the like; and the machine learning algorithm is adopted to perform operations such as feature analysis, extraction and classification on the multimedia data, so that the data can be classified, cached and stored conveniently.
The multimedia data processing module carries out transcoding, compression and other processing on the analyzed multimedia data;
according to the processing results of the data processing module and the data analysis module, performing storage management of data objects, including but not limited to data clustering storage, data cutting, compression storage and the like, and returning a HOST end user command;
the DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing module and an analysis module; the data write-in control module is used for controlling intermediate data of the data processing module and the analysis module to be written and stored into the DDR data cache and the ZNS solid state disk; the write-in control module comprises a judging execution unit, and the judging execution unit is used for inquiring and reading the multimedia data corresponding to the data instruction from the DDR data cache when receiving the data reading instruction; if the DDR data cache contains the multimedia data corresponding to the data reading instruction, stopping the data writing operation in a first storage unit of the DDR data cache, simultaneously continuing to write the generated multimedia data into a second storage unit of the DDR data cache, reading the multimedia data corresponding to the data reading instruction from the first storage unit of the DDR data cache, and then recovering the missing multimedia data in the first storage unit of the DDR data cache according to the multimedia data in the second storage unit of the DDR data cache;
if the DDR data cache comprises the multimedia data corresponding to the write-in data instruction, stopping the data reading operation in a first storage unit of the DDR data cache, meanwhile, continuing to perform the generated data reading operation in a second storage unit of the DDR data cache, writing the multimedia data corresponding to the data instruction in the first storage unit of the DDR data cache, and then recovering the missing multimedia data in the first storage unit of the DDR data cache according to the multimedia data read in the second storage unit of the DDR data cache.
As shown in fig. 2, for the storage type command, first, write object resource allocation is performed, including storage space allocation, DDR data cache allocation, metadata allocation, and the like, then write data are received and sent to the DDR cache, the operation data processing module performs in-disk data processing on the multimedia data, the operation data parsing module performs in-disk feature extraction and parsing on the multimedia data, then an NVMe storage command is generated for the processed and parsed object data, data destaging is completed, and finally, the object metadata is updated, and the storage type command operation is completed.
The data cache comprises a DDR data cache region, a ZNS solid state disk and a ZNS solid state disk, wherein the DDR data cache region is used for caching written multimedia data, and the ZNS solid state disk is used for storing the written multimedia data. And the multimedia data writing module is used for simultaneously writing the multimedia data into the DDR data cache region and the ZNS solid state disk.
When the multimedia data are written and stored in the ZNS solid state disk, the resource efficiency of the system can be improved in a partition storage mode. The partitioned writing distribution method for storing multimedia data into a ZNS solid state disk mainly performs differentiated balancing processing on an active zone resource difference value and an idle superblock resource, and the specific implementation method comprises the following steps:
s1, a distribution management module core monitors the real-time active zone resource quantity and the idle superblock resource quantity of each loadable kernel module core at the back end; the method comprises the following specific steps:
s11, the distribution management module core monitors the real-time active zone resource quantity and the idle superblock resource quantity of each loadable kernel module core at the back end; s12, when the number of active zone resources and the number of idle superblock resources on the loadable kernel module core increase or decrease, the distribution management module core carries out phase recording;
s2, the distribution management module core counts the loadable kernel module core with the maximum number of active zone resources and the loadable kernel module core with the minimum number of active zone resources in the loadable kernel module cores; the distribution management module core counts the loadable kernel module core with the largest quantity of idle superblock resources and the loadable kernel module core with the smallest quantity of idle superblock resources in each loadable kernel module core;
s3, the distribution management module core obtains the loadable kernel module core with the maximum number of active zone resources and the loadable kernel module core with the minimum number of active zone resources in the loadable kernel module cores, and calculates the difference value of the active zone resources of the loadable kernel module core and the loadable kernel module core; the distribution management module core obtains a loadable core module core with the largest quantity of idle superblock resources and a loadable core module core with the smallest quantity of idle superblock resources in each loadable core module core, and calculates an idle superblock resource difference value;
s4, when the distribution management module core receives a write distribution task of a front-end DM (distribution module) core, carrying out equalization processing according to the active zone resource difference value and the idle superblock resource difference value, and selecting a loadable kernel module core to be distributed; the method comprises the following specific steps:
the distribution management module core judges whether the difference value of the active zone resources is larger than a first set difference value or not, and the difference value of the idle superblock resources is larger than a second set difference value; the first setting difference and the second setting difference can be assigned according to the efficiency value of the system resource optimization calculation, for example, the first setting difference is 4-5, and the second setting difference is 6-8; the general principle is that when the active zone resources exceed a first set difference value, the loadable kernel module with the largest number of idle superblock resources is preferentially selected to distribute:
when the difference value of the active zone resources is larger than a first set difference value and the difference value of the idle superblock resources is larger than a second set difference value, the distribution management module core selects a loadable kernel module core with the largest number of the idle superblock resources as a loadable kernel module core to be distributed;
when the difference value of the active zone resources is larger than a first set difference value and the difference value of the idle superblock resources is smaller than a second set difference value, the distribution management module core selects a loadable kernel module core with the largest number of the idle superblock resources as a loadable kernel module core to be distributed;
when the difference value of the idle superblock resources is larger than a second set difference value and the difference value of the active zone resources is smaller than a first set difference value, the distribution management module core selects the loadable kernel module core with the largest number of the idle superblock resources as the loadable kernel module core to be distributed;
and when the active zone resource difference value is larger than a first set difference value and the idle superblock resource difference value is smaller than a second set difference value, or when the idle superblock resource difference value is larger than the second set difference value and the active zone resource difference value is smaller than the first set difference value, the distribution management module core selects the loadable kernel module core with the largest number of idle superblock resources as the loadable kernel module core to be distributed.
And when the write resource allocation is carried out, the distribution tasks are written in sequence until the allocation is successful.
The core concurrent resources of the loadable kernel module at the rear end are used in a balanced manner, so that the balance of the performance and the resource use of the multimedia data with large data volume when the multimedia data is stored in the ZNS solid state disk is realized, the algorithm complexity is reduced, and the processing capacity and the processing speed of the whole storage system are enhanced.
As shown in fig. 3, for the query type command, the multimedia data object management module first performs object metadata parsing, indexes to an actual storage location, then generates an NVMe command read command for the object, scans and reads out the classified data of part of the partitions from the ZNS solid state disk, then controls the data parsing module to perform in-partition disk parsing and data feature extraction corresponding to the command on the multimedia data, controls the data processing module to perform in-partition disk data processing, and finally returns data or/and other information required by Host through the interface module according to the parsed and processed object data.
The process of scanning and reading classification data of partial partitions from the ZNS solid state disk comprises the following steps: scanning unit for scanning data stored in ZNS solid state disk, the scanning unit may include:
the data scanning subunit is used for scanning the data in the ZNS solid state disk based on the multimedia type to obtain multimedia data corresponding to the multimedia type in the ZNS solid state disk; or distinguishing multimedia data and non-multimedia data in the ZNS solid state disk to obtain multimedia data corresponding to the multimedia type; and the information acquisition subunit is used for acquiring the name information and the storage path information of the multimedia data.
In the embodiment of the application, when a query request aiming at multimedia data is received, file data corresponding to the multimedia data is obtained in the ZNS solid state disk according to the storage path information of the multimedia data, and the data is analyzed to obtain the multimedia data and the description information of the multimedia data, and all the multimedia data in the ZNS solid state disk do not need to be analyzed one by one. Therefore, by adopting the inquiring and reading mode of the application, the data processing process of the multimedia management system can be greatly shortened, and the data processing capacity is improved.
The embodiments in the present description are described in a progressive manner, or in a parallel manner, or in a combination of a progressive manner and a parallel manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments can be referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It should be noted that in the description of the present application, it is to be understood that the terms "upper", "lower", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, are only used for convenience in describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be configured and operated in a specific orientation, and thus, should not be construed as limiting the present application. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or intervening components may also be present.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such article or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in an article or device that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. An intelligent storage disk comprising a multimedia management system, characterized in that: the intelligent storage disk with the multimedia management system comprises at least one standard ZNS solid state hard disk, an in-disk multimedia processing accelerator and a DDR data cache; the multimedia processing accelerator module comprises a media data object management module, a multimedia data processing module, a multimedia data analysis module and the like; the multimedia data object management module runs a multimedia management algorithm, realizes access control by adopting an object storage mode and organizes multimedia data into an object mode for management; the multimedia data processing module carries out transcoding, compression and other processing on multimedia data such as voice, pictures, videos and the like; the multimedia data analysis module adopts a machine learning algorithm to analyze and extract the characteristics of multimedia; the DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing and analyzing module.
2. The intelligent storage disk with multimedia management system of claim 1, wherein: the multimedia data analysis module analyzes multimedia data and other data types from the transmitted data, analyzes the multimedia data according to the identification corresponding to the multimedia data type, analyzes the coding and decoding format of the received multimedia data, and distinguishes the multimedia data through different identifications.
3. The intelligent storage disk with multimedia management system of claim 2, wherein: the multimedia data analysis module determines a corresponding encoding format according to the corresponding identifier, such as an audio format class corresponding to the identifier 11H, a video format class corresponding to the identifier 12H, a picture format class corresponding to the identifier 13H, a text format class corresponding to the identifier 14H, and the like.
4. The intelligent storage disk with multimedia management system of claim 1, wherein: the multimedia data processing module carries out transcoding, compression and other processing on the analyzed multimedia data.
5. The intelligent storage disk with multimedia management system of claim 1, wherein: the DDR data cache is used for data transfer and comprises intermediate data temporary storage of a data processing module and an analysis module; the data write-in control module is used for controlling intermediate data of the data processing module and the analysis module to be written and stored into the DDR data cache and the ZNS solid state disk; and the write-in control module comprises a judging execution unit, and the judging execution unit is used for inquiring and reading the multimedia data corresponding to the data instruction from the DDR data cache when receiving the data reading instruction.
6. The intelligent storage disk with multimedia management system of claim 5, wherein: when the DDR data cache contains the multimedia data corresponding to the data reading instruction, stopping the data writing operation in a first storage unit of the DDR data cache, continuously writing the generated multimedia data into a second storage unit of the DDR data cache, reading the multimedia data corresponding to the data reading instruction from the first storage unit of the DDR data cache, and then recovering the missing multimedia data in the first storage unit of the DDR data cache according to the multimedia data in the second storage unit of the DDR data cache;
and when the DDR data cache comprises the multimedia data corresponding to the write-in data instruction, stopping the read data operation in the first storage unit of the DDR data cache, continuing to perform the generated read data operation in the second storage unit of the DDR data cache, writing the multimedia data corresponding to the data instruction in the first storage unit of the DDR data cache, and then recovering the missing multimedia data in the first storage unit of the DDR data cache according to the read multimedia data in the second storage unit of the DDR data cache.
7. The intelligent storage disk with multimedia management system of claim 1, wherein: when the multimedia data are stored, firstly, writing object resource allocation is carried out, wherein the writing object resource allocation comprises storage space allocation, DDR data cache allocation, metadata allocation and the like, then, data are transmitted and written to a DDR cache, a data processing module carries out data processing in a ZNS solid state disk on the multimedia data, a data analysis module carries out feature extraction and analysis in the ZNS solid state disk on the multimedia data, then, an NVMe storage command is generated for the processed and analyzed object data, data drop is completed, and finally, the object metadata are updated, and storage type command operation is completed.
8. The intelligent storage disk with multimedia management system of claim 1, wherein: and storing the multimedia data into a partition of the ZNS solid state disk by adopting the active zone resource difference value and the idle superblock resource difference value equalization processing.
9. The intelligent storage disk with multimedia management system of claim 8, wherein: the intelligent storage disk further comprises a distribution management module, the distribution management module assigns a first set difference value and a second set difference value according to the efficiency value of system resource optimization calculation, compares whether the active zone resource difference value is larger than the first set difference value and whether the idle superblock resource difference value is larger than the second set difference value, and preferentially selects the loadable kernel module with the largest number of idle superblock resources to distribute and execute write operation when the active zone resource exceeds the set first set difference value.
10. The intelligent storage disk with multimedia management system of claim 9, wherein: the first setting difference is 4-5 and the second setting difference is 6-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210088471.7A CN114415967A (en) | 2022-01-25 | 2022-01-25 | Multimedia intelligent storage disk design method based on ZNS solid state disk |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210088471.7A CN114415967A (en) | 2022-01-25 | 2022-01-25 | Multimedia intelligent storage disk design method based on ZNS solid state disk |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114415967A true CN114415967A (en) | 2022-04-29 |
Family
ID=81277635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210088471.7A Pending CN114415967A (en) | 2022-01-25 | 2022-01-25 | Multimedia intelligent storage disk design method based on ZNS solid state disk |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114415967A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115686372A (en) * | 2022-11-07 | 2023-02-03 | 武汉麓谷科技有限公司 | ZNS solid state disk ZRWA function-based data management method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101505547A (en) * | 2008-02-05 | 2009-08-12 | 北京掌讯远景数码信息技术有限公司 | Wireless terminal |
| CN101604548A (en) * | 2009-03-26 | 2009-12-16 | 成都市华为赛门铁克科技有限公司 | A kind of solid state hard disc and date storage method |
| CN102262657A (en) * | 2011-06-29 | 2011-11-30 | 成都市华为赛门铁克科技有限公司 | Method and system for storing multimedia data |
| CN108881942A (en) * | 2018-06-06 | 2018-11-23 | 西安交通大学 | A kind of super fusion normality recording and broadcasting system based on distributed objects storage |
| CN111694515A (en) * | 2020-05-23 | 2020-09-22 | 苏州浪潮智能科技有限公司 | Zone writing distribution method and system based on ZNS solid state disk |
| US20210334216A1 (en) * | 2020-04-22 | 2021-10-28 | Netapp, Inc. | Methods for managing storage systems with dual-port solid-state disks accessible by multiple hosts and devices thereof |
| US20210373785A1 (en) * | 2020-05-29 | 2021-12-02 | Seagate Technology Llc | Data storage device with adaptive object storage capabilities |
-
2022
- 2022-01-25 CN CN202210088471.7A patent/CN114415967A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101505547A (en) * | 2008-02-05 | 2009-08-12 | 北京掌讯远景数码信息技术有限公司 | Wireless terminal |
| CN101604548A (en) * | 2009-03-26 | 2009-12-16 | 成都市华为赛门铁克科技有限公司 | A kind of solid state hard disc and date storage method |
| CN102262657A (en) * | 2011-06-29 | 2011-11-30 | 成都市华为赛门铁克科技有限公司 | Method and system for storing multimedia data |
| CN108881942A (en) * | 2018-06-06 | 2018-11-23 | 西安交通大学 | A kind of super fusion normality recording and broadcasting system based on distributed objects storage |
| US20210334216A1 (en) * | 2020-04-22 | 2021-10-28 | Netapp, Inc. | Methods for managing storage systems with dual-port solid-state disks accessible by multiple hosts and devices thereof |
| CN111694515A (en) * | 2020-05-23 | 2020-09-22 | 苏州浪潮智能科技有限公司 | Zone writing distribution method and system based on ZNS solid state disk |
| US20210373785A1 (en) * | 2020-05-29 | 2021-12-02 | Seagate Technology Llc | Data storage device with adaptive object storage capabilities |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115686372A (en) * | 2022-11-07 | 2023-02-03 | 武汉麓谷科技有限公司 | ZNS solid state disk ZRWA function-based data management method |
| CN115686372B (en) * | 2022-11-07 | 2023-07-25 | 武汉麓谷科技有限公司 | ZNS solid state disk ZRWA function-based data management method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741053B2 (en) | Data management system, method, terminal and medium based on hybrid storage | |
| US10705974B2 (en) | Data processing method and NVME storage device | |
| US10296462B2 (en) | Method to accelerate queries using dynamically generated alternate data formats in flash cache | |
| CN103399823B (en) | The storage means of business datum, equipment and system | |
| US20170124077A1 (en) | Flash module provided with database operation unit, and storage device | |
| CN113821171B (en) | Key value storage method based on hash table and LSM tree | |
| KR20200122994A (en) | Key Value Append | |
| US20220164316A1 (en) | Deduplication method and apparatus | |
| CN112632069B (en) | Hash table data storage management method, device, medium and electronic equipment | |
| CN103902467A (en) | Compressed memory access control method, device and system | |
| US20120158804A1 (en) | Method for converting data in optical disc drive | |
| CN114546295B (en) | Intelligent writing distribution method and device based on ZNS solid state disk | |
| CN104298697A (en) | FAT32-format data file managing system | |
| US20240160603A1 (en) | Data input/output method using storage node-based key-value store | |
| CN114415967A (en) | Multimedia intelligent storage disk design method based on ZNS solid state disk | |
| WO2022037015A1 (en) | Column-based storage method, apparatus and device based on persistent memory | |
| CN1963810A (en) | Method for realizing cache memory relates to file allocation table on Flash storage medium | |
| WO2024032526A1 (en) | Data retrieval processing method and system | |
| US20060143313A1 (en) | Method for accessing a storage device | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| CN111581247B (en) | Data manager, time sequence database and information processing system | |
| CN110263005B (en) | Storage system management system for realizing data content locality read-write optimization | |
| US9069821B2 (en) | Method of processing files in storage system and data server using the method | |
| CN115964350A (en) | File system management model and system | |
| US8051090B2 (en) | File management method of a ring buffer and related file management apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220429 |
|
| RJ01 | Rejection of invention patent application after publication |