CN115454330A - Method for managing multiple SSD reads and writes in parallel - Google Patents
Method for managing multiple SSD reads and writes in parallel Download PDFInfo
- Publication number
- CN115454330A CN115454330A CN202210930161.5A CN202210930161A CN115454330A CN 115454330 A CN115454330 A CN 115454330A CN 202210930161 A CN202210930161 A CN 202210930161A CN 115454330 A CN115454330 A CN 115454330A
- Authority
- CN
- China
- Prior art keywords
- ssd
- buffer
- read
- file
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0616—Improving the reliability of storage systems in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Abstract
The invention discloses a method for managing reading and writing of a plurality of SSDs in parallel, and relates to the technical field of data storage. The specific implementation mode of the method comprises the following steps: the CPU creates a data processing thread and a plurality of SSD threads, and allocates one SSD thread for each SSD; receiving a data write request through a data processing thread; under the condition that the size of a file to be written reaches a threshold value of a written file, respectively judging whether buffered data in a buffer area of each SSD meets the requirement of a buffer threshold value; if so, dividing the file to be written into a plurality of subfiles, and respectively writing the plurality of subfiles into the SSD corresponding to the SSD thread by using a plurality of SSD threads. The embodiment can manage the plurality of SSDs, so that the plurality of SSDs can read and write different areas of the same file in parallel and synchronously, and data can be directly written in the physical address of the hard disk, thereby greatly improving the reading and writing speed of the file, meeting the reading and writing requirements of various data formats, preventing the loss of the SSDs and greatly prolonging the service life of the SSDs.
Description
Technical Field
The invention belongs to the technical field of data storage, and particularly relates to a method for parallelly managing reading and writing of multiple SSDs.
Background
A Solid State Disk (SSD, also called Solid State Drive for short) refers to a hard Disk manufactured by using a Solid State electronic memory chip array, and the hard Disk uses an electronic memory chip as a medium, and generally includes a control chip and a FLASH memory chip (also called FLASH chip).
In the existing SSD data read/write process, when a plurality of SSDs are used to read/write data simultaneously, a mode of a plurality of SSD groups of raid (Redundant Arrays of Independent Drives, that is, disk Arrays) is often adopted. However, on one hand, when a plurality of SSD disk arrays are used to read and write a same file, the file is usually divided into N parts, and a plurality of SSDs are read and written sequentially, that is, after one SSD finishes reading and writing, the next SSD can read and write subsequently, so that the read and write speed of the file is low; on the other hand, the read-write speed of the file does not increase correspondingly with the increase of the number of the SSDs, that is, increasing the number of the SSDs may not bring about the effect of increasing the read-write speed of the file, but rather increases the loss of the SSDs themselves.
Disclosure of Invention
In view of the above, the present invention provides a method for managing multiple SSDs in parallel, which can manage multiple SSDs, so that multiple SSDs read and write different areas of the same file in parallel and synchronously, and write data directly at a physical address of a hard disk, and the read-write rate of the SSDs can reach the theoretical maximum read-write rate of the hard disk, so that the read-write rate of the file is greatly increased, the read-write requirements of various data formats can be met, the self-loss of the SSDs is prevented, and the service life of the SSDs is greatly prolonged.
The technical scheme for realizing the invention is as follows:
a method of managing reading and writing of a plurality of SSDs in parallel, the method performed with a reactive recording device comprising a CPU and a memory card, the memory card comprising a plurality of SSDs, the method comprising:
the CPU creates a data processing thread and a plurality of SSD threads, and one SSD thread is allocated to each SSD;
receiving a data write request by the data processing thread; the data writing request comprises a file to be written;
under the condition that the size of the file to be written reaches a file writing threshold, respectively judging whether the buffered data of the buffer area of each SSD meets the requirement of a buffer threshold; wherein the write file threshold is determined according to a unit sector size and number of the SSD;
if so, dividing the file to be written into a plurality of subfiles, and respectively writing the plurality of subfiles into the SSD corresponding to the SSD thread by using the plurality of SSD threads; wherein the number of subfiles is the same as the number of SSD threads.
Optionally, the buffer threshold comprises a buffer maximum depth; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold includes:
respectively judging whether the buffered data of the buffer area of each SSD is smaller than the maximum depth of the buffer area; if yes, dividing the file to be written into a plurality of subfiles;
and respectively storing the plurality of subfiles into buffer areas of the plurality of SSDs.
Optionally, the buffer threshold further comprises a buffer write threshold; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold further includes:
monitoring a buffer area of an SSD corresponding to the SSD threads through the plurality of SSD threads, and judging whether the buffered data of the buffer area of the SSD is larger than a buffer writing threshold value;
if yes, executing the SSD thread to write the subfiles into the SSD corresponding to the SSD thread by utilizing the SSD threads respectively.
A method of managing reading and writing of a plurality of SSDs in parallel, the method performed with a reactive recording device comprising a CPU and a memory card, the memory card comprising a plurality of SSDs, the method comprising:
the CPU creates a data processing thread and a plurality of SSD threads, and allocates one SSD thread for each SSD;
receiving a data read request through the data processing thread; the data reading request comprises a plurality of subfiles to be read in the SSD and target users, and the number of the subfiles to be read is the same as that of the SSD threads;
respectively judging whether the buffered data of the buffer area of each SSD meets the requirement of a buffer threshold value;
if so, reading the plurality of subfiles to be read in the SSD corresponding to the SSD threads by using the plurality of SSD threads, combining the plurality of subfiles to be read into a target read file, and sending the target read file to the target user.
Optionally, the buffer threshold comprises a buffer maximum depth; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold includes:
judging whether the buffered data of the buffer area of the SSD corresponding to the SSD thread is smaller than the maximum depth of the buffer area or not through a plurality of SSD threads; if yes, executing the SSD thread to read the sub files to be read in the SSD corresponding to the SSD thread by utilizing the SSD threads respectively;
and respectively storing the subfiles to be read corresponding to the SSD threads to buffer areas of a plurality of SSDs corresponding to the SSD threads by utilizing the SSD threads.
Optionally, the buffering threshold further comprises a buffering read threshold; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold further includes:
monitoring the buffer area of each SSD by using the data processing thread, and respectively judging whether the buffered data of the buffer area of each SSD is larger than a buffer reading threshold value;
if so, reading each sub file to be read from the buffer area of each SSD respectively, and combining the plurality of sub files to be read into a target read file.
Optionally, the number of SSDs is 5, and the write file threshold of the SSD is 5M.
Optionally, the maximum depth of the buffer area is 16M, and the buffered writing threshold is 0M.
Has the beneficial effects that:
(1) The method has the advantages that the multiple SSDs can be managed, the multiple SSDs can read and write different areas of the same file in parallel and synchronously, the parallel reading and writing speed of the multiple SSD disks is improved, data can be directly written in a physical address of the hard disk, the reading and writing speed of the SSD can reach the theoretical maximum reading and writing speed of the hard disk, the problem that the multiple SSDs cannot read and write the same file in parallel is solved, the reading and writing speed of the file is greatly improved, and the reading and writing requirements of various data formats can be met.
(2) Compared with the traditional method of continuously reading and writing the file by utilizing 1 thread, the method and the device have the advantages that the file is read and written in parallel based on the combination of 1 data processing thread and 5SSD threads, the file reading and writing time is shortened, and the file reading and writing speed is greatly improved.
(3) For 5 reactance recording devices with 2TB SSDs in parallel, the unit sector size of the SSD is set to be 1M, and the write file threshold value is determined to be 5M according to the unit sector size and the number of the SSD. The read-write speed of the SSD is kept optimal, the loss of the SSD is reduced, and the service life of the SSD is greatly prolonged. Therefore, when the size of the file to be written does not reach 5M, the file to be written is waited to be accumulated to 5M, and then the disk is started to be written in parallel by using 5SSD threads, that is, the size of the subfile allocated to each SSD is accumulated to 1M, and then the corresponding SSD thread is used to start writing the disk, so as to ensure that the read-write speed of the SSD is optimal, and if the size of the subfile is not 1M, the file is waited to be enough for 1M to rewrite the disk.
(4) During writing, when the file to be written reaches 5MB, judging whether the SIZE of the SSD buffer area is smaller than LOGIC _ BUFF _ SIZE through a data processing thread, and determining whether the file to be written is divided into a plurality of subfiles; and monitoring whether the size of the buffer area of the SSD is larger than 0 by utilizing an SSD thread, and determining whether the subfile is written into the SSD hard disk.
(5) During reading, judging whether the SIZE of the buffer area of the SSD is smaller than LOGIC _ BUFF _ SIZE through an SSD thread, determining whether the subfile to be read in the SSD hard disk is read, and storing the subfile to be read into the buffer area of the SSD; and monitoring whether the size of the buffer area of each SSD is larger than the size of the buffer area of each SSD by using a data processing thread, determining whether the subfiles to be read are read from the buffer area of the SSD, and combining the subfiles to be read into a target read file.
Drawings
Fig. 1 is a schematic diagram of a main flow of a method for managing reading and writing of multiple SSDs in parallel according to a first embodiment of the present invention.
Fig. 2 (a) is a schematic diagram of the hardware composition of a reactance recording apparatus according to an embodiment of the present invention.
Fig. 2 (b) is a schematic diagram of signal flow according to an embodiment of the present invention.
Fig. 3 is a schematic diagram illustrating a main flow of a method for determining buffered data in a buffer of an SSD according to a first embodiment of the invention.
FIG. 4 is a diagram illustrating a main flow of a method for managing reading and writing of multiple SSDs in parallel according to a second embodiment of the present invention.
Fig. 5 is a schematic diagram illustrating a main flow of a method for determining buffered data in a buffer of an SSD according to a second embodiment of the invention.
Detailed Description
A conventional HDD (Hard Disk Drive, abbreviated as mechanical Hard Disk, also called Hard Disk Drive) is composed of a high-speed data cache chip, a data transmission chip, and a main control chip, and uses a magnetic Disk (i.e. magnetic medium) as a data storage medium, and performs mechanical addressing by using a combination of a magnetic head and a motor during reading and writing. Due to the limited speed of the mechanical drive, the read-write performance of the HDD is poor, and particularly the random read-write capability of the HDD cannot be improved due to the mechanical characteristics.
The invention utilizes the SSD for storage, the SSD uses a Flash chip (or called Flash chip) as a storage medium, data reading and writing are addressed through the SSD controller, and mechanical operation is not needed, so that the random reading and writing capability is excellent, and the invention can be applied to the fields of military affairs, vehicle-mounted, industrial control, video monitoring, network terminals, electric power, medical treatment, aviation, navigation equipment and the like.
The invention is described in detail below by way of example with reference to the accompanying drawings.
The invention provides a method for managing reading and writing of a plurality of SSDs in parallel, as shown in FIG. 1, the method for managing reading and writing of a plurality of SSDs in parallel according to the first embodiment of the invention comprises the following steps:
In this embodiment of the present invention, the method for managing reading and writing of multiple SSDs in parallel according to the first embodiment of the present invention is a writing method for managing multiple SSDs in parallel. The method for managing reading and writing of multiple SSDs in parallel is implemented by using a reactance recording device, which is a data processing and storing device for writing and reading data such as optical fiber data, network data and the like, as shown in fig. 2 (a), the reactance recording device of the present invention includes a front panel, a rear panel and a back panel, wherein a power switch, a power indicator and a key-destroying button are installed on the front panel, and a plurality of optical fiber connectors (4 examples in the figure), a plurality of network connectors (2 examples in the figure), a power line, a FUSE (safety device) and a ground line are installed on the rear panel.
In the embodiment of the present invention, the backplane is an IO board and is mainly responsible for providing external interfaces, and includes 3 gigabit network data interfaces, 8 channels of optical fiber interfaces, a system disk interface (e.g., MSATA), and a debugging interface (for fault debugging), the backplane is installed with a main control board and a memory card, the external interfaces are connected to the main control board through the backplane, and after power is supplied, cross-linking and communication of signals of each interface of the main control board, the memory card, the front panel, and the rear panel can be achieved. The main control board is inserted into the backplane through an LRM (line replaceable module) connector, and the access and storage processing of data, including the read-write, storage, key destruction and other operations of the data such as optical fiber data, network data and the like, is controlled by a CPU. The storage board is a data storage medium and is used as a record carrier for storing data, and 5 blocks of 2TB SSD are mounted on the storage board, and the total capacity of the SSD is 10TB. The interface of the SSD on the storage board is an m.2 interface, and a SATA (Serial ATA) bus is used, and the storage board is connected to and communicates with the CPU on the main control board through the PCIE interface, so as to achieve the purpose that the main control board writes data into the storage card and the main control board reads data from the storage card, for example, the signal flow of the main control board writing data into the storage card is as shown in fig. 2 (b).
In the embodiment of the present invention, taking the processing system as a Linux system as an example, the present invention takes 5 2TB SSDs as storage media, and the CPU of the reactance recording device stores a file to be written of the sender to the SSD of the memory card.
In the embodiment of the invention, the CPU of the reactance recording device performs parallel management on 5 SSDs on the memory card, and controls the 5 SSDs to write files simultaneously so as to meet the storage bandwidth requirement of a technical protocol. Wherein:
after the power is on, before a data write-in request is processed, the CPU creates a data processing thread and a plurality of SSD threads, one SSD thread is respectively allocated to each SSD, and 5SSD threads run simultaneously to write in a file, so that the file write-in speed is greatly improved.
102, receiving a data writing request through the data processing thread; and the data writing request comprises a file to be written.
In the embodiment of the invention, the data processing thread is used as a thread of comprehensive management and is used for receiving data writing requests of all data sources and determining the file to be written corresponding to the data writing requests.
Further, the CPU controls the data processing thread to receive the data writing request and determines a file to be written corresponding to the data writing request.
103, respectively judging whether the buffered data in the buffer area of each SSD meets the requirement of a buffer threshold value or not under the condition that the size of the file to be written reaches the threshold value of the written file; wherein the write file threshold is determined according to a unit sector size and number of the SSD.
In the embodiment of the invention, a data processing thread judges the size of a file to be written, determines whether the size of the file to be written reaches a threshold value of the written file, and if so, executes to respectively judge whether the buffered data of a buffer area of each SSD meets the requirement of the buffer threshold value; if not, continuing to wait until the size of the file to be written is accumulated to the written file threshold value.
In the embodiment of the present invention, since the larger the read/write rate of the SSD is, the better the read/write rate is, the larger the read/write rate is, and the higher the loss of the SSD, the unit sector size of the SSD is set to be 1M, and accordingly, the write file threshold is determined according to the unit sector size and number of the SSD, that is, the write file threshold is 1mb × SSD number =1mb × 5=5M. For example, the data processing thread determines whether a file to be written reaches a written file threshold value by taking 5MB as a unit, and if so, judges whether buffered data in a buffer area of the SSD meets the buffer threshold value; if not, continuing to wait until the size of the file to be written is accumulated to 5MB.
In the embodiment of the present invention, the unit sector size of the SSD may be modified according to a practical application scenario of the present invention, for example, based on the capacity of the SSD.
In the embodiment of the invention, under the condition that the size of the file to be written reaches the threshold value of the written file, the buffered data of the buffer area of the SSD is judged, whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold value is determined, and the buffer threshold value comprises the maximum depth of the buffer area and the buffer writing threshold value. The data processing threads judge whether the buffered data of the buffer area of the SSD meet the maximum depth of the buffer area, and each SSD thread judges whether the buffered data of the buffer area of the SSD meet a buffer writing threshold.
104, if yes, dividing the file to be written into a plurality of subfiles, and respectively writing the plurality of subfiles into the SSD corresponding to the SSD thread by using the plurality of SSD threads; wherein the number of subfiles is the same as the number of SSD threads.
In the embodiment of the invention, under the condition that a data processing thread determines that the buffered data of a buffer area of an SSD meets the maximum depth requirement of the buffer area, a file to be written is divided into a plurality of subfiles and stored in the corresponding buffer area of the SSD; in the event that the SSD thread determines that the buffered data of the buffer of the SSD meets the buffered write threshold requirement, performing writing the plurality of subfiles to the SSD corresponding to the SSD thread using the plurality of SSD threads.
As shown in fig. 3, the method for determining buffered data in a buffer area of an SSD according to the first embodiment of the present invention includes the following steps:
In the embodiment of the present invention, the buffer structure of the write file is defined as follows:
// number of hard disks
#define DISK_COUNT (5)
// buffer maximum depth, or logical data buffer size
#define LOGIC_BUFF_SIZE (16)
// write File threshold, or read-write sector size (in bytes)
#define SINGLE_SECTOR_SIZE (UINT64)(5*1024*1024)
struct st_record_dataBuff
{
// head refers to the buffer head pointer, used for data writing
UINT32 head；
// tail refers to the buffer tail pointer, used for data write-out, computed separately for each SSD
UINT32 tail[DISK_COUNT]；
// size refers to the size of the buffered data in the buffer of the SSD, each SSD calculated separately
UINT32 size[DISK_COUNT]；
// dataBuff refers to a two-dimensional array for storing actual data
UINT8 dataBuff[LOGIC_BUFF_SIZE][SINGLE_SECTOR_SIZE]；
}；
In the embodiment of the present invention, the maximum depth of the buffer area is 16M, which means that the upper limit of the capacity/the upper limit of the rate of the buffer area of each SSD is 16M.
Further, the size of the maximum depth of the buffer is related to the CPU memory, and therefore, the size of the maximum depth of the buffer can be modified according to the actual application scenario of the present invention.
In the embodiment of the present invention, before receiving a data write request, initializing a buffer structure includes:
the head, tail for each buffer are set to 0, and the dataBuff array is emptied.
In the embodiment of the present invention, the step of respectively judging whether the buffered data in the buffer area of each SSD is smaller than the maximum depth of the buffer area by the data processing thread includes:
the data processing thread respectively judges whether the SIZE of the buffer area of each SSD is smaller than LOGIC _ BUFF _ SIZE, namely, the data processing thread respectively judges whether the SIZE of the buffer area of each SSD is smaller than 16M.
In the embodiment of the invention, under the condition that the buffered data of the buffer area of the SSD is smaller than the maximum depth of the buffer area, the data processing thread divides the file to be written into a plurality of subfiles. For example, the data processing thread divides a file to be written in units of 5MB into 5 subfiles of 1MB.
In the embodiment of the invention, the data processing thread respectively stores the plurality of subfiles into the buffer areas of the plurality of SSDs. For example, the data processing thread stores 5 subfiles of 1MB into 5SSD buffers, respectively. Specifically, the method comprises the following steps:
in a buffer area of each SSD, a data processing thread stores 1MB subfiles into dataBuff [ head ], and correspondingly, the size and the head of the buffer area are respectively added with 1; the value of the head is monitored and reset to 0 if the head equals LOGIC _ BUFF _ SIZE.
In the embodiment of the invention, each SSD thread is used for monitoring the buffered data of the corresponding SSD buffer area and judging whether the buffered data of the SSD buffer area is larger than the buffer writing threshold value or not. For example, the buffer write threshold is 0, the SSD thread monitors the corresponding SSD buffer to determine if the size of the SSD buffer is greater than 0.
In the embodiment of the invention, when the buffered data of the buffer area of the SSD is larger than the buffer writing threshold value, the subfile corresponding to the SSD buffer area is written into the corresponding SSD by utilizing the SSD thread. For example, when the size of the buffer area of the SSD is greater than 0, the SSD threads write 1MB of subfiles into the corresponding SSD, and correspondingly, 5SSD threads write 5MB of files to be written into each SSD. Specifically, the method comprises the following steps:
each SSD thread writes the subfiles in the dataBuff [ tail ] of the corresponding buffer area into an SSD disk, and accordingly, the size of the buffer area is reduced by 1, and the tail is increased by 1; the value of tail is monitored and if it equals LOGIC _ BUFF _ SIZE, it is reset to 0.
In the embodiment of the present invention, the buffer write threshold is 0, which indicates that the SSD thread writes the corresponding SSD as soon as the buffer is stored with data. The SSD uses 1M as a unit sector to ensure an optimal read/write speed, that is, the SSD reads and writes data in units of 1M, so that when the buffered write threshold is 0, the size of the SSD buffer is substantially equal to or greater than 0, which indicates that the SSD buffer at least includes 1M subfiles.
In the embodiment of the present invention, when the data write request stops being received, even if the file to be written does not satisfy the write file threshold (for example, 5M), the file to be written that does not satisfy the write file threshold is directly written to the disk by using the plurality of SSD threads.
In the embodiment of the invention, under the condition that the buffered data of the buffer area of each SSD is more than or equal to the maximum depth of the buffer area or the buffered data of the buffer area of the SSD is less than or equal to the buffer writing threshold, the judgment is carried out again after the time delay; in the delay waiting process, there may be situations that buffered data is read by a corresponding SSD thread, the buffered data is unchanged, and the buffer buffers the data. For example, the delayed waiting time is 10ms, and when the data processing thread determines that the size of the buffer area of each SSD is greater than or equal to 16M and the SSD thread monitors that the size of the buffer area of the corresponding SSD is less than or equal to 0, the process goes to step 301 to perform the judgment again after waiting for 10 ms.
The invention provides a method for managing reading and writing of a plurality of SSDs in parallel, as shown in FIG. 4, the method for managing reading and writing of a plurality of SSDs in parallel according to the second embodiment of the invention comprises the following steps:
In an embodiment of the present invention, a method for managing reading and writing of multiple SSDs in parallel according to a second embodiment of the present invention is a reading method for managing multiple SSDs in parallel. The method of managing multiple SSD reads and writes in parallel of the present invention is performed using a reactive recording device. Taking a processing system as a Linux system as an example, the invention takes 5 2TB SSDs as storage media, and a CPU of a reactance recording device acquires a plurality of subfiles to be read from the SSD of the memory card, combines the subfiles into a target read file, and sends the target read file to a target user.
In the embodiment of the invention, the CPU of the reactance recording device performs parallel management on 5 SSDs on the memory card, and controls the 5 SSDs to read files simultaneously so as to meet the storage bandwidth requirement of a technical protocol. Wherein:
after the power is on, before a data reading request is processed, the CPU creates a data processing thread and a plurality of SSD threads, one SSD thread is respectively allocated to each SSD, and 5SSD threads run simultaneously to read a file, so that the file reading speed is greatly improved.
In the embodiment of the present invention, the data processing thread is used as a thread of comprehensive management, and is configured to receive a data reading request, determine each subfile to be read corresponding to the data reading request, receive the subfile to be read sent by each SSD thread after each SSD thread reads the corresponding subfile to be read, combine the subfiles to be read into a target read file corresponding to the data reading request, and send the target read file to a reader.
In the embodiment of the present invention, the unit sector size of the SSD is also selected and set to be 1M in the reading method of the parallel management SSD of the present invention for the same reason as the writing method of the parallel management SSD of the embodiment of fig. 1.
In the embodiment of the invention, the buffered data of the buffer area of each SSD is judged, and whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold value is determined, wherein the buffer threshold value comprises the maximum depth of the buffer area and the buffer reading threshold value. Each SSD thread judges whether the buffered data of the buffer area of the SSD meets the maximum depth of the buffer area, and the data processing thread judges whether the buffered data of the buffer area of the SSD meets a buffer reading threshold value.
And 404, if yes, reading the plurality of subfiles to be read by using the plurality of SSD threads respectively, combining the plurality of subfiles to be read into a target read file, and sending the target read file to the target user.
In the embodiment of the invention, under the condition that each SSD thread determines that the buffered data of the buffer area of the corresponding SSD meets the maximum depth requirement of the buffer area, reading the subfile to be read in the corresponding SSD and storing the subfile to be read in the corresponding SSD into the buffer area of the corresponding SSD; and under the condition that the data processing thread determines that the buffered data of the buffer area of the SSD meets the requirement of a buffer reading threshold, acquiring each sub-file to be read from the buffer area of each SSD respectively, combining the sub-files to be read into a target reading file and sending the target reading file to a reading party (namely a target user).
As shown in fig. 5, the method for determining buffered data in a buffer area of an SSD according to the second embodiment of the present invention includes the following steps:
In the embodiment of the present invention, the definition of the buffer structure of the read file is the same as that of the buffer structure of the write file, that is, the member variables of the buffer structure of the read file and the buffer structure of the write file are the same.
In the embodiment of the present invention, the maximum buffer depth of the read method of the parallel management SSD of the present invention is 16M for the same reason as the write method of the parallel management SSD of the embodiment of fig. 1.
In the embodiment of the present invention, the determining, by each SSD thread, whether buffered data of a buffer of the SSD corresponding to the SSD thread is smaller than the maximum depth of the buffer includes:
each SSD thread determines whether the SIZE of the buffer of the SSD corresponding thereto is smaller than LOGIC _ BUFF _ SIZE, that is, each SSD thread determines whether the SIZE of the buffer of the SSD corresponding thereto is smaller than 16M.
In the embodiment of the invention, when the buffered data of the buffer area of the SSD is smaller than the maximum depth of the buffer area, the SSD thread reads the subfile to be read in the corresponding SSD. For example, each SSD thread reads 1MB of the subfile to be read in the SSD corresponding to the SSD thread, and correspondingly, 5SSD threads read 5MB of the subfile to be read.
In the embodiment of the invention, each SSD thread stores the subfile to be read corresponding to the SSD thread to the buffer area of the SSD corresponding to the SSD thread. For example, each SSD thread stores the read subfile to be read of 1MB into its corresponding buffer. Specifically, the method comprises the following steps:
in a buffer area of each SSD, an SSD thread stores 1MB of sub-files to be read into dataBuff [ tail ], and correspondingly, the size and the tail of the buffer area are respectively added with 1; the value of tail is monitored and if it equals LOGIC _ BUFF _ SIZE, it is reset to 0.
In the embodiment of the invention, the buffered data of the buffer area of each SSD is monitored by using the data processing thread, and whether the buffered data of the buffer area of the SSD is larger than the buffer reading threshold value is judged. For example, when the buffer read threshold is 0, the data processing thread monitors the buffer area of each SSD, and determines whether the size of the buffer area of each SSD is greater than 0.
In step 505, the data processing thread reads each subfile to be read from the buffer area of each SSD.
In the embodiment of the invention, under the condition that the buffered data of the buffer area of the SSD is larger than the buffer reading threshold value, the data processing thread reads each subfile to be read from the buffer area of each SSD respectively. For example, when the size of the buffer area of the SSD is greater than 0, the data processing thread reads each sub-file to be read from the buffer area of each SSD, and accordingly, the data processing thread reads 5 sub-files to be read of 1MB from the buffer area of 5 SSDs. Specifically, the method comprises the following steps:
reading the subfiles to be read in each dataBuff [ head ] by the data processing thread, and correspondingly subtracting 1 from the size and adding 1 to the head of the buffer; the value of the head is monitored and set to 0 if the head equals LOGIC _ BUFF _ SIZE.
In the embodiment of the present invention, the buffer read threshold is 0, which indicates that the buffer is read by the data processing thread as soon as the buffer is stored with data. The SSD uses 1M as a unit sector to ensure an optimal read/write speed, that is, the SSD reads and writes data in units of 1M, so that when the buffer read threshold is 0, as long as the size of the SSD buffer is greater than 0, the essence indicates that the buffer of the SSD includes at least 1M subfiles.
In the embodiment of the invention, 5 subfiles to be read of 1MB are combined into 1 target read file of 5MB by the data processing thread.
And step 507, time delay judgment, and turning to step 501.
In the embodiment of the invention, under the condition that the buffered data of the buffer area of each SSD is more than or equal to the maximum depth of the buffer area or the buffered data of the buffer area of the SSD is less than or equal to the buffer reading threshold value, the judgment is carried out again after the time delay; during the delay waiting process, there may be situations that the SSD thread suspends reading from the SSD hard disk, the SSD thread stores in the buffer, and the data processing thread suspends reading from the buffer. For example, the delayed waiting time is 10ms, and when the SSD thread determines that the size of the buffer area of each SSD is greater than or equal to 16M and the data processing thread monitors that the size of the buffer area of the corresponding SSD is less than or equal to 0, the process goes to step 501 to perform the judgment again after waiting for 10 ms.
In the embodiment of the present invention, or at the time of power-on initialization of the reactance recording device, 5 2TB SSDs on the memory card may be identified for easy distinction, and five discs may be organized into parallel discs of 1, 2, 3, 4, and 5. For example, the SSD-1, SSD-2, SSD-3, SSD-4, SSD-5, while the data (in 5 MB) is written, the SSD-1, SSD-2, SSD-3, SSD-4, SSD-5 are written simultaneously, as follows:
- > SSD-1 disk writes 1MB
- > SSD-2 disk write 1MB
- > SSD-3 disk writes 1MB
- > SSD-4 disk writes 1MB
SSD-5 writes 1MB.
When data (5 MB is taken as a unit) is read, SSD-1, SSD-2, SSD-3, SSD-4 and SSD-5 are read simultaneously as follows:
- > SSD-1 disk read 1MB
- > SSD-2 disk read 1MB
- > SSD-3 disk read 1MB
- > SSD-4 disk read 1MB
- > SSD-5 disk reads 1MB.
In summary, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A method of managing reading and writing of a plurality of SSDs in parallel, the method being performed using a reactive recording device, the reactive recording device including a CPU and a memory card, the memory card including a plurality of SSDs, the method comprising:
the CPU creates a data processing thread and a plurality of SSD threads, and allocates one SSD thread for each SSD;
receiving a data write request by the data processing thread; the data writing request comprises a file to be written;
under the condition that the size of the file to be written reaches a written file threshold value, respectively judging whether the buffered data of the buffer area of each SSD meets the requirement of a buffer threshold value; wherein the write file threshold is determined according to a unit sector size and number of the SSD;
if so, dividing the file to be written into a plurality of subfiles, and respectively writing the plurality of subfiles into the SSD corresponding to the SSD thread by using the plurality of SSD threads; wherein the number of subfiles is the same as the number of SSD threads.
2. The method of claim 1, wherein the buffer threshold comprises a buffer maximum depth; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold includes:
respectively judging whether the buffered data of the buffer area of each SSD is smaller than the maximum depth of the buffer area; if yes, dividing the file to be written into a plurality of subfiles;
and respectively storing the plurality of subfiles into buffer areas of the plurality of SSDs.
3. The method of claim 2, wherein the buffering threshold further comprises a buffering write threshold; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold further includes:
monitoring a buffer area of an SSD corresponding to the SSD threads through the plurality of SSD threads, and judging whether the buffered data of the buffer area of the SSD is larger than a buffer writing threshold value;
if so, executing the SSD thread to write the plurality of subfiles into the SSD corresponding to the SSD thread by using the plurality of SSD threads respectively.
4. A method of managing reading and writing of a plurality of SSDs in parallel, the method being performed using a reactive recording device, the reactive recording device including a CPU and a memory card, the memory card including a plurality of SSDs, the method comprising:
the CPU creates a data processing thread and a plurality of SSD threads, and one SSD thread is allocated to each SSD;
receiving a data read request through the data processing thread; the data reading request comprises a plurality of subfiles to be read in the SSD and target users, and the number of the subfiles to be read is the same as that of the SSD threads;
respectively judging whether the buffered data of the buffer area of each SSD meets the requirement of a buffer threshold value;
if so, reading the plurality of subfiles to be read in the SSD corresponding to the SSD threads by using the plurality of SSD threads, combining the plurality of subfiles to be read into a target read file, and sending the target read file to the target user.
5. The method of claim 4, wherein the buffer threshold comprises a buffer maximum depth; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold includes:
judging whether the buffered data of the buffer area of the SSD corresponding to the SSD thread is smaller than the maximum depth of the buffer area or not through a plurality of SSD threads; if so, executing the SSD thread to read the sub-files to be read in the SSD corresponding to the SSD thread by using the SSD threads respectively;
and respectively storing the subfiles to be read corresponding to the SSD threads to buffer areas of a plurality of SSDs corresponding to the SSD threads by utilizing the SSD threads.
6. The method of claim 5, wherein the buffering threshold further comprises a buffering read threshold; the determining whether the buffered data of the buffer area of each SSD meets the requirement of the buffer threshold further includes:
monitoring the buffer area of each SSD by using the data processing thread, and respectively judging whether the buffered data of the buffer area of each SSD is larger than a buffer reading threshold value;
if so, reading each sub file to be read from the buffer area of each SSD respectively, and combining the plurality of sub files to be read into a target read file.
7. The method of any of claims 1-3, wherein the number of SSDs is 5, and the write file threshold of the SSDs is 5M.
8. The method of any of claims 2-3, 5-6, wherein the buffer maximum depth is 16M and the buffered write threshold is 0M.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210930161.5A CN115454330A (en) | 2022-08-03 | 2022-08-03 | Method for managing multiple SSD reads and writes in parallel |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210930161.5A CN115454330A (en) | 2022-08-03 | 2022-08-03 | Method for managing multiple SSD reads and writes in parallel |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115454330A true CN115454330A (en) | 2022-12-09 |
Family
ID=84296303
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210930161.5A Pending CN115454330A (en) | 2022-08-03 | 2022-08-03 | Method for managing multiple SSD reads and writes in parallel |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115454330A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101498994A (en) * | 2009-02-16 | 2009-08-05 | 华中科技大学 | Solid state disk controller |
| CN107273304A (en) * | 2017-05-24 | 2017-10-20 | 记忆科技(深圳)有限公司 | A kind of method and solid state hard disc for improving solid state hard disc order reading performance |
| CN207833486U (en) * | 2018-02-26 | 2018-09-07 | 湖南国科微电子股份有限公司 | A kind of solid-state memory system based on FPGA |
| CN108701081A (en) * | 2016-01-11 | 2018-10-23 | 美光科技公司 | Device and method for the multiple subregions for accessing nonvolatile memory simultaneously |
| CN113111038A (en) * | 2021-03-31 | 2021-07-13 | 北京达佳互联信息技术有限公司 | File storage method, device, server and storage medium |
| CN113568572A (en) * | 2021-07-09 | 2021-10-29 | 浙江工业大学 | Database parallel sequencing connection system based on solid state disk |
-
2022
- 2022-08-03 CN CN202210930161.5A patent/CN115454330A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101498994A (en) * | 2009-02-16 | 2009-08-05 | 华中科技大学 | Solid state disk controller |
| CN108701081A (en) * | 2016-01-11 | 2018-10-23 | 美光科技公司 | Device and method for the multiple subregions for accessing nonvolatile memory simultaneously |
| CN107273304A (en) * | 2017-05-24 | 2017-10-20 | 记忆科技(深圳)有限公司 | A kind of method and solid state hard disc for improving solid state hard disc order reading performance |
| CN207833486U (en) * | 2018-02-26 | 2018-09-07 | 湖南国科微电子股份有限公司 | A kind of solid-state memory system based on FPGA |
| CN113111038A (en) * | 2021-03-31 | 2021-07-13 | 北京达佳互联信息技术有限公司 | File storage method, device, server and storage medium |
| CN113568572A (en) * | 2021-07-09 | 2021-10-29 | 浙江工业大学 | Database parallel sequencing connection system based on solid state disk |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9223508B2 (en) | Storage system | |
| US7730257B2 (en) | Method and computer program product to increase I/O write performance in a redundant array | |
| US7844794B2 (en) | Storage system with cache threshold control | |
| US7401252B2 (en) | Apparatus and method to convert data from a first sector format to a second sector format | |
| US8392648B2 (en) | Storage system having a plurality of flash packages | |
| US7076605B1 (en) | Method and apparatus for writing data to a storage device | |
| US7953940B2 (en) | Storage system and control method thereof | |
| US20120023292A1 (en) | Storage apparatus and storage control method for the same | |
| US20070266218A1 (en) | Storage system and storage control method for the same | |
| JP5638744B2 (en) | Command queue loading | |
| US20090282191A1 (en) | Operating Method for a Memory Subsystem and Devices for Executing the Operating Method | |
| US20090210618A1 (en) | Apparatus and method to allocate resources in a data storage library | |
| US20060140008A1 (en) | Storage apparatus | |
| US8615625B2 (en) | Disk array system | |
| US8769147B2 (en) | Method, apparatus and system to dynamically manage logical path resources | |
| CN109074226A (en) | Data de-duplication method, storage system and controller in a kind of storage system | |
| US20230153024A1 (en) | System and method for nand multi-plane and multi-die status signaling | |
| WO2008093961A1 (en) | System and method for processing read request | |
| CN104991737B (en) | A kind of hard disk implementation method based on storage card array architecture | |
| US20100235599A1 (en) | Access control device, storage system, and access control method | |
| KR101209916B1 (en) | Raid-based storage control board having fibre channel interface controller and driving method thereof | |
| US20080195832A1 (en) | Storage controller and storage system | |
| US9201598B2 (en) | Apparatus and method for sharing resources between storage devices | |
| US20060031601A1 (en) | Apparatus and method to convert data payloads from a first sector format to a second sector format | |
| US7984245B2 (en) | Storage system, storage subsystem and storage control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |