CN110658978B - Data processing method and device, electronic equipment and readable storage medium - Google Patents

Data processing method and device, electronic equipment and readable storage medium Download PDF

Info

Publication number
CN110658978B
CN110658978B CN201810692024.6A CN201810692024A CN110658978B CN 110658978 B CN110658978 B CN 110658978B CN 201810692024 A CN201810692024 A CN 201810692024A CN 110658978 B CN110658978 B CN 110658978B
Authority
CN
China
Prior art keywords
data
read
offset
page
size
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810692024.6A
Other languages
Chinese (zh)
Other versions
CN110658978A (en
Inventor
林鹏
林起芊
汪渭春
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision System Technology Co Ltd
Original Assignee
Hangzhou Hikvision System Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision System Technology Co Ltd filed Critical Hangzhou Hikvision System Technology Co Ltd
Priority to CN201810692024.6A priority Critical patent/CN110658978B/en
Publication of CN110658978A publication Critical patent/CN110658978A/en
Application granted granted Critical
Publication of CN110658978B publication Critical patent/CN110658978B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device

Abstract

The application provides a data processing method, a data processing device, an electronic device and a readable storage medium, wherein the method comprises the following steps: when a data writing request is received, determining a first starting offset and a first ending offset of current data writing according to the size of data to be written and the writing offset; when the first starting offset is not the starting position of the page, determining the starting position of the page where the first starting offset is located as the actual starting offset for writing the current data; combining the data in the page where the first initial offset is located and the data to be written into the size of an integral number of pages for writing; and when the first ending offset is not the ending position of the page, determining the ending position of the page where the first ending offset is located as the actual ending offset, and filling the remaining data to be written into the page until the size of the page is larger than the size of the page to be written into the page. The method can improve the data writing efficiency and reduce the writing punishment.

Description

Data processing method and device, electronic equipment and readable storage medium
Technical Field
The present application relates to data storage technologies, and in particular, to a data processing method and apparatus, an electronic device, and a readable storage medium.
Background
A file system is a method and data structure used by an operating system to formulate files on a storage node, i.e., a method of organizing files on a storage node.
SSD (Solid State Drives) are hard disks made using Solid State electronic memory chip arrays, and NAND FLASH is used as a storage medium.
The minimum unit of reading and writing data by the SSD is a page (page), and the minimum unit of erasing is a block (block), and one block may include a plurality of pages.
With the application and popularization of the SSD, a file system implemented based on the SSD becomes a popular research direction, but because the conventional file system cannot adapt to the page size of the SSD, data reading and writing cannot be performed according to the page size, resulting in more reading and writing penalties in the data writing process and poor data reading and writing efficiency.
Disclosure of Invention
In view of the above, the present application provides a data processing method, an apparatus, an electronic device and a readable storage medium.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of embodiments of the present application, a data processing method is provided, which is applied to a storage node, the storage node runs a file system implemented based on an SSD, the file system includes a start block, an index block, and a data block, the size of the start block is an integer multiple of the size of the block, the size of the index block is an integer multiple of a page, and logical addresses of the index block in the SSD are continuous, the size of an index block area formed by the index block is an integer multiple of the size of the block, the size of the data block is an integer multiple of the size of the block, the index block corresponds to the data block one to one, and the method includes:
when a data writing request is received, determining a first starting offset and a first ending offset of current data writing according to the size of data to be written and the writing offset;
when the first starting offset is not the starting position of the page, determining the starting position of the page where the first starting offset is located as the actual starting offset of the current data writing;
combining the data in the page where the first initial offset is located and the data to be written into the size of an integral number of pages for writing;
and when the first ending offset is not the ending position of the page, determining the ending position of the page where the first ending offset is located as the actual ending offset, and filling the remaining data to be written into the page until the size of the page is larger than the size of the page to be written into the page.
Optionally, before combining the data in the page where the first start offset is located and the data to be written into an integer size of the page for writing, the method further includes:
if the current data writing is the first writing, reading the data in the page where the initial offset is in a write cache;
after filling the remaining data to be written into the page to a size of the page and writing the data into the page, the method further includes:
and storing the data in at least the last page in the write cache.
Optionally, the method further includes:
when a data reading request is received, judging whether the reading offset of the data to be read is in a reading cache or not;
when the read offset is not in the read cache, determining a second starting offset and a second ending offset of the current data read according to the size of the data to be read and the read offset;
when the second starting offset is not the starting position of the page, determining the starting position of the page where the second starting offset is located as the actual starting offset of the current data reading;
when the second ending offset is not the ending position of the page, determining the ending position of the page where the second ending offset is located as the actual ending offset of the current data reading;
reading data between an actual starting offset and an actual ending offset of current data reading, returning data to be read to data reading request initiator equipment, and storing data in at least the last page in the read data in a read cache.
Optionally, the method further includes:
when the read offset is in the read cache, judging whether the data to be read is in the read cache completely;
when partial data in the data to be read is not in the read cache, determining the starting position of the next page of the partial data to be read in the read cache as the actual starting offset of the current data reading;
when the second ending offset is not the ending position of the page, determining the ending position of the page where the second ending offset is located as the actual ending offset of the current data reading;
reading data between an actual starting offset and an actual ending offset of current data reading, returning the part of data to be read and the read data to be read in the read cache to data reading request initiator equipment, and storing data in at least the last page in the read data in the read cache.
Optionally, the method further includes:
when a data reading request is received, judging whether the reading offset of the data to be read is in a reading cache or not;
when the read offset is not in the read cache, determining a second starting offset and a second ending offset of the current data read according to the size of the data to be read and the read offset;
according to the pre-reading configuration, the size of a reading cache and the size of data to be read, determining the start position of an Nth page before the page where the second start offset is located as the actual start offset of the current data reading, and determining the end position of an Mth page after the page where the second end offset is located as the actual end offset of the current data reading; wherein the size of M + N pages is smaller than or equal to the size of a read cache, and M is larger than or equal to 0,N and larger than or equal to 0;
reading data between an actual starting offset and an actual ending offset of current data reading, returning data to be read to data reading request initiator equipment, and storing data in at least the last page in the read data in a read cache.
Optionally, the method further includes:
when the read offset is in the read cache, judging whether the data to be read is in the read cache completely;
when partial data in the data to be read is not in the read cache, determining the starting position of the next page of the partial data to be read in the read cache as the actual starting offset of the current data reading;
determining the end position of the Kth page after the page where the second end offset is located as the actual end offset of the current data reading; the size of K pages is smaller than or equal to the size of the residual space in the read cache;
reading data between an actual starting offset and an actual ending offset of current data reading, returning the part of data to be read and the read data to be read in the read cache to data reading request initiator equipment, and storing data in at least the last page in the read data in the read cache.
Optionally, the method further includes:
releasing all space of the SSD when a formatting request is received; the formatting request carries formatting parameters, wherein the formatting parameters comprise the size of an index block and the size of a data block;
determining the total number of blocks according to the formatting parameters, the size of the page, the size of the block and the total capacity of the SSD; wherein the total number of blocks is the number of data blocks or index blocks;
and writing initialization information of the starting block and the index block into the designated position of the SSD according to the storage structure and the total number of the blocks.
Optionally, the file system includes a Trim operation detection module, and the method further includes:
when the Trim operation detection module is started for the first time, according to the data block use condition of the file system, releasing the unused space of the data block through Trim operation;
periodically detecting whether the SSD meets Trim operating conditions;
and if so, releasing the space of the unused data block through Trim operation according to the use condition of the data block of the file system.
According to the second aspect of the embodiment of the application, a data processing device is provided, which is applied to a storage node, the storage node runs a file system realized based on an SSD, the file system comprises a starting block, an index block and a data block, the size of the starting block is integral multiple of the size of the block, the size of the index block is integral multiple of a page, the logical address of the index block in the SSD is continuous, the size of an index block area formed by the index block is integral multiple of the size of the block, the size of the data block is integral multiple of the block, the index block corresponds to the data block one by one, and the device comprises:
a receiving unit configured to receive a data write request;
a determining unit, configured to determine, when the receiving unit receives a data write request, a first start offset and a first end offset of current data writing according to a size of data to be written and a write offset;
the determining unit is further configured to determine, when the first start offset is not the start position of the page, the start position of the page where the first start offset is located as an actual start offset for current data writing;
the data processing unit is used for combining the data in the page where the first starting offset is located and the data to be written into the size of an integral number of pages for writing;
the determining unit is further configured to determine, as an actual ending offset, an ending position of a page where the first ending offset is located when the first ending offset is not the ending position of the page;
the data processing unit is further configured to fill the remaining data to be written into a page to the size of the page and write the data into the page.
Optionally, the data processing unit is further configured to, if the current data writing is the first writing, read the data in the page where the start offset is located into a write cache;
the data processing unit is further configured to fill the remaining data to be written to a size of one page, and store the data in at least the last page in the write cache after the page is written.
Optionally, the data processing unit is further configured to determine whether a read offset of the data to be read is in the read cache when the receiving unit receives the data read request;
the determining unit is further configured to determine a second starting offset and a second ending offset of current data reading according to the size of the data to be read and the read offset when the read offset is not in the read cache;
the determining unit is further configured to determine, when the second start offset is not the start position of the page, the start position of the page where the second start offset is located as an actual start offset for reading the current data;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
Optionally, the data processing unit is further configured to determine whether the data to be read is completely in the read cache when the read offset is in the read cache;
the determining unit is further configured to determine, when part of the data to be read is not in the read cache, a start position of a next page of the part of the data to be read in the read cache as an actual start offset of the current data reading;
the determining unit is further configured to determine, when the second ending offset is not the ending position of the page, the ending position of the page where the second ending offset is located as an actual ending offset of current data reading;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return the part of data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
Optionally, the data processing unit is further configured to determine whether a read offset of the data to be read is in the read cache when the receiving unit receives the data read request;
the determining unit is further configured to determine a second starting offset and a second ending offset of the current data reading according to the size of the data to be read and the read offset when the read offset is not in the read cache;
the determining unit is further configured to determine, according to a pre-reading configuration, a size of a read cache, and a size of data to be read, a start position of an nth page before a page where the second start offset is located as an actual start offset of current data reading, and determine an end position of an mth page after the page where the second end offset is located as an actual end offset of current data reading; wherein the size of M + N pages is smaller than or equal to the size of a read cache, and M is larger than or equal to 0,N and larger than or equal to 0;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
Optionally, the data processing unit is further configured to determine whether the data to be read is all in the read cache when the read offset is in the read cache;
the determining unit is further configured to determine, when part of the data to be read is not in the read cache, a start position of a next page of the part of the data to be read in the read cache as an actual start offset of the current data reading;
the determining unit is further configured to determine an end position of a kth page after the page where the second end offset is located as an actual end offset of current data reading; the size of K pages is smaller than or equal to the size of the residual space in the read cache;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return the part of data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
Optionally, the data processing unit is further configured to release all the space of the SSD when the receiving unit receives the format request; the formatting request carries formatting parameters, wherein the formatting parameters comprise the size of an index block and the size of a data block;
the data processing unit is further configured to determine the total number of blocks according to the formatting parameter, the size of the page, the size of the block, and the total capacity of the SSD; wherein the total number of blocks is the number of data blocks or index blocks;
and the data processing unit is also used for writing initialization information of the starting block and the index block into the designated position of the SSD according to the storage structure and the total number of the blocks.
Optionally, the file system includes a Trim operation detection module;
the data processing unit is further configured to trigger the Trim operation detection module when the Trim operation detection module is started for the first time, and release the unused space of the data block through the Trim operation according to the use condition of the data block of the file system;
the data processing unit is also used for periodically detecting whether the SSD meets Trim operating conditions;
and the data processing unit is also used for triggering the Trim operation detection module to release the unused space of the data block through Trim operation according to the using condition of the data block of the file system if the data processing unit meets the requirement.
According to a third aspect of the embodiments of the present application, there is provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the data processing method when executing the program stored in the memory.
According to a fourth aspect of embodiments of the present application, there is provided a computer-readable storage medium having stored therein a computer program which, when executed by a processor, implements the above-described data processing method.
According to the data processing method, the sizes of the starting block, the index block and the data block in the storage structure of the file system based on the SSD are matched with the sizes of the page and the block of the SSD, when a data writing request is received, data are written in according to a page alignment mode, the data writing efficiency is improved, and the writing punishment is reduced.
Drawings
FIG. 1 is a schematic diagram illustrating a storage structure of a file system implemented based on an SSD according to an exemplary embodiment of the present application;
FIG. 2 is a flow chart illustrating a method of data processing according to an exemplary embodiment of the present application;
FIG. 3 is a flow chart illustrating one SSD formatting in accordance with an exemplary embodiment of the present application;
FIG. 4 is a diagram illustrating data read and write offsets in accordance with an exemplary embodiment of the present application;
FIG. 5 is a flow chart illustrating a method of writing data according to an exemplary embodiment of the present application;
FIG. 6 is a flow chart illustrating a method of data reading in accordance with an exemplary embodiment of the present application;
FIG. 7 is a block diagram of a data processing apparatus according to an exemplary embodiment of the present application;
fig. 8 is a schematic diagram illustrating a hardware structure of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
In order to make those skilled in the art better understand the technical solution provided by the embodiment of the present application, a simple description is first given below to a storage structure of a file system implemented based on an SSD, which is provided by the embodiment of the present application.
Referring to fig. 1, a schematic view of a storage structure of a file system implemented based on an SSD according to an embodiment of the present application is shown in fig. 1, where the file system includes a start block, an index block, and a data block. Wherein:
the system comprises a starting block, a checking block and a storage block, wherein the starting block is used for storing key information of a file system, and the key information comprises but is not limited to UUID (Universal Unique Identifier) (used for uniquely identifying SSD), index block size, data block number, checking information and the like; the size of the start block is an integer multiple of the size of the block.
Preferably, the size of the start block is one block size.
In order to improve the information reliability, the starting block comprises a main starting block and a standby starting block which have the same content and size.
An index block for recording key information written into a corresponding data block, including but not limited to data size (i.e. size of user data (non-padding data) stored in the data block), data type, time, check information, etc.; the size of the index block is an integer multiple of the size of the page.
In order to improve the reliability of information, the index blocks comprise main index blocks and standby index blocks with the same content and size. The index blocks are continuous on the logic address of the SSD, all the index blocks form an index block area (comprising a main index block area and a standby index block area), and the size of the index block area is integral multiple of the size of the block.
And the data blocks are used for storing user data, the size of each data block is integral multiple of the size of each block, and all the data blocks form a data block area.
The data blocks are in one-to-one correspondence with the index blocks (one data block corresponds to one main index block and one standby index block), and the index blocks and the corresponding data blocks form a logical block available for users.
In the embodiment of the present application, data is written in an additional write mode in order to avoid space waste and disk fragmentation.
In order to make the aforementioned objects, features and advantages of the embodiments of the present application more comprehensible, embodiments of the present application are described in detail below with reference to the accompanying drawings.
Referring to fig. 2, a schematic flow chart of a data processing method provided in an embodiment of the present application is shown, where the data processing method may be applied to a storage node, where the storage node runs a file system implemented based on an SSD (for example, the file system implemented based on the SSD shown in fig. 1), and as shown in fig. 2, the data processing method may include the following steps:
step S200, when a data writing request is received, determining a first starting offset and a first ending offset of current data writing according to the size of data to be written and the writing offset.
In the embodiment of the present application, when a storage node receives a data write request, the storage node may determine a start offset (referred to as a first start offset herein) and an end offset (referred to as a first end offset herein) of current data write according to a size of data to be written and a write offset (a value of offset carried in the data write request).
The first starting offset is a writing offset of the data to be written, and the first ending offset is the sum of the writing offset and the size of the data to be written.
Step S210, when the first start offset is not the start position of the page, determining the start position of the page where the first start offset is located as the actual start offset for writing the current data.
In this embodiment of the present application, after determining a first start offset and a first end offset of current data writing, a storage node may respectively determine whether the first start offset and the first end offset are start positions of a page.
In the embodiment of the present application, the starting position of one page (not the first page) is the ending position of the previous page.
In the embodiment of the present application, it is considered that the minimum unit for data writing of the SSD is a page, and a start offset (such as the first start offset) for data writing of the file system is usually not at a start position of the page, and data writing cannot be performed in a page alignment manner, so that data writing efficiency is low.
Accordingly, in this embodiment of the present application, when the storage node determines that the first start offset is not the start position of the page, the storage node may determine the start position of the page where the first start offset is located as the actual start offset of the current data write.
Step S220, combining the data in the page where the first starting offset is located and the data to be written into the size of an integral number of pages for writing.
In this embodiment of the application, after determining the actual start offset, the storage node may merge data (user data, non-padding data) in the page where the first start offset is located and data to be written, split the data according to the size of the page, split the data into one or more blocks of the size of the page, and sequentially write the data in the page where the first start offset is located and the subsequent pages (if the data include data of the sizes of multiple pages).
Step S230, when the first ending offset is not the ending position of the page, determining the ending position of the page where the first ending offset is located as the actual ending offset, and filling the remaining data to be written into the page to the size of one page and writing the data into the page.
In this embodiment of the application, when the first ending offset is not the ending position of the page, after the data writing is performed in step S220, there is remaining data to be written in a size smaller than one page, at this time, the storage node may determine the ending position of the page where the first ending offset is located as the actual ending offset, and fill the remaining data to be written in the page where the page size is written in the first ending offset by a data filling method.
It can be seen that, in the method flow shown in fig. 2, for a file system implemented based on an SSD, matching optimization between a start block, an index block, and a data block and a page in the SSD is performed, when data is written, data is written in a page alignment manner, so that efficiency of data writing is improved.
Furthermore, in the embodiment of the present application, it is considered that data writing is performed by an additional writing manner, and therefore, when an ending offset of one data writing is not an ending position of a page, data (user data, non-padding data) in the page where the ending offset is located usually needs to be read into a write cache at the next data writing, and is written in combination with data to be written, and therefore, in order to further improve data writing efficiency and reduce writing penalty, after each data writing, data in at least the last page where the data is written may be stored in the write cache.
Correspondingly, in one embodiment of the present application, before combining the data in the page where the first start offset is located and the data to be written into the size of an integer number of pages for writing, the method may further include:
if the current data writing is the first writing, opening a block to be written, and reading the data in the page where the initial offset is located into a write cache;
after filling the remaining data to be written into the page to a size of the page and writing the data into the page, the method may further include:
and storing the data in at least the last page in the write cache.
In this embodiment, in order to further improve the efficiency of additional writing and reduce the write penalty, after each data write, the storage node may store the data in at least the last page to which the data write this time is performed in the write cache.
For example, for one-time data writing, assuming that the pages to which data is written are pages 1 to 4, and page4 only includes 4K data (user data, non-padding data), and the remaining 4K are padding data (taking page size as 8K as an example), after the storage node completes this data writing, at least 4K data included in page4 may be saved in the write cache (data in pages 3 and 4, or data in pages 2, 3, and 4, etc., and data in the last page is saved in the following as an example). Because the data is written in by means of additional writing, the write offset of the next continuous data write is usually the end position of the user data of the last data write (in this example, the position of 4K of the page 4), and because the data in the page where the write offset of the data write is already stored in the cache, the storage node may not need to read the data any more, but may directly merge the data in the page stored in the cache and the data to be written into an integer size of the page for writing, so that the data writing efficiency is improved, and the write penalty is reduced.
Accordingly, in this embodiment, when data writing is required and it is determined that the first start offset is not at the start position of a page, the storage node may determine whether data in the page where the first start offset is located is stored in the write cache, if so, merge the data in the page stored in the cache and the data to be written into an integer size of the page for writing, and store the data in the page where the first end offset is located in the cache when the first end offset is not at the end position of the page.
Through the mode, except that the data of the page where the first initial offset is located needs to be read into the write cache when the data is written in for the first time (under the condition that the first initial offset is not the initial position of the page), the data can not be read when the data is written in every time, the data writing efficiency is further improved, and the writing punishment is reduced.
It should be noted that, in the embodiment of the present application, when the first start offset is the start position of the page, the storage node may directly perform data writing (data reading is not required); when the first end offset is the end position of the page, the storage node may directly perform data writing (data padding is not required), and in this case, the storage node may not store the data of the page in which the first end offset is located in the cache.
Further, in the embodiment of the present application, in order to improve the efficiency of data reading, data reading may also be performed in a page alignment manner.
Correspondingly, the data processing method may further include:
when a data reading request is received, judging whether the reading offset of the data to be read is in a reading cache or not;
when the read offset is not in the read cache, determining a second starting offset and a second ending offset of the current data read according to the size of the data to be read and the read offset;
when the second starting offset is not the starting position of the page, determining the starting position of the page where the second starting offset is located as the actual starting offset of the current data reading;
when the second ending offset is not the ending position of the page, determining the ending position of the page where the second ending offset is located as the actual ending offset of the current data reading;
reading data between an actual starting offset and an actual ending offset of current data reading, returning data to be read to data reading request initiator equipment, and storing data in at least the last page in the read data in a read cache.
In this embodiment, when the storage node receives a data read request, a start offset (referred to as a second start offset herein) and an end offset (referred to as a second end offset herein) of a current data read may be determined according to a size of data to be read and a read offset carried in the data read request.
The second starting offset is a reading offset of the data to be read, and the second ending offset is a sum of the reading offset and the size of the data to be read.
In this embodiment, after the storage node determines the second starting offset for the data read, it may determine whether the second starting offset is within the read cache. If not, the storage node may further determine whether the second start offset is the start position of the page.
In this embodiment, it is considered that the minimum unit of data reading of the SSD is a page, and a start offset (such as the second start offset) of data reading of the file system is usually not at the start position of the page, so that data reading cannot be performed in a page-aligned manner, which results in low data reading efficiency.
Accordingly, in this embodiment, when the storage node determines that the second start offset is not the start position of the page, the storage node may determine the start position of the page at which the first start offset is located as the actual start offset of the current data read.
Further, the storage node may determine whether the second ending offset is at the ending position of the page, and if not, the storage node may determine the ending position of the page where the second ending offset is located as the actual ending offset of the current data reading.
After determining the actual start offset and the actual end offset of the current data reading, the storage node may read data between the actual start offset and the actual end offset, return the data to be read included in the data to the data reading request initiator device, and store data in at least the last page of the read data in the read cache (the number of cached pages may be determined according to the size of the remaining space of the read cache).
Furthermore, when continuous data reading is carried out, the storage node can read all or part of data from the read cache, so that when the initial offset of the data to be read is not at the initial position of the page, the data to be read in the page where the initial offset is located can be directly read from the read cache without reading the data of the whole page from a disk, and the reading punishment is reduced; in addition, when data reading from the disk needs to be performed again, the reading offset can be ensured to be at the starting position of the page, and the data reading efficiency can be further improved.
Further, in an implementation manner of this embodiment, when the storage node determines that the read offset is in the read cache, it is determined whether the data to be read is all in the read cache.
When the storage node determines that part of data in the data to be read is not read in the cache any more, determining the start position of the next page of the part of data to be read in the cache as the actual start offset of the current data reading.
For example, assuming that the read offset of the current data read is in the middle of page4, and data after the middle of page4 exists in the read cache, the storage node may determine the start position of page5 as the actual start offset of the current data read.
In this embodiment, when the second end offset is not the end position of the page, the storage node may determine the end position of the page where the second end offset is located as the actual end offset of the current data reading.
Furthermore, the storage node may read data between an actual start offset and an actual end offset of current data reading, return the part of the data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
It should be noted that, in this embodiment of the present application, when all the data to be read is in the read cache, the storage node may read the data to be read from the read cache and return the data to the data read request initiator device.
Further, in this embodiment of the present application, in order to further improve data reading efficiency, a pre-read configuration may be further performed when data reading is performed (that is, data of a certain number of pages before the data to be read is read or/and data of a certain number of pages after the data to be read is stored in the read cache), at this time, when the storage node receives the data read request and determines that the data to be read is not all in the read cache, the number of pages that are read forward or/and backward may also be determined according to the pre-read configuration, the size of the data to be read, and the size of the cache, and a specific implementation thereof will be described below with reference to a specific example.
Further, in the embodiment of the present application, it is considered that only the index is usually deleted when the existing file system deletes data, and the space corresponding to the data block cannot be really released, so that IO (Input/output) processing efficiency of the SSD in the file system implemented based on the SSD is affected.
Accordingly, in one embodiment of the present application, a file system implemented based on an SSD may run a Trim operation detection module,
correspondingly, the data processing method may further include:
when the Trim operation detection module is started for the first time, according to the use condition of the data blocks of the file system, releasing the space of the unused data blocks through Trim operation;
regularly detecting whether the SSD meets Trim operating conditions;
and if so, releasing the space of the unused data block through Trim operation according to the use condition of the data block of the file system.
In this embodiment, the Trim operation detection module may be run in a file system implemented based on an SSD. When the Trim operation detection module is started for the first time (for example, when the storage node is started to automatically start the Trim operation detection module), the space of the data block which is not used by the file system in the SSD can be released through the Trim operation according to the use condition of the data block of the file system.
After the operations are completed, the storage node can periodically detect whether the SSD meets the Trim operation conditions through the Trim operation detection module.
The Trim operation condition may include, but is not limited to, that a valid data occupancy of the SSD (a ratio of a data amount in a data block used by the file system to a total data amount in the SSD) is lower than a preset threshold, a space usage rate of the SSD exceeds a preset threshold, and the like.
In this embodiment, when the Trim operation detection module detects that the SSD satisfies the Trim operation condition, the Trim operation detection module may release, according to the data block usage of the file system, a space of a data block in the SSD that is not used by the file system through the Trim operation.
In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.
In this embodiment, when the storage node running the file system implemented based on the SSD is initially run, the SSD may be formatted.
It should be noted that the formatting of the SSD by the storage node is not limited to the initial operation, and may also be triggered by a user according to an actual requirement during the operation of the storage node.
Referring to fig. 3, the implementation of the SSD formatting by the storage node may include the following steps:
step S300, when a formatting request is received, releasing all idleness of the SSD; the format request carries format parameters, and the format parameters include the size of the index block and the size of the data block.
In this embodiment, a user may issue a format request for an SSD to a storage node according to actual requirements, where the format request may carry at least a format parameter, and the format parameter includes, but is not limited to, an index block size (integer multiple of a page size) and a data block size (integer multiple of a block size).
It should be noted that the format parameter may also include a starting block size (an integer multiple of the size of the block); when the starting block size is not included in the formatting parameters, the starting block size may be determined to be a default size (i.e., the size of one block).
Step S310, determining the total number of blocks according to the formatting parameters, the size of the page, the size of the block and the total capacity of the SSD; wherein the total number of blocks is the number of data blocks or index blocks.
In this embodiment, the storage node may determine the size of the index block, the size of the data block, and the size of the start block according to the formatting parameter, and further, the storage node may determine the total number of blocks according to the size of the index block, the size of the data block, the size of the start block, the size of the page, the size of the block, and the total capacity of the SSD.
In this embodiment, the number of data chunks (i.e., the data of the index chunk) is taken as the total number of chunks.
And step S320, writing initialization information of the starting block and the index block into the designated position of the SSD according to the storage structure and the total number of the blocks.
In this embodiment, after the storage node determines the total number of the blocks, the storage node may determine the locations of the start block, the index block, and the data block in the SSD according to the storage structure and the total number of the blocks, and write corresponding initialization information into the start block and the index block, respectively, to complete formatting of the SSD.
In this embodiment, after the SSD formatting is completed, data reading and writing can be performed.
The data writing and the data reading are respectively explained below with reference to examples.
Referring to fig. 4, assuming that the first start offset and the first end offset determined by the storage node according to the received data write request are start offset1 and end offset1, respectively, a specific flowchart of the data write performed by the storage node may be as shown in fig. 5, which may include the following steps:
step S500, determine whether the data writing is the first data writing. If yes, go to step S510; otherwise, go to step S520.
Step S510, opening the block, reading the data in the page2, putting the data into a cache, and going to step S540.
In this embodiment, the storage node may first determine whether the data write is the first data write.
If so, the storage node may open a block (the block hit by the write request), read the data in the page (i.e., page 2) where start offset1 is located, and store the data in the write cache.
Step S520, judging whether data in page2 exists in the write cache. If yes, go to step S540; otherwise, go to step S530.
Step S530, reading the data in the page2, putting the data into a cache, and turning to step S540.
Step S540, determining the starting position of the page2 as the actual starting offset of the current data writing, and merging the data in the page2 and the data to be written into the size of an integer number of pages for writing.
In this embodiment, the storage node may determine the start position of page2 (i.e., the position of arrow 2) as the actual start offset of the current data writing, and merge the data in page2 and the data to be written into an integer number (2 in this embodiment) for writing (writing to page2 and page 3).
Step S550, determine the end position of the page4 as the actual end offset of the current data writing, and fill the remaining data to the size of one page and write the data into the page4.
In this embodiment, since the end offset1 is not at the end position of the page, a part of data remains after the data writing in step S540, and at this time, the storage node may determine the end position of the page4 (i.e. the position of the arrow 5) as the actual end offset of the current data writing, and fill the remaining data to the size of one page (fill by complementing 0) and write the remaining data into the page4.
And step S560, storing the data in the page4 in a write cache.
In this embodiment, after the storage node completes the data writing, the data in the page4 may be stored in the write cache, and when the data is written next time and the data writing offset is at the end offset1 position, the data in the page4 may be directly obtained from the write cache without reading the data from the page4, and the data is merged with the data to be written and written, so that the data writing efficiency is improved, and the writing penalty is reduced.
Still taking fig. 4 as an example, assuming that the second start offset and the second end offset determined by the storage node according to the received data read request are start offset1 and end offset1, respectively, a specific flowchart of the data read by the storage node may be as shown in fig. 6, which may include the following steps:
step S600, determine whether the data reading is the first data reading. If yes, the block is opened and the process goes to step S640; otherwise, go to step S610.
Step S610, judging whether the data of the page2 is in the read cache. If yes, go to step S620; otherwise, go to step S640.
In this embodiment, when the storage node determines that the current data read is not the first data read, the storage node may determine whether the data (data following start offset1 in page 2) of the page (i.e., page 2) in which the read offset (i.e., start offset 1) is located is in the read cache.
Step S620, judging whether all the data of the pages 2 to 4 are in the read cache; if yes, go to step S670; otherwise, go to step S630.
In this embodiment, when the storage node determines that the data of page2 is in the read cache, the storage node may further determine whether the data of page3 and page4 are both in the read cache. If yes, directly obtaining data to be read from the read cache; otherwise, the part of the data to be read which is not in the read cache needs to be read from the SSD.
Step S630, determine the start position of the next page of the part of data to be read in the read cache as the actual start offset of the current data, and determine the end position of the kth page after the page where the end offset1 is located as the actual end offset of the current data reading.
Step S640 determines the start position of the nth page before page2 as the actual start position of the current data reading, and determines the end position of the mth page after the page where end offset1 is located as the actual end offset of the current data reading.
In this embodiment, the storage node may determine the values of M, N and K according to the pre-read configuration, the cache size, and the size of the data to be read; wherein M, N and K are both natural numbers.
For example, assuming that the read offset is no longer read into the cache, and the read-ahead configuration is to read-ahead one page backward, the storage node may determine the start position of page1 (i.e. the position of arrow 1) as the actual start offset (start offset 2) of the current data read, and determine the end position of page5 (i.e. the position of arrow 6) as the actual end offset (end offset 2) of the current data read (assuming that the read cache space is sufficient).
And step S650, reading data according to the actual starting offset and the actual ending offset.
Step S660, returning the data to be read to the data read request initiator device, and storing the data in at least the last page of the read data in the read cache.
In this embodiment, when a part of the data to be read is in the read cache, the storage node may return the part of the data in the read cache and the remaining data to be read, which is read from the SSD, to the data read request initiator device.
After the storage node returns the data to be read to the data read request initiator device, the data in at least the last page in the read data can be stored in the read cache, so that when the data is continuously read, the data of the non-whole page in the page where the read offset is located can be directly read from the read cache during the next data reading, the data reading efficiency is improved, and the reading punishment is reduced.
Step S670, obtaining the data to be read from the read cache, and returning the data to the data read request initiator device.
In this embodiment, when all the data to be read is in the read cache, the storage node may directly obtain the data to be read from the read cache and return the data to the data read request initiator device, so as to improve data reading efficiency and reduce read penalty.
In the embodiment of the application, the sizes of the starting block, the index block and the data block in the storage structure of the file system based on the SSD are matched with the sizes of the page and the block of the SSD, and when a data writing request is received, data is written in according to a page alignment mode, so that the data writing efficiency is improved, and the writing punishment is reduced.
The methods provided herein are described above. The following describes the apparatus provided in the present application:
referring to fig. 7, a schematic structural diagram of a data processing apparatus provided in an embodiment of the present application is shown, where the data processing apparatus may be applied to a storage node in the foregoing method embodiment, and as shown in fig. 7, the data processing apparatus may include:
a receiving unit 710 for receiving a data write request;
a determining unit 720, configured to determine, when the receiving unit 710 receives a data writing request, a first starting offset and a first ending offset of current data writing according to a size of data to be written and a writing offset;
the determining unit 720 is further configured to determine, when the first start offset is not the start position of the page, the start position of the page where the first start offset is located as an actual start offset for current data writing;
the data processing unit 730 is configured to combine the data in the page where the first start offset is located and the data to be written into an integral number of pages for writing;
the determining unit 720 is further configured to determine, when the first ending offset is not the ending position of a page, the ending position of the page where the first ending offset is located as an actual ending offset;
the data processing unit 730 is further configured to fill the remaining data to be written into a page to the size of the page, and write the data into the page.
In an optional implementation manner, the data processing unit 730 is further configured to, if the current data writing is the first writing, read the data in the page where the start offset is located into a write cache;
the data processing unit 730 is further configured to fill the remaining data to be written to a size of a page, and store the data in at least the last page in the write cache after the data is written into the page.
In an optional implementation manner, the data processing unit 730 is further configured to determine, when the receiving unit 710 receives a data read request, whether a read offset of data to be read is in a read cache;
the determining unit 720 is further configured to determine a second starting offset and a second ending offset of the current data reading according to the size of the data to be read and the read offset when the read offset is not in the read cache;
the determining unit 720 is further configured to determine, when the second start offset is not the start position of the page, the start position of the page where the second start offset is located as an actual start offset for reading the current data;
the data processing unit 730 is further configured to read data between an actual start offset and an actual end offset of current data reading, return data to be read to the data reading request initiator apparatus, and store data in at least the last page in the read data in the read cache.
In an optional implementation manner, the data processing unit 730 is further configured to determine whether all data to be read is in the read cache when the read offset is in the read cache;
the determining unit 720 is further configured to determine, when part of data in the data to be read is not in the read cache, a start position of a next page of the part of data to be read in the read cache as an actual start offset of the current data reading;
the determining unit 720 is further configured to determine, when the second ending offset is not the ending position of the page, the ending position of the page where the second ending offset is located as an actual ending offset of current data reading;
the data processing unit 730 is further configured to read data between an actual start offset and an actual end offset of current data reading, return the part of data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
In an optional implementation manner, the data processing unit 730 is further configured to determine, when the receiving unit 710 receives a data read request, whether a read offset of data to be read is in a read cache;
the determining unit 720 is further configured to determine a second starting offset and a second ending offset of the current data reading according to the size of the data to be read and the read offset when the read offset is not in the read cache;
the determining unit 720 is further configured to determine, according to a pre-reading configuration, a size of a read cache, and a size of data to be read, a start position of an nth page before a page where the second start offset is located as an actual start offset of current data reading, and determine an end position of an mth page after the page where the second end offset is located as an actual end offset of current data reading; wherein the size of M + N pages is smaller than or equal to the size of a read cache, and M is larger than or equal to 0,N and larger than or equal to 0;
the data processing unit 730 is further configured to read data between an actual start offset and an actual end offset of current data reading, return data to be read to the data reading request initiator apparatus, and store data in at least the last page in the read data in the read cache.
In an optional implementation manner, the data processing unit 730 is further configured to determine, when the read offset is in a read cache, whether all data to be read is in the read cache;
the determining unit 720 is further configured to determine, when part of the data to be read is not in the read cache, a start position of a next page of the part of the data to be read in the read cache as an actual start offset of the current data reading;
the determining unit 720 is further configured to determine an end position of a kth page after the page where the second end offset is located as an actual end offset of current data reading; the size of K pages is smaller than or equal to the size of the residual space in the read cache;
the data processing unit 730 is further configured to read data between an actual start offset and an actual end offset of current data reading, return the part of data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
In an optional embodiment, the data processing unit 730 is further configured to release all the space of the SSD when the receiving unit 710 receives the format request; the formatting request carries formatting parameters, wherein the formatting parameters comprise the size of an index block and the size of a data block;
the data processing unit 730 is further configured to determine the total number of blocks according to the formatting parameter, the size of the page, the size of the block, and the total capacity of the SSD; wherein the total number of blocks is the number of data blocks or index blocks;
the data processing unit 730 is further configured to write initialization information of the start block and the index block to the specified location of the SSD according to the storage structure and the total number of the blocks.
In an optional implementation manner, the file system runs a Trim operation detection module;
the data processing unit 730 is further configured to trigger the Trim operation detection module when the Trim operation detection module is started for the first time, and release the unused space of the data block through the Trim operation according to the use condition of the data block of the file system;
the data processing unit 730 is further configured to periodically detect whether the SSD satisfies a Trim operation condition;
the data processing unit 730 is further configured to, if the data block usage condition is met, trigger the Trim operation detection module to release the unused space of the data block through the Trim operation according to the data block usage condition of the file system.
Please refer to fig. 8, which is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present disclosure. The electronic device may include a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate with each other via a communication bus 804. Among them, the memory 803 stores a computer program; the processor 801 may implement the data processing method described in the above-described method embodiment by executing a program stored on the memory 803.
The memory 803 referred to herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the memory 802 may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
The present embodiment also provides a machine-readable storage medium, such as the memory 803 in fig. 8, storing a computer program, which can be executed by the processor 801 in the electronic device shown in fig. 8 to implement the data processing method described in the above method embodiment.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims (18)

1. A data processing method is applied to a storage node, the storage node runs a file system realized based on a Solid State Disk (SSD), the file system is characterized by comprising a starting block, index blocks and data blocks, the size of the starting block is integral multiple of the size of a block, the size of each index block is integral multiple of a page, the logical addresses of the index blocks in the SSD are continuous, the size of an index block area formed by the index blocks is integral multiple of the size of the block, the size of each data block is integral multiple of the block, and the index blocks correspond to the data blocks one to one, and the method comprises the following steps:
when a data writing request is received, determining a first starting offset and a first ending offset of current data writing according to the size of data to be written and the writing offset;
when the first starting offset is not the starting position of the page, determining the starting position of the page where the first starting offset is located as the actual starting offset of the current data writing;
combining the data in the page where the first initial offset is located and the data to be written into the size of an integral number of pages for writing;
and when the first end offset is not the end position of the page, determining the end position of the page where the first end offset is located as an actual end offset, and filling the remaining data to be written into the page to the size of one page and writing the data into the page.
2. The method of claim 1, wherein before combining the data in the page with the first start offset and the data to be written into an integer page size for writing, further comprising:
if the current data writing is the first writing, reading the data in the page where the initial offset is located into a write cache;
after filling the remaining data to be written into the page to a size of the page and writing the data into the page, the method further includes:
and storing the data in at least the last page in the write cache.
3. The method of claim 1, further comprising:
when a data reading request is received, judging whether the reading offset of the data to be read is in a reading cache or not;
when the read offset is not in the read cache, determining a second starting offset and a second ending offset of the current data read according to the size of the data to be read and the read offset;
when the second starting offset is not the starting position of the page, determining the starting position of the page where the second starting offset is located as the actual starting offset of the current data reading;
when the second ending offset is not the ending position of the page, determining the ending position of the page where the second ending offset is located as the actual ending offset of the current data reading;
reading data between an actual starting offset and an actual ending offset of current data reading, returning data to be read to data reading request initiator equipment, and storing data in at least the last page in the read data in a read cache.
4. The method of claim 3, further comprising:
when the read offset is in the read cache, judging whether the data to be read is in the read cache completely;
when partial data in the data to be read is not in the read cache, determining the starting position of the next page of the partial data to be read in the read cache as the actual starting offset of the current data reading;
when the second ending offset is not the ending position of the page, determining the ending position of the page where the second ending offset is located as the actual ending offset of the current data reading;
reading data between an actual starting offset and an actual ending offset of current data reading, returning the part of data to be read and the read data to be read in the read cache to data reading request initiator equipment, and storing data in at least the last page in the read data in the read cache.
5. The method of claim 1, further comprising:
when a data reading request is received, judging whether the reading offset of the data to be read is in a reading cache or not;
when the read offset is not in the read cache, determining a second starting offset and a second ending offset of the current data read according to the size of the data to be read and the read offset;
according to the pre-reading configuration, the size of a reading cache and the size of data to be read, determining the start position of an Nth page before the page where the second start offset is located as the actual start offset of the current data reading, and determining the end position of an Mth page after the page where the second end offset is located as the actual end offset of the current data reading; wherein the size of M + N pages is smaller than or equal to the size of a read cache, and M is larger than or equal to 0,N and larger than or equal to 0;
reading data between an actual starting offset and an actual ending offset of current data reading, returning data to be read to data reading request initiator equipment, and storing data in at least the last page in the read data in a read cache.
6. The method of claim 5, further comprising:
when the read offset is in the read cache, judging whether the data to be read is in the read cache completely;
when partial data in the data to be read is not in the read cache, determining the starting position of the next page of the partial data to be read in the read cache as the actual starting offset of the current data reading;
determining the end position of the Kth page after the page where the second end offset is located as the actual end offset of the current data reading; the size of K pages is smaller than or equal to the size of the residual space in the read cache;
reading data between an actual starting offset and an actual ending offset of current data reading, returning the part of data to be read and the read data to be read in the read cache to data reading request initiator equipment, and storing data in at least the last page in the read data in the read cache.
7. The method of claim 1, further comprising:
releasing all space of the SSD when a formatting request is received; the formatting request carries formatting parameters, wherein the formatting parameters comprise the size of an index block and the size of a data block;
determining the total number of blocks according to the formatting parameters, the size of the page, the size of the block and the total capacity of the SSD; wherein the total number of blocks is the number of data blocks or index blocks;
and writing initialization information of the starting block and the index block into the designated position of the SSD according to the storage structure and the total number of the blocks.
8. The method of claim 1, wherein the file system comprises a Trim operation detection module, the method further comprising:
when the Trim operation detection module is started for the first time, according to the data block use condition of the file system, releasing the unused space of the data block through Trim operation;
periodically detecting whether the SSD meets Trim operation conditions;
and if so, releasing the space of the unused data block through Trim operation according to the use condition of the data block of the file system.
9. The utility model provides a data processing device, is applied to storage node, storage node operation has the file system based on solid state disk SSD realizes, a serial communication port, including starting block, index block and data block in the file system, starting block's size is the integral multiple of the size of block, index block's size is the integral multiple of page, just the logical address of index block in SSD is continuous, the size of the index block district that index block constitutes is the integral multiple of the size of block, data block's size is the integral multiple of block, index block with the data block one-to-one, the device includes:
a receiving unit configured to receive a data write request;
a determining unit, configured to determine, when the receiving unit receives a data write request, a first start offset and a first end offset of current data writing according to a size of data to be written and a write offset;
the determining unit is further configured to determine, when the first start offset is not the start position of the page, the start position of the page where the first start offset is located as an actual start offset for current data writing;
the data processing unit is used for combining the data in the page where the first starting offset is located and the data to be written into the page into the size of an integer number of pages for writing;
the determining unit is further configured to determine, as an actual ending offset, an ending position of a page where the first ending offset is located when the first ending offset is not the ending position of the page;
and the data processing unit is also used for filling the residual data to be written into a page to the size of the page and writing the data into the page.
10. The apparatus of claim 9,
the data processing unit is further configured to read data in the page where the initial offset is located into a write cache if the current data write is a first write;
the data processing unit is further configured to fill the remaining data to be written to a size of one page, and store the data in at least the last page in the write cache after the page is written.
11. The apparatus of claim 9,
the data processing unit is further configured to determine whether a read offset of the data to be read is in a read cache when the receiving unit receives the data read request;
the determining unit is further configured to determine a second starting offset and a second ending offset of current data reading according to the size of the data to be read and the read offset when the read offset is not in the read cache;
the determining unit is further configured to determine, when the second start offset is not the start position of the page, the start position of the page where the second start offset is located as an actual start offset for reading current data;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
12. The apparatus of claim 11,
the data processing unit is further configured to determine whether the data to be read is completely in the read cache when the read offset is in the read cache;
the determining unit is further configured to determine, when part of the data to be read is not in the read cache, a start position of a next page of the part of the data to be read in the read cache as an actual start offset of the current data reading;
the determining unit is further configured to determine, when the second ending offset is not the ending position of the page, the ending position of the page where the second ending offset is located as an actual ending offset of current data reading;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return the part of data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
13. The apparatus of claim 9,
the data processing unit is further configured to determine whether a read offset of the data to be read is in the read cache when the receiving unit receives the data read request;
the determining unit is further configured to determine a second starting offset and a second ending offset of current data reading according to the size of the data to be read and the read offset when the read offset is not in the read cache;
the determining unit is further configured to determine, according to a pre-reading configuration, a size of a read cache, and a size of data to be read, a start position of an nth page before a page where the second start offset is located as an actual start offset of current data reading, and determine an end position of an mth page after the page where the second end offset is located as an actual end offset of current data reading; wherein the size of M + N pages is smaller than or equal to the size of a read cache, and M is larger than or equal to 0,N and larger than or equal to 0;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
14. The apparatus of claim 13,
the data processing unit is further configured to determine whether the data to be read is completely in the read cache when the read offset is in the read cache;
the determining unit is further configured to determine, when part of the data to be read is not in the read cache, a start position of a next page of the part of the data to be read in the read cache as an actual start offset of the current data reading;
the determining unit is further configured to determine an end position of a kth page after the page where the second end offset is located as an actual end offset of current data reading; the size of the K pages is smaller than or equal to the size of the residual space in the read cache;
the data processing unit is further configured to read data between an actual start offset and an actual end offset of current data reading, return the part of data to be read in the read cache and the read data to be read to the data reading request initiator device, and store data in at least the last page of the read data in the read cache.
15. The apparatus of claim 9,
the data processing unit is further configured to release all the space of the SSD when the receiving unit receives the formatting request; the formatting request carries formatting parameters, wherein the formatting parameters comprise the size of an index block and the size of a data block;
the data processing unit is further configured to determine the total number of blocks according to the formatting parameter, the size of the page, the size of the block, and the total capacity of the SSD; wherein the total number of blocks is the number of data blocks or index blocks;
and the data processing unit is also used for writing initialization information of the starting block and the index block into the designated position of the SSD according to the storage structure and the total number of the blocks.
16. The apparatus of claim 9, wherein the file system comprises a clean-up Trim operation detection module;
the data processing unit is further configured to trigger the Trim operation detection module when the Trim operation detection module is started for the first time, and release the unused space of the data block through the Trim operation according to the use condition of the data block of the file system;
the data processing unit is also used for periodically detecting whether the SSD meets Trim operating conditions;
and the data processing unit is also used for triggering the Trim operation detection module to release the unused space of the data block through Trim operation according to the using condition of the data block of the file system if the data processing unit meets the requirement.
17. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing the communication between the processor and the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 8 when executing a program stored in the memory.
18. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-8.
CN201810692024.6A 2018-06-28 2018-06-28 Data processing method and device, electronic equipment and readable storage medium Active CN110658978B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810692024.6A CN110658978B (en) 2018-06-28 2018-06-28 Data processing method and device, electronic equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810692024.6A CN110658978B (en) 2018-06-28 2018-06-28 Data processing method and device, electronic equipment and readable storage medium

Publications (2)

Publication Number Publication Date
CN110658978A CN110658978A (en) 2020-01-07
CN110658978B true CN110658978B (en) 2022-11-01

Family

ID=69027481

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810692024.6A Active CN110658978B (en) 2018-06-28 2018-06-28 Data processing method and device, electronic equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN110658978B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114356232B (en) * 2021-12-30 2024-04-09 西北工业大学 Data reading and writing method and device
CN116049113B (en) * 2022-08-29 2023-10-20 荣耀终端有限公司 File system arrangement method, electronic device and computer readable storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103257831A (en) * 2012-02-20 2013-08-21 深圳市腾讯计算机系统有限公司 Reading-writing control method of storage and corresponding storage
CN103425602A (en) * 2013-08-15 2013-12-04 深圳市江波龙电子有限公司 Data reading and writing method and device for flash memory equipment and host system
WO2014015828A1 (en) * 2012-07-27 2014-01-30 腾讯科技（深圳）有限公司 Data storage space processing method and processing system, and data storage server
CN106372008A (en) * 2015-07-23 2017-02-01 福州瑞芯微电子股份有限公司 Data caching method and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6080799B2 (en) * 2014-05-28 2017-02-15 インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ A method for reading and writing through a file system for a tape recording system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103257831A (en) * 2012-02-20 2013-08-21 深圳市腾讯计算机系统有限公司 Reading-writing control method of storage and corresponding storage
WO2014015828A1 (en) * 2012-07-27 2014-01-30 腾讯科技（深圳）有限公司 Data storage space processing method and processing system, and data storage server
CN103425602A (en) * 2013-08-15 2013-12-04 深圳市江波龙电子有限公司 Data reading and writing method and device for flash memory equipment and host system
CN106372008A (en) * 2015-07-23 2017-02-01 福州瑞芯微电子股份有限公司 Data caching method and device

Also Published As

Publication number Publication date
CN110658978A (en) 2020-01-07

Similar Documents

Publication Publication Date Title
US11797191B2 (en) Systems and methods for storing data in SSD
EP2631916B1 (en) Data deletion method and apparatus
US11782632B2 (en) Selective erasure of data in a SSD
JP5418808B2 (en) Adaptive hybrid density memory storage control method and adaptive hybrid density memory storage
JP5649742B2 (en) Transaction log restore
RU2319227C2 (en) Recording method with double logging and data carrier for use with the method
US20160004474A1 (en) Data Erasing Method and Apparatus Applied to Flash Memory
US20050163014A1 (en) Duplicate data storing system, duplicate data storing method, and duplicate data storing program for storage device
TW201118877A (en) Flash memory device, data storage system, and operation method of a data storage system
WO2003088044A1 (en) Data storage device
KR20140016430A (en) A method for operating a memory unit, and a memory controller
CN109522154B (en) Data recovery method and related equipment and system
CN108628542B (en) File merging method and controller
CN110489357B (en) Method and system for hiding data on removable storage device
US11204864B2 (en) Data storage devices and data processing methods for improving the accessing performance of the data storage devices
CN112799595B (en) Data processing method, device and storage medium
US20090132757A1 (en) Storage system for improving efficiency in accessing flash memory and method for the same
US8473707B2 (en) Method for achieving sequential I/O performance from a random workload
US20210216448A1 (en) Data storage device in a key-value storage architecture with data compression, and non-volatile memory control method
CN110658978B (en) Data processing method and device, electronic equipment and readable storage medium
CN113568582A (en) Data management method and device and storage equipment
US20090083477A1 (en) Method and apparatus for formatting portable storage device
CN112035065B (en) Data writing method, device and equipment and computer readable storage medium
CN109032532B (en) Flash memory storage management method and flash memory device
CN112148220A (en) Method and device for realizing data processing, computer storage medium and terminal

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant