CN111930307B - Data reading method, device and equipment and computer readable storage medium - Google Patents
Data reading method, device and equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111930307B CN111930307B CN202010752681.2A CN202010752681A CN111930307B CN 111930307 B CN111930307 B CN 111930307B CN 202010752681 A CN202010752681 A CN 202010752681A CN 111930307 B CN111930307 B CN 111930307B
- Authority
- CN
- China
- Prior art keywords
- reading
- target
- target data
- stripe group
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9014—Indexing; Data structures therefor; Storage structures hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Memory System Of A Hierarchy Structure (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data reading method, which comprises the following steps: analyzing the received target data reading request to obtain a target data stripe group to be read; judging whether the target data strip group is read in sequence; if so, starting sequential reading operation and acquiring a target pre-reading window; reading a target data stripe group and a target pre-reading stripe group with a target pre-reading window length; and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into the pre-established hash chain table. By applying the technical scheme provided by the embodiment of the invention, the read cache hit rate is greatly improved, the memory utilization rate is improved, and the io reading performance is greatly improved. The invention also discloses a data reading device, equipment and a storage medium, and has corresponding technical effects.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data reading method, apparatus, device, and computer-readable storage medium.
Background
In cloud computing data centers, the performance of data storage is a core concern for users. While the cache usage of the storage system can greatly affect the performance of the storage system. However, the storage system is quite complex, and different application scenarios have different requirements for different io modes. High demands are put on the use and management of the cache. The cache is particularly important in the io aspect of the raid of the disk array, and directly determines each key performance index of the io bandwidth, the reading and writing times iops per second, the io time delay and the like.
Because the disk array is large, the system memory is relatively small. The conventional disk array cache usually has less read cache reservation, which causes that the io reading performance is limited during data reading, and the io performance, the read-write times per second and the io time delay are greatly influenced.
In summary, how to effectively solve the problems that the performance of reading io is limited in the conventional data reading mode, and the performance of io, the number of times of reading and writing per second, io delay and the like are greatly affected is a problem that needs to be solved by a person skilled in the art at present.
Disclosure of Invention
The invention aims to provide a data reading method, which greatly improves the hit rate of read cache, the utilization rate of memory and the io reading performance; another object of the present invention is to provide a data reading apparatus, a device and a computer readable storage medium.
In order to solve the technical problems, the invention provides the following technical scheme:
a data reading method comprising:
analyzing the received target data reading request to obtain a target data stripe group to be read;
judging whether the target data strip group is read in sequence;
if so, starting sequential reading operation and acquiring a target pre-reading window;
reading the target data stripe group and a target pre-reading stripe group with the length of the target pre-reading window;
and returning the target data strip group to a data request end, and storing the target pre-reading strip group into a pre-established hash chain table.
In a specific embodiment of the present invention, determining whether the target data stripe group is read sequentially includes:
judging whether a target data stripe arranged at the head in the target data stripe group is an initial stripe; wherein, the target data stripes in the target data stripe group are arranged according to the sequence to be read.
In a specific embodiment of the present invention, determining whether the target data stripe group is read sequentially includes:
acquiring a last data stripe in a preamble data stripe group corresponding to a previous data reading request;
judging whether a target data stripe arranged at the head in the target data stripe group is continuous with the last data stripe;
and the data stripes in the preorder data stripe group and the target data stripe group are arranged according to the sequence to be read.
In a specific embodiment of the present invention, determining whether the target data stripe group is read sequentially includes:
obtaining each preorder preread strip contained in the Hash chain table;
and judging whether a stripe continuous with a target data stripe arranged at the head in the target data stripe group exists in each preorder preread stripe.
In a specific embodiment of the present invention, after returning the target data stripe group to the data request end and storing the target pre-read stripe group in a pre-established hash chain table, the method further includes:
and storing the target data stripe group and the target pre-reading stripe group into a pre-established lru doubly-linked list according to the sequence of request time.
In an embodiment of the present invention, reading the target data stripe group includes:
judging whether the lru doubly linked list has the target data stripe group;
if yes, reading the target data stripe group from the lru doubly linked list, and adjusting the target data stripe group to the head of the lru doubly linked list;
and if not, reading the target data stripe group from the disk array.
In a specific embodiment of the present invention, before storing the target data stripe group and the target pre-read stripe group into a pre-established lru doubly linked list according to a request time sequence, the method further includes:
acquiring the total data amount of the target data stripe group and the target pre-reading stripe group;
obtaining an idle memory of the lru bidirectional linked list;
judging whether the free memory is more than or equal to the total data amount;
if not, gradually deleting the prestored data strips at the tail part of the lru bidirectional linked list according to a preset step length until the idle memory is more than or equal to the total data amount.
A data reading apparatus comprising:
the request analysis module is used for analyzing the received target data reading request to obtain a target data strip group to be read;
the first judgment module is used for judging whether the target data strip group is read in sequence;
a pre-reading window obtaining module, configured to start a sequential reading operation when it is determined that the target data stripe group is sequential reading, and obtain a target pre-reading window;
the stripe group reading module is used for reading the target data stripe group and a target pre-reading stripe group with the length of the target pre-reading window;
and the data return module is used for returning the target data strip group to a data request end and storing the target pre-reading strip group into a pre-established hash chain table.
A data reading apparatus comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data reading method as described above when executing the computer program.
A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data reading method as set forth above.
By applying the method provided by the embodiment of the invention, the received target data reading request is analyzed to obtain a target data stripe group to be read; judging whether the target data strip group is read in sequence; if so, starting sequential reading operation and acquiring a target pre-reading window; reading a target data stripe group and a target pre-reading stripe group with a target pre-reading window length; and returning the target data strip group to the data request end, and storing the target pre-reading strip group into a pre-established hash chain table. When the target data stripe group corresponding to the target data reading request is determined to be sequential reading operation, the target pre-reading stripe group with the target pre-reading window length is read into the Hash chain table in advance, and when data reading is needed, data reading is directly carried out according to the Hash chain table, so that the reading cache hit rate is greatly improved, the memory utilization rate is improved, and the io reading performance is greatly improved.
Accordingly, embodiments of the present invention further provide a data reading apparatus, a device and a computer-readable storage medium corresponding to the data reading method, which have the above technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without paying creative efforts.
FIG. 1 is a flowchart illustrating an implementation of a data reading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of data block storage in a disk array according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a hash chain table data stripe cache according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating another embodiment of a data reading method according to the present invention;
FIG. 5 is a flowchart illustrating another embodiment of a data reading method according to the present invention;
FIG. 6 is a flowchart illustrating another embodiment of a data reading method according to the present invention;
FIG. 7 is a block diagram of a data reading apparatus according to an embodiment of the present invention;
fig. 8 is a block diagram of a data reading apparatus according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the scope of protection of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a flowchart of an implementation of a data reading method according to an embodiment of the present invention, where the method may include the following steps:
s101: and analyzing the received target data reading request to obtain a target data stripe group to be read.
When data reading is needed, a target data reading request is sent to a data reading management center, and the target data reading request comprises a target data stripe group to be read. And the data reading management center receives the target data reading request and analyzes the received target data reading request to obtain a target data stripe group to be read.
S102: and judging whether the target data stripe group is read sequentially, if not, executing the step S103, and if so, executing the step S104.
After the target data stripe group is obtained through analysis, whether the target data stripe group is read sequentially is judged, if not, step S103 is executed, and if so, step S104 is executed.
Fig. 2 is a schematic diagram illustrating storage of data blocks in a disk array according to an embodiment of the present invention. Dividing the raid of the DISK array according to the stripe, and dividing the stripe into blocks according to the DISK of each DISK; the read cache is established by taking a stripe as a unit, and the whole DISK array is composed of four hard DISKs including DISK1, DISK2, DISK3 and DISK 4. I.e. read according to: d1, D2, D3, P for strip 0; p, D1, D2, D3 for lane 1; d3, P, D1, D2 for strip 2; reading at D2, D3, P, D1 of stripe 3 is sequential reading.
S103: and reading the target data stripe group and returning the target data stripe group to the data request terminal.
When the target data stripe group is determined to be non-sequential reading, the target data stripe group is directly read without pre-reading, and the target data stripe group is returned to the data request end.
S104: and starting sequential reading operation and acquiring a target pre-reading window.
When the target data stripe group is determined to be sequential reading, pre-reading is needed at the moment, sequential reading operation is started, and a target pre-reading window is obtained.
The size of the target pre-reading window can be set and adjusted according to actual conditions, and if the target pre-reading window is determined to be sequentially read in the initial step, the target pre-reading window can be set to be a strip; when the next sequential reading still is determined, the target pre-read window may be adjusted to two stripes. It should be noted that, in order to avoid affecting the data reading performance, the maximum value of the target pre-reading window may be set, for example, 8 stripes may be set.
S105: and reading the target data stripe group and the target pre-reading stripe group with the length of the target pre-reading window.
And after starting the sequential reading operation and acquiring the target pre-reading window, reading the target data stripe group and the target pre-reading stripe group with the length of the target pre-reading window.
S106: and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into the pre-established hash chain table.
As shown in fig. 3, fig. 3 is a schematic diagram of a hash chain table data stripe cache according to an embodiment of the present invention. And (3) a Hash chain table is established in advance, wherein 0-1023 represent 1024 nodes, and nodes corresponding to the strips are obtained in a mode of taking the strip numbers of the strips to be surplus to 1024. As shown in FIG. 3, stripe 0 is left to 1024 to yield 0, so stripe 0 is at node 0; the remainder of the stripe 1024 for 1024 is also 0, so stripe 1024 is at node 0 and stripe 1028 for 1024 is 4, so stripe 1028 is at node 4. After the target data stripe group and the target pre-reading stripe group are read, the target data stripe group is returned to the data request end, and the target pre-reading stripe group is stored in a pre-established hash chain table. By storing the target pre-reading strip group into the hash chain table, when sequentially reading, the target data strip to be read can be quickly called through the hash chain table, and the data reading efficiency is improved.
By applying the method provided by the embodiment of the invention, the received target data reading request is analyzed to obtain a target data stripe group to be read; judging whether the target data strip group is read in sequence; if so, starting sequential reading operation and acquiring a target pre-reading window; reading a target data stripe group and a target pre-reading stripe group with a target pre-reading window length; and returning the target data strip group to the data request end, and storing the target pre-reading strip group into a pre-established hash chain table. When the target data stripe group corresponding to the target data reading request is determined to be sequential reading operation, the target pre-reading stripe group with the target pre-reading window length is read into the Hash chain table in advance, and when data reading is needed, data reading is directly carried out according to the Hash chain table, so that the reading cache hit rate is greatly improved, the memory utilization rate is improved, and the io reading performance is greatly improved.
It should be noted that, based on the first embodiment, the embodiment of the present invention further provides a corresponding improvement scheme. In the following embodiments, steps that are the same as or correspond to those in the first embodiment may be referred to each other, and corresponding advantageous effects may also be referred to each other, which are not described in detail in the following modified embodiments.
Example two:
referring to fig. 4, fig. 4 is a flowchart of another implementation of a data reading method according to an embodiment of the present invention, where the method may include the following steps:
s401: and analyzing the received target data reading request to obtain a target data stripe group to be read.
S402: and judging whether the target data stripe arranged at the head in the target data stripe group is an initial stripe, if not, executing the step S403, and if so, executing the step S404.
And arranging the target data stripes in the target data stripe group according to the sequence to be read.
After the target data stripe group to be read is obtained through analysis, whether a target data stripe arranged at the head in the target data stripe group is an initial stripe is judged, if yes, the target data stripe arranged at the head in the target data stripe group is judged to be a stripe 0 in the disk array, if not, the current reading is sequential reading, step S404 is executed, and if not, the reading is non-sequential reading, and step S403 is executed.
S403: and reading the target data stripe group and returning the target data stripe group to the data request terminal.
S404: and starting sequential reading operation and acquiring a target pre-reading window.
S405: and judging whether a target data strip group exists in the pre-established lru doubly linked list, if so, executing step S406, and if not, executing step S407.
Referring to fig. 5, fig. 5 is a schematic diagram of a stripe cache of an lru doubly linked list according to an embodiment of the present invention. The doubly linked list is pre-established lru and all stripe buffers are placed into one lru doubly linked list according to the time order of their requests. After the sequential reading is determined, whether a target data stripe group exists in the pre-established lru doubly linked list is determined, if yes, step S406 is executed, and if not, step S407 is executed.
S406: the target data stripe group is read from the lru doubly linked list and adjusted to the head of the lru doubly linked list.
When it is determined lru that a target data stripe group exists in the doubly linked list, the target data stripe group is read directly from the lru doubly linked list and adjusted to the head of the lru doubly linked list. By adjusting the target data stripe group to the head of lru doubly linked list, the buffering time of hot data in lru doubly linked list is prolonged.
S407: and reading the target data stripe group from the disk array.
When it is determined lru that the target data stripe group does not exist in the doubly linked list, the target data stripe group is read from the disk array.
S408: and reading the target pre-reading strip group with the length of the target pre-reading window.
S409: and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into a pre-established hash chain table.
S410: and acquiring the total data quantity of the target data stripe group and the target pre-reading stripe group.
And after the total data amount of the target data stripe group and the target pre-reading stripe group is obtained, the total data amount of the target data stripe group and the target pre-reading stripe group is obtained.
S411: obtaining lru free memory of the doubly linked list.
And acquiring lru the free memory of the doubly linked list after acquiring the total data amount of the target data stripe group and the target pre-read stripe group.
S412: and judging whether the free memory is larger than or equal to the total data amount, if not, executing the step S413, and if so, skipping the step S413 and executing the step S414.
After the idle memory of the lru doubly linked list is acquired, whether the idle memory is greater than or equal to the total data amount is judged, if not, it is described that the existing cache data of the lru doubly linked list needs to be deleted, step S413 is executed, if so, it is described that the idle memory of the current lru doubly linked list can store the target data stripe group and the target pre-read stripe group, and step S414 is executed by skipping step S413.
S413: and gradually deleting the prestored data strips at the tail part of the lru doubly linked list according to a preset step length until the free memory is more than or equal to the total data amount.
And when the idle memory is determined to be smaller than the total data amount, gradually deleting the pre-stored data stripes of tail of the lru bidirectional linked list according to a preset step length until the idle memory is larger than or equal to the total data amount. If the stripes outside the maximum pre-read window of the current io can be deleted firstly, if the memory application still cannot be met, the doubly linked list is scanned lru again, and the stripe cache of the lru doubly linked list is released from tail to head.
S414: and storing the target data stripe group and the target pre-reading stripe group into an lru doubly linked list according to the sequence of the request time.
Example three:
referring to fig. 6, fig. 6 is a flowchart of another implementation of a data reading method according to an embodiment of the present invention, where the method may include the following steps:
s601: and analyzing the received target data reading request to obtain a target data stripe group to be read.
S602: and acquiring a last data stripe in the preamble data stripe group corresponding to the previous data reading request.
And after the target data stripe group to be read is obtained through analysis, acquiring a last data stripe in the preamble data stripe group corresponding to the previous data reading request.
S603: and judging whether the target data stripe arranged at the head in the target data stripe group is continuous with the last data stripe, if not, executing the step S604, and if so, executing the step S605.
After the last data stripe in the preamble data stripe group corresponding to the previous data reading request is obtained, whether a target data stripe arranged at the head in the target data stripe group is continuous with the last data stripe is judged, if yes, the current sequence reading is performed, step S605 is performed, and if not, the non-sequence reading is performed, and step S604 is performed.
And all the data stripes in the preamble data stripe group and the target data stripe group are arranged according to the sequence to be read.
S604: and reading the target data stripe group and returning the target data stripe group to the data request terminal.
S605: and starting sequential reading operation and acquiring a target pre-reading window.
S606: and judging whether a target data stripe group exists in the pre-established lru doubly linked list, if so, executing step S607, and if not, executing step S608.
S607: the target set of data stripes is read from the lru doubly linked list and adjusted to the head of the lru doubly linked list.
S608: and reading the target data stripe group from the disk array.
S609: and reading the target pre-reading strip group with the length of the target pre-reading window.
S610: and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into a pre-established hash chain table.
S611: and acquiring the total data quantity of the target data stripe group and the target pre-reading stripe group.
S612: obtain lru free memory for the doubly linked list.
S613: and judging whether the free memory is larger than the total data amount, if not, executing the step S614, and if so, skipping the step S614 to execute the step S615.
S614: and gradually deleting the prestored data strips at the tail part of the lru doubly linked list according to a preset step length until the free memory is larger than the total data amount.
S615: and storing the target data stripe group and the target pre-reading stripe group into an lru doubly linked list according to the sequence of the request time.
Example four:
referring to fig. 7, fig. 7 is a flowchart of another implementation of a data reading method according to an embodiment of the present invention, where the method may include the following steps:
s701: and analyzing the received target data reading request to obtain a target data stripe group to be read.
S702: and acquiring each pre-read stripe of each preamble contained in the hash chain table.
And after the target data stripe group to be read is obtained through analysis, obtaining each preorder preread stripe contained in the hash chain table.
S703: judging whether a strip continuous with a target data strip arranged at the head in the target data strip group exists in each pre-reading strip, if not, executing a step S704, and if so, executing a step S705.
And the data stripes in the preorder data stripe group and the target data stripe group are arranged according to the sequence to be read.
After obtaining each pre-read preamble strip included in the hash chain table, determining whether a strip continuous to a target data strip arranged at the head of the target data strip group exists in each pre-read preamble strip, if so, indicating that the current reading is sequential reading, executing step S705, and if not, indicating that the reading is non-sequential reading, and executing step S704.
S704: and reading the target data stripe group and returning the target data stripe group to the data request terminal.
S705: and starting sequential reading operation and acquiring a target pre-reading window.
S706: and judging whether the pre-established lru doubly linked list has the target data stripe group, if so, executing step 707, and if not, executing step 708.
S707: the target data stripe group is read from the lru doubly linked list and adjusted to the head of the lru doubly linked list.
S708: and reading the target data stripe group from the disk array.
S709: and reading the target pre-reading strip group with the length of the target pre-reading window.
S710: and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into a pre-established hash chain table.
S711: and acquiring the total data quantity of the target data stripe group and the target pre-reading stripe group.
S712: obtain lru free memory for the doubly linked list.
S713: and judging whether the free memory is larger than or equal to the total data amount, if not, executing the step S614, and if so, skipping S714 and executing the step S715.
S714: and gradually deleting the prestored data strips at the tail part of the lru doubly linked list according to a preset step length until the free memory is more than or equal to the total data amount.
S715: and storing the target data stripe group and the target pre-reading stripe group into lru bidirectional linked lists according to the sequence of request time.
Corresponding to the above method embodiments, embodiments of the present invention further provide a data reading apparatus, and the data reading apparatus described below and the data reading method described above may be referred to in correspondence with each other.
Referring to fig. 8, fig. 8 is a block diagram of a data reading apparatus according to an embodiment of the present invention, where the apparatus may include:
a request analysis module 81, configured to analyze the received target data reading request to obtain a target data stripe group to be read;
a first judging module 82, configured to judge whether the target data stripe group is read sequentially;
a pre-reading window obtaining module 83, configured to start a sequential reading operation when it is determined that the target data stripe group is sequential reading, and obtain a target pre-reading window;
a stripe group reading module 84, configured to read a target data stripe group and a target pre-read stripe group of a target pre-read window length;
and the data returning module 85 is configured to return the target data stripe group to the data requesting end, and store the target pre-read stripe group in the pre-established hash chain table.
By applying the device provided by the embodiment of the invention, the received target data reading request is analyzed to obtain a target data stripe group to be read; judging whether the target data strip group is read in sequence; if so, starting sequential reading operation and acquiring a target pre-reading window; reading a target data stripe group and a target pre-reading stripe group with a target pre-reading window length; and returning the target data strip group to the data request end, and storing the target pre-reading strip group into a pre-established hash chain table. When the target data stripe group corresponding to the target data reading request is determined to be sequential reading operation, the target pre-reading stripe group with the target pre-reading window length is read into the Hash chain table in advance, and when data reading is needed, data reading is directly carried out according to the Hash chain table, so that the reading cache hit rate is greatly improved, the memory utilization rate is improved, and the io reading performance is greatly improved.
In an embodiment of the present invention, the first determining module 82 is specifically a module for determining whether a first target data stripe in the target data stripe group is an initial stripe; and arranging the target data stripes in the target data stripe group according to the sequence to be read.
In one embodiment of the present invention, the first determining module 82 includes:
the preorder last stripe acquisition submodule is used for acquiring a last data stripe in a preorder data stripe group corresponding to the previous data reading request;
the first judgment submodule is used for judging whether a target data stripe arranged at the head in the target data stripe group is continuous with a last data stripe;
and the data stripes in the preorder data stripe group and the target data stripe group are arranged according to the sequence to be read.
In one embodiment of the present invention, the first determining module 82 includes:
a preamble pre-read stripe obtaining submodule for obtaining each preamble pre-read stripe contained in the hash chain table;
and the second judgment submodule is used for judging whether a strip continuous with a target data strip arranged at the head in the target data strip group exists in each pre-reading strip.
In one embodiment of the present invention, the apparatus may further include:
and the strip group storage module is used for returning the target data strip group to the data request end, storing the target pre-reading strip group into the pre-established hash chain table, and then storing the target data strip group and the target pre-reading strip group into the pre-established lru bidirectional chain table according to the request time sequence.
In one embodiment of the present invention, the stripe group reading module 84 includes a target stripe group reading sub-module, and the target stripe group reading sub-module includes:
the judging unit is used for judging lru whether a target data stripe group exists in the doubly linked list;
the first target stripe group reading unit is used for reading a target data stripe group from the lru bidirectional linked list and adjusting the target data stripe group to the head of the lru bidirectional linked list when the target data stripe group exists in the lru bidirectional linked list;
and a second target stripe group reading unit, configured to read the target data stripe group from the disk array when it is determined lru that the target data stripe group does not exist in the doubly linked list.
In one embodiment of the present invention, the apparatus may further include:
the data volume acquisition module is used for acquiring the total data volume of the target data stripe group and the target pre-reading stripe group before the target data stripe group and the target pre-reading stripe group are stored in the pre-established lru bidirectional linked list according to the request time sequence;
an idle memory obtaining module, configured to obtain lru an idle memory of a doubly linked list;
the second judgment module is used for judging whether the free memory is more than or equal to the total data amount;
and the strip deleting module is used for gradually deleting the prestored data strips at the tail part of the lru bidirectional linked list according to a preset step length until the idle memory is more than or equal to the total data amount when the idle memory is determined to be less than the total data amount.
Corresponding to the above method embodiment, referring to fig. 8, fig. 8 is a schematic diagram of a data reading apparatus provided by the present invention, where the apparatus may include:
a memory 91 for storing a computer program;
the processor 92, when executing the computer program stored in the memory 91, may implement the following steps:
analyzing the received target data reading request to obtain a target data stripe group to be read; judging whether the target data strip group is read in sequence; if so, starting sequential reading operation and acquiring a target pre-reading window; reading a target data stripe group and a target pre-reading stripe group with a target pre-reading window length; and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into a pre-established hash chain table.
For the description of the apparatus provided by the present invention, reference is made to the above-mentioned method embodiment, and the present invention is not redundantly described here.
Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of:
analyzing the received target data reading request to obtain a target data stripe group to be read; judging whether the target data strip group is read in sequence; if so, starting sequential reading operation and acquiring a target pre-reading window; reading a target data stripe group and a target pre-reading stripe group with a target pre-reading window length; and returning the target data stripe group to the data request end, and storing the target pre-reading stripe group into a pre-established hash chain table.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk.
For the introduction of the computer-readable storage medium provided by the present invention, please refer to the above method embodiments, which are not described herein again.
In the present specification, the embodiments are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the embodiments are referred to each other. The apparatuses, devices and computer-readable storage media disclosed in the embodiments correspond to the methods disclosed in the embodiments, so that the description is simple, and reference may be made to the description of the method for relevant points.
The principle and the implementation of the present invention are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (4)
1. A data reading method, comprising:
analyzing the received target data reading request to obtain a target data stripe group to be read;
judging whether the target data strip group is read in sequence;
if so, starting sequential reading operation and acquiring a target pre-reading window;
reading the target data stripe group and a target pre-reading stripe group with the length of the target pre-reading window;
returning the target data stripe group to a data request end, and storing the target pre-reading stripe group into a pre-established hash chain table; so that when the sequential reading is carried out subsequently, the target data strip needing to be read is called through the hash chain table;
judging whether the target data stripe group is read in sequence or not, including:
judging whether a target data stripe arranged at the head in the target data stripe group is an initial stripe; the target data stripes in the target data stripe group are arranged according to the sequence to be read;
after returning the target data stripe group to the data request end and storing the target pre-read stripe group in a pre-established hash chain table, the method further comprises the following steps:
storing the target data stripe group and the target pre-reading stripe group into a pre-established lru bidirectional linked list according to the sequence of request time;
reading the target data stripe group, comprising:
judging whether the lru doubly linked list has the target data stripe group;
if yes, reading the target data stripe group from the lru doubly linked list, and adjusting the target data stripe group to the head of the lru doubly linked list;
if not, reading the target data stripe group from the disk array;
before storing the target data stripe group and the target pre-read stripe group into the pre-established lru bi-directional linked list according to the request time sequence, the method further comprises:
acquiring the total data amount of the target data stripe group and the target pre-reading stripe group;
obtaining an idle memory of the lru bidirectional linked list;
judging whether the free memory is more than or equal to the total data amount;
if not, gradually deleting the prestored data strips at the tail part of the lru bidirectional linked list according to a preset step length until the idle memory is more than or equal to the total data amount.
2. A data reading apparatus, comprising:
the request analysis module is used for analyzing the received target data reading request to obtain a target data strip group to be read;
the first judgment module is used for judging whether the target data strip group is read in sequence;
a pre-reading window obtaining module, configured to start a sequential reading operation when it is determined that the target data stripe group is sequential reading, and obtain a target pre-reading window;
the stripe group reading module is used for reading the target data stripe group and the target pre-reading stripe group with the target pre-reading window length;
the data return module is used for returning the target data strip group to a data request end and storing the target pre-reading strip group into a pre-established hash chain table; when sequentially reading in the subsequent process, the data reading device is also used for calling a target data strip needing to be read through the hash chain table;
the first judging module is specifically configured to judge whether a first-ranked target data stripe in the target data stripe group is an initial stripe; the target data stripes in the target data stripe group are arranged according to the sequence to be read;
the stripe group storage module is used for storing the target data stripe group and the target pre-reading stripe group into a pre-established lru bi-directional linked list according to the request time sequence after the target data stripe group is returned to a data request end and the target pre-reading stripe group is stored into the pre-established hash linked list;
the strip group reading module comprises a target strip group reading submodule;
the target stripe group read sub-module includes:
a judging unit, configured to judge whether the lru bidirectional linked list has the target data stripe group; if yes, triggering a first target strip group reading unit; if not, triggering a second target strip group reading unit;
the first target stripe group reading unit is used for reading the target data stripe group from the lru doubly-linked list and adjusting the target data stripe group to the head of the lru doubly-linked list;
the second target stripe group reading unit is used for reading the target data stripe group from the disk array;
a data volume obtaining module, configured to obtain total data volumes of the target data stripe group and the target pre-read stripe group before the target data stripe group and the target pre-read stripe group are stored in a pre-established lru bidirectional linked list according to a request time sequence;
an idle memory obtaining module, configured to obtain an idle memory of the lru doubly linked list;
the second judgment module is used for judging whether the free memory is more than or equal to the total data amount; if not, triggering the strip deleting module;
and the strip deleting module is used for gradually deleting the prestored data strips at the tail part of the lru bidirectional linked list according to a preset step length until the idle memory is more than or equal to the total data amount.
3. A data reading apparatus, characterized by comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data reading method as claimed in claim 1 when executing the computer program.
4. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data reading method as claimed in claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010752681.2A CN111930307B (en) | 2020-07-30 | 2020-07-30 | Data reading method, device and equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010752681.2A CN111930307B (en) | 2020-07-30 | 2020-07-30 | Data reading method, device and equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111930307A CN111930307A (en) | 2020-11-13 |
| CN111930307B true CN111930307B (en) | 2022-06-17 |
Family
ID=73314368
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010752681.2A Active CN111930307B (en) | 2020-07-30 | 2020-07-30 | Data reading method, device and equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111930307B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112558866B (en) * | 2020-12-03 | 2022-12-09 | Oppo（重庆）智能科技有限公司 | Data pre-reading method, mobile terminal and computer readable storage medium |
| CN114237518B (en) * | 2022-02-22 | 2022-05-24 | 苏州浪潮智能科技有限公司 | Data reading method, system, device and terminal |
| CN115543871B (en) * | 2022-11-29 | 2023-03-10 | 苏州浪潮智能科技有限公司 | Data storage method and related equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914341A (en) * | 2013-01-06 | 2014-07-09 | 中兴通讯股份有限公司 | Data queue de-queuing control method and device |
| CN105243030A (en) * | 2015-10-26 | 2016-01-13 | 北京锐安科技有限公司 | Data caching method |
| CN106557431A (en) * | 2016-11-25 | 2017-04-05 | 郑州云海信息技术有限公司 | A kind of pre-head method and device for multichannel sequential flow |
| CN108052651A (en) * | 2017-12-26 | 2018-05-18 | 新华三技术有限公司 | Collocation method, device, pre-reading method of files and electronic equipment |
| CN108804234A (en) * | 2017-04-28 | 2018-11-13 | 腾讯科技（深圳）有限公司 | Data-storage system and its operating method |
| CN109976679A (en) * | 2019-04-11 | 2019-07-05 | 苏州浪潮智能科技有限公司 | A kind of distributed type assemblies volume pre-head method, system, equipment and computer media |
| CN109992209A (en) * | 2019-03-29 | 2019-07-09 | 新华三技术有限公司成都分公司 | Data processing method, device and distributed memory system |
| CN111258967A (en) * | 2020-02-11 | 2020-06-09 | 西安奥卡云数据科技有限公司 | Data reading method and device in file system and computer readable storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1280063A3 (en) * | 2001-07-27 | 2005-03-30 | Fujitsu Limited | Cache control methods and apparatus for hard disk drives |
| US8161244B2 (en) * | 2009-05-13 | 2012-04-17 | Microsoft Corporation | Multiple cache directories |
| US8549274B2 (en) * | 2010-04-14 | 2013-10-01 | Jade Quantum Technologies, Inc. | Distributive cache accessing device and method for accelerating to boot remote diskless computers |
| US9990296B2 (en) * | 2015-07-31 | 2018-06-05 | Oracle International Corporation | Systems and methods for prefetching data |
-
2020
- 2020-07-30 CN CN202010752681.2A patent/CN111930307B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914341A (en) * | 2013-01-06 | 2014-07-09 | 中兴通讯股份有限公司 | Data queue de-queuing control method and device |
| CN105243030A (en) * | 2015-10-26 | 2016-01-13 | 北京锐安科技有限公司 | Data caching method |
| CN106557431A (en) * | 2016-11-25 | 2017-04-05 | 郑州云海信息技术有限公司 | A kind of pre-head method and device for multichannel sequential flow |
| CN108804234A (en) * | 2017-04-28 | 2018-11-13 | 腾讯科技（深圳）有限公司 | Data-storage system and its operating method |
| CN108052651A (en) * | 2017-12-26 | 2018-05-18 | 新华三技术有限公司 | Collocation method, device, pre-reading method of files and electronic equipment |
| CN109992209A (en) * | 2019-03-29 | 2019-07-09 | 新华三技术有限公司成都分公司 | Data processing method, device and distributed memory system |
| CN109976679A (en) * | 2019-04-11 | 2019-07-05 | 苏州浪潮智能科技有限公司 | A kind of distributed type assemblies volume pre-head method, system, equipment and computer media |
| CN111258967A (en) * | 2020-02-11 | 2020-06-09 | 西安奥卡云数据科技有限公司 | Data reading method and device in file system and computer readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| F/M-CIP: Implementing Flash Memory Cache Using Conservative Insertion and Promotion;Jing Yang等;《2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing》;20150709;第81-90页 * |
| PRCD技术及其软件实现;谢长生等;《计算机工程》;20030305(第05期);第11-13页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111930307A (en) | 2020-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111930307B (en) | Data reading method, device and equipment and computer readable storage medium | |
| US9519587B2 (en) | Pre-reading file containers storing unread file segments and segments that do not belong to the file | |
| KR100338224B1 (en) | A very efficient technique for dynamically tracking locality of a reference | |
| CN107870732B (en) | Method and apparatus for flushing pages from solid state storage devices | |
| US9367262B2 (en) | Assigning a weighting to host quality of service indicators | |
| KR102538126B1 (en) | Tail latency aware foreground garbage collection algorithm | |
| Gotlieb et al. | Performance of movable-head disk storage devices | |
| CN103562881A (en) | Defragmentation of data storage pools | |
| US9229864B1 (en) | Managing metadata synchronization for reducing host system latency in a storage system | |
| CN107430551B (en) | Data caching method, storage control device and storage equipment | |
| US20150186453A1 (en) | Tiered index management | |
| CN112100293A (en) | Data processing method, data access method, data processing device, data access device and computer equipment | |
| US10929025B2 (en) | Data storage system with I/O determinism latency optimization | |
| US20170123975A1 (en) | Centralized distributed systems and methods for managing operations | |
| US11461239B2 (en) | Method and apparatus for buffering data blocks, computer device, and computer-readable storage medium | |
| CN109407997B (en) | Data processing method, device and equipment and readable storage medium | |
| CN110750211B (en) | Storage space management method and device | |
| CN109582649A (en) | A kind of metadata storing method, device, equipment and readable storage medium storing program for executing | |
| US9361402B2 (en) | Tiered index management | |
| CN116340198B (en) | Data writing method and device of solid state disk and solid state disk | |
| CN109977074B (en) | HDFS-based LOB data processing method and device | |
| CN113608679A (en) | File storage method, device, equipment and computer readable storage medium | |
| CN114327272A (en) | Data processing method, solid state disk controller and solid state disk | |
| CN109992217B (en) | Service quality control method and device, electronic equipment and storage medium | |
| CN109408412B (en) | Memory prefetch control method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |