CN110703981A - Data reading and writing method, terminal and computer readable storage medium - Google Patents
Data reading and writing method, terminal and computer readable storage medium Download PDFInfo
- Publication number
- CN110703981A CN110703981A CN201810751082.1A CN201810751082A CN110703981A CN 110703981 A CN110703981 A CN 110703981A CN 201810751082 A CN201810751082 A CN 201810751082A CN 110703981 A CN110703981 A CN 110703981A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- read
- data index
- hit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 239000007787 solid Substances 0.000 claims abstract description 10
- 238000012512 characterization method Methods 0.000 claims 3
- 238000010586 diagram Methods 0.000 description 18
- 238000013500 data storage Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000002452 interceptive effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000002688 persistence Effects 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000001052 transient effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The embodiment of the invention provides a data reading and writing method, a terminal and a computer readable storage medium, wherein the method comprises the following steps: receiving a data reading instruction, wherein the data reading instruction comprises characteristic information of data to be read; determining a hit data index in a memory according to the characteristic information; and reading data in a cache according to the hit data index, wherein the cache is a Solid State Disk (SSD).
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a data reading and writing method, a terminal, and a computer-readable storage medium.
Background
In the fields of distributed storage systems, virtualization, and the like, Input/Output (I/O) speeds of mechanical hard disks often become a bottleneck of the system because of the need to simultaneously handle pressures from different services. The general method for solving the problem is to use a high-performance memory with a high read-write speed between a service system and a disk as an I/O cache, and exchange hot data which is frequently read and written into the cache as much as possible so as to improve the response speed of the system.
However, in a large-scale system, the total amount of hot data easily exceeds the memory capacity, so that the memory cache data needs to be frequently eliminated and reloaded to meet the business requirement, which greatly affects the caching effect. Meanwhile, when the node is restarted or powered off, the memory cache needs to be reloaded, and the possibility of hot spot data loss can be met.
Disclosure of Invention
Embodiments of the present invention provide a data reading and writing method, a terminal, and a computer-readable storage medium, which can solve the problem of memory limitation, increase cache capacity, and ensure that hot-point data is not lost after power failure.
The technical scheme of the embodiment of the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a data reading and writing method, where the method includes:
receiving a data reading instruction, wherein the data reading instruction comprises characteristic information of data to be read;
determining a hit data index in a memory according to the characteristic information;
and reading data in a cache according to the hit data index, wherein the cache is a Solid State Disk (SSD).
In a second aspect, an embodiment of the present invention provides a data reading and writing method, where the method includes:
receiving a data writing instruction, wherein the data writing instruction comprises characteristic information of data to be written;
determining a hit data index in a memory according to the characteristic information;
and updating the pointed position of the hit data index in a cache according to the characteristic information, wherein the cache is a Solid State Disk (SSD).
In a third aspect, an embodiment of the present invention provides a terminal, where the terminal at least includes: a processor and a storage medium configured to store executable instructions, wherein: the processor is configured to execute stored executable instructions;
the executable instructions are configured to perform the above-described data read-write method.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and the computer-executable instructions are configured to perform the above data reading and writing method.
The embodiment of the invention provides a data reading and writing method, a terminal and a computer readable storage medium, wherein a data reading instruction is received, and the data reading instruction comprises characteristic information of data to be read; determining a hit data index in a memory according to the characteristic information; reading data in a cache according to the hit data index, wherein the cache is a Solid State Disk (SSD); therefore, the cache is the SSD, so that the problem of memory limitation can be solved, the cache capacity is increased, and the hot point data can be ensured not to be lost after power failure because the data is read according to the hit data index.
Drawings
In the drawings, which are not necessarily drawn to scale, like reference numerals may describe similar components in different views. Like reference numerals having different letter suffixes may represent different examples of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed herein.
FIG. 1 is a schematic diagram of a flow chart of a data reading and writing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a flow chart of implementing a second data read-write method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an implementation flow of a data read/write method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a flow chart of a method for reading and writing data according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a structure of a log according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a cache structure according to an embodiment of the present invention;
FIG. 7 is a flowchart illustrating a method for recovering a cache by replaying a log according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the following describes specific technical solutions of the present invention in further detail with reference to the accompanying drawings in the embodiments of the present invention. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
Example one
An embodiment of the present invention provides a data reading and writing method, and fig. 1 is a schematic flow chart illustrating an implementation of a data reading and writing method according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S101, receiving a data reading instruction, wherein the data reading instruction comprises characteristic information of data to be read.
Here, the data reading instruction is for performing a reading operation on multimedia data or interactive information stored in a system or a data storage device. The data reading instruction comprises characteristic information of data to be read, and the characteristic information of the data to be read can comprise operation information, data position information, data length and other information. The operation information may be a recorded specific operation flow of the user in the system, the data location information may be an actual storage address of the data, and the data length is an actual length of the read-write data.
And step S102, determining a hit data index in the memory according to the characteristic information.
The data index is used to provide a data location pointing to actual data stored in the system or data storage device, and then to quickly retrieve the actual data according to the pointed location. Here, the data index may be any existing data index, such as a hash index.
The hit data index in this embodiment is located in a Memory (MEM), and the data index is searched in the Memory according to the feature information, and when the corresponding data index is found, the data index is considered to be hit, and the found data index is the hit data index.
And step S103, reading data in the cache according to the hit data index.
Here, the cache is a Solid State Drive (SSD), and the SSD is used as the cache in this embodiment, and the SSD uses a flash memory as the storage medium, so that the read/write speed is faster than that of a mechanical hard disk. In addition, the SSD has a larger capacity than the conventional memory, and has a nonvolatile data characteristic than the conventional memory. Therefore, in the embodiment, the SSD is used as the cache, which not only can enlarge the cache capacity and improve the response speed of the system, but also can ensure stable storage of data, and is not easy to lose. In addition, in the embodiment, the SSD is paired with the mechanical hard disk, and the SSD is paired with the mechanical hard disk in a one-to-many manner according to the capacity of the SSD, so that the SSD is used as a cache to store data to be read.
The reading of data according to the hit data index may specifically be: when the data index is hit, that is, the data index is found, it can be considered that the data to be read has the data index corresponding to the data index in the memory, and at this time, the data to be read can be directly read from the SSD.
And if the data index is not hit, namely the data index is not found, the data to be read is considered to have no data index corresponding to the data index in the memory, at the moment, the data is directly read from the mechanical hard disk, meanwhile, if the read data is hot data with high access frequency, the hot data is written into the SSD, the data index is generated according to the characteristic information of the hot data, and the generated data index is loaded into the memory, so that whether the data index can be hit or not can be determined in the memory according to the characteristic information when the hot data is read subsequently, and when the data index is hit, the hot data can be directly read in the SSD without reading the hot data from the mechanical hard disk. Therefore, the missed hot data is written into the SSD with the higher reading speed, and the data index is generated for the missed hot data, so that the hot data can be read in the SSD in a fast way when the hot data is read by a subsequent access system or after the system is powered off and restarted, and the problem of hot data loss is avoided; and the data storage capacity of the mechanical hard disk is reduced, so that the aim of accelerating the mechanical hard disk is fulfilled.
The data reading and writing method provided by the embodiment of the invention comprises the steps of firstly receiving a data reading instruction, wherein the data reading instruction comprises characteristic information of data to be read; then determining a hit data index in the memory according to the characteristic information; and finally, reading data in a cache according to the hit data index, wherein the cache is a Solid State Disk (SSD). The cache in the embodiment is the SSD, so that the problem of memory limitation can be solved, the cache capacity is increased, and the hot point data can be ensured not to be lost after power failure because the data is read according to the hit data index.
Example two
Based on the foregoing embodiment, an embodiment of the present invention further provides a data reading and writing method, fig. 2 is a schematic flow chart illustrating an implementation of a second data reading and writing method according to an embodiment of the present invention, and as shown in fig. 2, the method includes the following steps:
step S201, receiving a data reading instruction, where the data reading instruction includes characteristic information of data to be read.
Here, the data reading instruction is for performing a reading operation on multimedia data or interactive information stored in a system or a data storage device.
Step S202, searching a data index in the memory according to the characteristic information.
Here, the characteristic information of the data to be read includes a location of the data to be read, where the location of the data may be an actual storage address of the data. Correspondingly, the step S202 includes:
step S2021, obtain a data index table from the memory.
Here, the data index table includes at least one data index identifier and data location information corresponding to the data index identifier.
Step S2022, find a data index in a data index table according to the position of the data to be read.
Here, a data index is searched in the data index table according to the position of the data to be read, and when data position information corresponding to the data index identifier is the same as the position of the data to be read, it indicates that the data index corresponding to the characteristic information exists in the data index table, at this time, the data index corresponding to the data index identifier is determined as the searched data index, and the search result is marked as the searched data index.
Otherwise, searching the data index in the data index table according to the position of the data to be read, and when the data position information corresponding to the data index identification is not the same as the position of the data to be read, indicating that the data index corresponding to the characteristic information is not available in the data index table, and at the moment, directly marking the search result as the data index which is not searched.
Step S203, when the data index is found in the memory, determining the found data index as the hit data index.
Here, when the search result is that the data index is found, it indicates that the hit data index exists in the memory, and the found data index is determined as the hit data index.
Step S204, when the hit data index is determined in the memory, the data to be read corresponding to the position of the data to be read is read.
Here, when the hit data index is determined in the memory, it indicates that the data to be read is the data already stored in the SSD, and the data is directly read according to the position of the data to be read.
Otherwise, when the hit data index is not determined in the memory, it indicates that the data to be read is data not stored in the SSD, and step S204 may include:
step S2041: and reading the data to be read from the mechanical hard disk.
Here, since the hit data index is not determined in the memory, the data to be read is data stored in the mechanical hard disk, and then the data to be read is directly read from the mechanical hard disk.
The mechanical hard disk of this embodiment may be any conventional ordinary hard disk, such as serial data advanced (SATA).
In the embodiment, the SSD and the mechanical hard disk are paired for use, and the SSD and the mechanical hard disk are paired in a one-to-many manner according to the capacity of the SSD, and when the plurality of mechanical hard disks are used for storing data, the plurality of mechanical hard disks may be the same type of mechanical hard disk or different types of mechanical hard disks. When the SSD and the mechanical hard disk are used simultaneously, the SSD is used as a cache to store the data to be stored; the mechanical hard disk is used as an auxiliary storage device to store a large amount of data in a large-scale system.
Step S2042: and when the data to be read is hot spot data, writing the hot spot data into the SSD.
In the embodiment of the present invention, before data storage and data processing, the system performs pre-determination on the data, determines the frequency of access to the data, and when the data is accessed frequently, indicates that the data is important data relative to other data, and marks the data as the hot data. In addition, the user may mark data that may need to be frequently accessed later as hot spot data in advance, so that the data may be directly accessed as hot spot data in the subsequent data processing process.
In this embodiment, when the data to be read is hot data, it indicates that the data to be read is relatively important data, and then the data to be read is frequently accessed, and since the data to be read is stored in the mechanical hard disk, it is obvious that the time required for reading the data to be read from the mechanical hard disk is long because the reading and writing speed of the mechanical hard disk is slow, and at this time, the data to be read can be written into the SSD, so that since the data to be read is hot data, the hot data can be directly and quickly read from the SSD in the following process.
Step S2043; and acquiring the characteristic information of the hot spot data, and generating a data index according to the characteristic information of the hot spot data.
Here, a data index is generated according to the feature information of the hot spot data, and the generated data index is loaded into the memory. Therefore, whether the data index can be hit or not can be determined in the memory according to the characteristic information when the hot spot data is read subsequently, and when the data index is hit, the hot spot data can be directly read in the SSD without reading the hot spot data from the mechanical hard disk.
In the embodiment of the present invention, the data to be read may be stored in the SSD in the form of a Log, where the Log includes metadata and actual data, and the metadata includes the feature information.
In the embodiment of the invention, a concept of "read Log" is introduced, that is, when the data to be read is hot spot data, the data to be read is also written into the SSD in a Log form, but metadata in the Log is fixed, that is, the Log does not involve metadata change, but includes actual data itself, so that the hot spot data can be conveniently used as a cache subsequently.
In this embodiment, when the data to be read is not hotspot data and the actual data in the data to be read is not changed, the metadata in the Log is not updated.
And when the data to be read is not hotspot data and the actual data in the data to be read changes, updating the metadata in the Log.
The data reading and writing method provided by the embodiment of the invention comprises the steps of firstly receiving a data reading instruction, wherein the data reading instruction comprises characteristic information of data to be read; then, searching a data index in a memory according to the characteristic information, and determining the searched data index as the hit data index when the data index is searched in the memory; and finally, when the hit data index is determined in the memory, reading the data to be read corresponding to the position of the data to be read in the cache SSD. The cache in the embodiment is the SSD, so that the problem of memory limitation can be solved, the cache capacity is increased, and the hot point data can be ensured not to be lost after power failure because the data is read according to the hit data index.
EXAMPLE III
An embodiment of the present invention provides a data reading and writing method, fig. 3 is a schematic diagram of an implementation flow of a data reading and writing method according to an embodiment of the present invention, and as shown in fig. 3, the method includes the following steps:
step S301, receiving a data writing instruction, wherein the data writing instruction comprises characteristic information of data to be written.
Here, the data writing instruction is for writing and storing multimedia data or interactive information into a system or a data storage device. The data writing instruction comprises characteristic information of data to be written, and the characteristic information of the data to be written can comprise operation information, data position information, data length and other information. The operation information may be a recorded specific operation flow of the user in the system, the data location information may be an actual storage address of the data, and the data length is an actual length of the read-write data.
Step S302, determining a hit data index in the memory according to the feature information.
The data index is used to provide a data location pointing to actual data stored in the system or data storage device, and then to quickly retrieve the actual data according to the pointed location. Here, the data index may be any existing data index, such as a hash index.
The hit data index in this embodiment is located in the memory, the data index is searched in the memory according to the feature information, when the corresponding data index is found, the data index is considered to be hit, and the found data index is the hit data index.
Step S303, updating the position pointed by the hit data index in the cache according to the feature information.
Here, the cache is a solid state disk SSD. In the embodiment, the SSD is used as the cache, and the SSD uses the flash memory as the storage medium, so that the read/write speed is faster compared to a mechanical hard disk. In addition, the SSD has a larger capacity than the conventional memory, and has a nonvolatile data characteristic than the conventional memory. Therefore, in the embodiment, the SSD is used as the cache, which not only can enlarge the cache capacity and improve the response speed of the system, but also can ensure stable storage of data, and is not easy to lose. In addition, in the embodiment, the SSD is paired with the mechanical hard disk, and the SSD is paired with the mechanical hard disk in a one-to-many manner according to the capacity of the SSD, so that the SSD is used as a cache to store data to be written.
Updating the hit data index according to the feature information may specifically be: when the data index is hit, that is, the data index is found, it may be considered that the original data corresponding to the data to be written is stored in the SSD, but the original data needs to be updated because the data is continuously changed, and accordingly, after the original data is updated, the original data index corresponding to the original data also needs to be updated to adapt to the updated original data. Therefore, the hit data index is updated according to the feature information, where the hit data index is the original data index corresponding to the original data. And updating the hit data index to enable the updated data index to correspond to the data to be written.
If the data index is not hit, that is, the data index is not found, it is determined that any original data corresponding to the data to be written is not stored in the SSD, at this time, the data index needs to be constructed according to the feature information of the data to be written to generate the data index corresponding to the data to be written, and the generated data index is loaded into the memory, so that the data index can be hit in the memory when the data to be written needs to be read subsequently, and the data to be written is read according to the data index. Therefore, as the data index is generated according to the characteristic information of the data to be written and loaded into the memory, and the data to be written is written into the SSD, the data consistency of the data to be written can be enhanced, and the problem that the data to be written is different from the actually written data is avoided.
The data reading and writing method provided by the embodiment of the invention comprises the steps of firstly receiving a data writing instruction, wherein the data writing instruction comprises characteristic information of data to be written; then determining a hit data index in the memory according to the characteristic information; and finally, updating the pointed position of the hit data index in a cache according to the characteristic information, wherein the cache is a Solid State Disk (SSD). The cache in the embodiment is the SSD, so that the problem of memory limitation can be solved, the cache capacity is increased, and the read-write speed of the SSD is fast, so that the write operation can be considered to be completed when the data to be written is written into the SSD, and whether the data is written into the mechanical hard disk is not concerned, that is, the write operation can be considered to be completed without waiting for the mechanical hard disk to complete the data writing, thereby achieving the effect of reducing the write latency.
Example four
Based on the foregoing embodiment, an embodiment of the present invention further provides a data reading and writing method, fig. 4 is a schematic flow chart illustrating an implementation of a four data reading and writing method according to an embodiment of the present invention, and as shown in fig. 4, the method includes the following steps:
step S401, receiving a data writing instruction, wherein the data writing instruction comprises characteristic information of data to be written.
Here, the data writing instruction is for writing and storing multimedia data or interactive information into a system or a data storage device.
Step S402, searching a data index in the memory according to the characteristic information.
Here, the characteristic information of the data to be written includes a location of the data to be written, where the location of the data may be an actual storage address of the data. Correspondingly, the step S202 includes:
step S4021, acquiring a data index table from the memory.
Here, the data index table includes at least one data index identifier and data location information corresponding to the data index identifier.
Step S4022, searching a data index in the data index table according to the position of the data to be written.
Here, a data index is searched in the data index table according to the position of the data to be written, and when data position information corresponding to the data index identifier is the same as the position of the data to be written, it indicates that a data index corresponding to the characteristic information exists in the data index table, at this time, the data index corresponding to the data index identifier is determined as the searched data index, and the search result is marked as the searched data index.
Otherwise, searching a data index in the data index table according to the position of the data to be written, and when the data position information corresponding to the data index identification is not the same as the position of the data to be written, indicating that the data index corresponding to the characteristic information is not in the data index table, and at the moment, directly marking the search result as the data index which is not searched.
Step S403, when the data index is found in the memory, determining the found data index as the hit data index.
Here, when the search result is that the data index is found, it indicates that the hit data index exists in the memory, and the found data index is determined as the hit data index.
Step S404, when the hit data index is determined in the memory, updating the location pointed by the hit data index in the SSD according to the location of the data to be written.
Here, when the hit data index is determined in the memory, it indicates that original data corresponding to data to be written has been stored in the SSD before, and the hit data index is the data index of the original data, and the location pointed by the hit data index in the SSD is updated according to the location pointed by the data to be written, so that the location pointed by the hit data index in the SSD is consistent with the location pointed by the data to be written.
Otherwise, when the hit data index is not determined in the memory, it indicates that any original data corresponding to the data to be written is not stored in the SSD before, step S204 may include:
step S4041, the characteristic information of the data to be written is obtained.
Step S4042, constructing a data index according to the characteristic information of the data to be written.
Here, a data index is constructed according to the characteristic information of the data to be written, and the data index corresponding to the data to be written is regenerated. Therefore, when the data is changed, the data to be written and the data index can be kept corresponding, so that the data to be written can be directly read according to the data index by hitting the data index when the data to be written needs to be read subsequently.
Step S4043, the generated data index is loaded into the memory.
Here, the generated data index is loaded into the memory, so that when the data to be written is subsequently accessed, the data index can be directly hit in the memory according to the characteristic information of the data to be written, and then the data to be written is accessed according to the hit data index.
In an embodiment of the present invention, the data reading and writing method further includes:
step S405, when the data to be written is hot data, persistently writing the data to be written into the SSD.
In the embodiment of the present invention, before data storage and data processing, the system performs pre-determination on the data, determines the frequency of access to the data, and when the data is accessed frequently, indicates that the data is important data relative to other data, and marks the data as the hot data. In addition, the user may mark data that may need to be frequently accessed later as hot spot data in advance, so that the data may be directly accessed as hot spot data in the subsequent data processing process.
For easier understanding of the embodiments of the present invention, the concept of persistence is explained here:
persistence refers to the mechanism by which program data is transitioned between a persistent state and a transient state. Colloquially, transient data (such as data in a memory, which cannot be permanently stored) is persisted into persistent data (such as persisted into a database, which can be permanently stored). That is, data (e.g., objects in memory) is saved to a storage device (e.g., disk) that may be permanently saved.
In this embodiment, the operation of persisting the data to be written into the SSD may be performed before the execution determines the hit data index in the cache according to the feature information, so that, because the hot data is persisted into the SSD before the determination of the hit data index, even if there is a system abnormal power failure or the like, it can be ensured that the hot data is not lost. At the moment, after the system is restarted, the hot spot data can be recovered only by replaying the data in the SSD, and the data index is regenerated according to the characteristic information of the hot spot data so as to recover the cache, thereby achieving the possibility of preventing the hot spot data from being lost.
In the embodiment of the present invention, the data to be written may be written in the SSD in the form of a Log, where the Log includes metadata and actual data, and the metadata includes the feature information.
The data reading and writing method provided by the embodiment of the invention comprises the following steps of firstly, receiving a data writing instruction, wherein the data writing instruction comprises characteristic information of data to be written; then, searching a data index in a memory according to the characteristic information; when the data index is found in the memory, determining the found data index as the hit data index; and finally, when the hit data index is determined in the memory, updating the pointed position of the hit data index in the SSD according to the position of the data to be written. The cache in the embodiment is the SSD, so that the problem of memory limitation can be solved, the cache capacity is increased, and the read-write speed of the SSD is fast, so that the write operation can be considered to be completed when the data to be written is written into the SSD, and whether the data is written into the mechanical hard disk is not concerned, that is, the write operation can be considered to be completed without waiting for the mechanical hard disk to complete the data writing, thereby achieving the effect of reducing the write latency. And, since data is persisted into the SSD, hot spot data loss can be prevented.
EXAMPLE five
An embodiment of the present invention provides a data reading and writing method, which optimizes a Log of a file system based on the Log, and is a schematic structural diagram of the Log in the embodiment of the present invention as shown in fig. 5. As can be seen from fig. 5, the Log is recorded in the form of a metadata portion 501+ an actual data portion 502. The metadata portion 501 (feature information) is recorded on the SSD, but not on the mechanical hard disk, and the actual data portion 502 (data to be written or data to be read) is written into the SSD as a cache, and then it is determined whether to write into the mechanical hard disk as needed. In the embodiment, the SSD and the mechanical hard disk are paired for use, and the SSD and the mechanical hard disk are paired in a one-to-many manner according to the capacity of the SSD, so that the SSD is used as a cache to store the Log.
The metadata part 501 of the Log is metadata information (including information such as specific operations and data locations) such as index changes, and the actual data part 502 is actually read and written data, i.e., actual data. The Log is mainly composed of the following parts:
OpInfo: operating; used to indicate specific operational information.
Offset: offsetting; the actual location of the data is read and written.
Length: the length of the data; the actual length of the read and write data.
Data: data; the actual data read and written.
Meanwhile, the embodiment introduces a concept of "read Log", that is, for the read operation identified as the hot data, the read operation is also written into the SSD in the form of Log, and the Log does not involve metadata change, but includes the actual data itself, so that the actual data can be used as a cache in the following.
Fig. 6 is a schematic view of a cache structure according to an embodiment of the present invention, and as shown in fig. 6, the cache structure of this embodiment adopts a Memory (MEM) 601+ SSD 602 structure. Where MEM 601 serves as a cache storing only data indexes and SSD 602 stores log information. Because the data index is small, the data indexes of all logs can be loaded in the MEM 601, thereby achieving the purpose of increasing the cache capacity.
The following describes an embodiment of the present invention in two schemes.
The first process is as follows: and recovering and rebuilding the cache.
In this embodiment, because the log has persisted into the SSD, after exception recovery, the log may be read and the cache (i.e., hot point data) may be recovered from the log metadata through log playback. Fig. 7 is a schematic flowchart of a method for recovering a cache through log playback according to an embodiment of the present invention, and as shown in fig. 7, the method for recovering a cache through log playback includes the following steps:
step S701 reads the log information item by item.
Step S702, obtaining the metadata information in the log, and reconstructing the data index according to the metadata information.
Step S703 loads the reconstructed data index into the memory, and causes the reconstructed data index to point to the position of the actual data in the log.
And a second process: actual I/O processing.
For the actual I/O request, the operation type is different, and the operation type is divided into two types of read operation and write operation, and the read operation and the write operation are respectively processed in the actual I/O request. The processing flow of the read operation and the write operation will be described below.
The processing flow of the reading operation is as follows:
A) and if the cache is hit, directly reading the actual data in the SSD according to the data index information.
B) If the cache misses, the actual data is read from the SATA. Meanwhile, if the actual data is identified as the hot data, generating a Log for the read operation, and writing the Log into the SSD. And meanwhile, generating a data index and loading the data index to a memory.
The processing flow of the write operation is as follows:
A) firstly, writing a Log and recording Log information; and when the Log is successfully written, recovering the upper-layer application of the system.
B) And if the cache is hit, updating the data index according to the metadata information in the Log, so that the data index points to the new Log.
C) And if the cache is not hit, reconstructing the data index according to the metadata information in the Log, and loading the data index into the memory.
It should be noted that the cache hit refers to finding a data index in the cache, and correspondingly, the cache miss refers to not finding a data index in the cache.
In the embodiment of the invention, for the Log Log generated by the read operation or the write operation, the hash index can be constructed in the memory, and the cache is realized through the hash index. The hash index is firstly searched in the cache in the read operation, and if the cache is hit, the actual data is directly read from the SSD, so that the aim of accelerating the mechanical hard disk is fulfilled. After the log persistence is finished, updating the memory cache for subsequent access to the data in the cache in the write operation; meanwhile, after the log record is finished, the write operation can be considered to be finished, and the write operation is considered to be finished without waiting for the mechanical hard disk to finish the data write-in, so that the aim of reducing the write delay is fulfilled. In addition, after the system is restarted after power failure, the method of the embodiment can recover data and reconstruct data index through log playback, and recover cache, thereby achieving the purposes of preventing hot data from being lost and ensuring strong consistency of data.
EXAMPLE six
An embodiment of the present invention provides a terminal, fig. 8 is a schematic diagram of a composition structure of a terminal according to an embodiment of the present invention, and as shown in fig. 8, the terminal 800 at least includes: a processor 801 and a storage medium 802 configured to store executable instructions, wherein:
the processor 801 is configured to execute stored executable instructions configured to perform the data reading and writing methods provided in any of the embodiments described above.
It should be noted that the description of the terminal embodiment is similar to the description of the method embodiment, and has similar beneficial effects to the method embodiment, and therefore, the description is not repeated. For technical details not disclosed in the terminal embodiments of the present invention, reference is made to the description of the method embodiments of the present invention for understanding.
Correspondingly, an embodiment of the present invention provides a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and the computer-executable instructions are configured to execute the data reading and writing method provided in another embodiment of the present invention.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (12)
1. A method for reading and writing data, the method comprising:
receiving a data reading instruction, wherein the data reading instruction comprises characteristic information of data to be read;
determining a hit data index in a memory according to the characteristic information;
and reading data in a cache according to the hit data index, wherein the cache is a Solid State Disk (SSD).
2. The method of claim 1, wherein determining the hit data index in the memory based on the characterization information comprises:
searching a data index in the memory according to the characteristic information;
and when the data index is found in the memory, determining the found data index as the hit data index.
3. The method according to claim 2, wherein the characteristic information of the data to be read includes a location of the data to be read, and the searching the data index in the memory according to the characteristic information includes:
acquiring a data index table from the memory;
and searching a data index in the data index table according to the position of the data to be read.
4. The method of claim 3, wherein reading data in a cache according to the hit data index comprises:
and when the hit data index is determined in the memory, reading the data corresponding to the position of the data to be read.
5. The method according to any one of claims 1 to 4, wherein the data to be read is stored in the SSD in the form of a Log Log, wherein the Log comprises metadata and actual data, and the metadata comprises the characteristic information; the method further comprises the following steps:
when the data to be read is hot data, or the data to be read is not the hot data and the actual data in the data to be read is not changed, not updating the metadata in the Log;
and when the data to be read is not hotspot data and the actual data in the data to be read changes, updating the metadata in the Log.
6. A method for reading and writing data, the method comprising:
receiving a data writing instruction, wherein the data writing instruction comprises characteristic information of data to be written;
determining a hit data index in a memory according to the characteristic information;
and updating the pointed position of the hit data index in a cache according to the characteristic information, wherein the cache is a Solid State Disk (SSD).
7. The method of claim 6, wherein determining the hit data index in the memory based on the characterization information comprises:
searching a data index in the memory according to the characteristic information;
and when the data index is found in the memory, determining the found data index as the hit data index.
8. The method according to claim 7, wherein the characteristic information of the data to be written includes a location of the data to be written, and the searching the data index in the memory according to the characteristic information includes:
acquiring a data index table from the memory;
and searching a data index in the data index table according to the position of the data to be written.
9. The method of claim 8, wherein updating the hit data index according to the characterization information comprises: when the hit data index is determined in the memory, updating the pointed position of the hit data index in the SSD according to the position of the data to be written;
alternatively, the method further comprises: when the hit data index is not determined in the cache according to the characteristic information, the data index is regenerated according to the characteristic information;
or before the determining, according to the feature information, a hit data index in the cache, the method further includes: and when the data to be written is the hot data, persistently writing the data to be written into the SSD.
10. The method according to claims 6 to 9, wherein the data to be written is written in the SSD in the form of Log, wherein the Log comprises metadata and actual data, and the metadata comprises the characteristic information.
11. A terminal, characterized in that the terminal comprises at least: a processor and a storage medium configured to store executable instructions, wherein: the processor is configured to execute stored executable instructions;
the executable instructions are configured to perform the data reading and writing method provided by any one of the above claims 1 to 5 or claims 6 to 10.
12. A computer-readable storage medium having computer-executable instructions stored therein, the computer-executable instructions being configured to perform the method for reading and writing data as provided in any one of claims 1 to 5 or claims 6 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810751082.1A CN110703981A (en) | 2018-07-10 | 2018-07-10 | Data reading and writing method, terminal and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810751082.1A CN110703981A (en) | 2018-07-10 | 2018-07-10 | Data reading and writing method, terminal and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110703981A true CN110703981A (en) | 2020-01-17 |
Family
ID=69192485
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810751082.1A Pending CN110703981A (en) | 2018-07-10 | 2018-07-10 | Data reading and writing method, terminal and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110703981A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102364474A (en) * | 2011-11-17 | 2012-02-29 | 中国科学院计算技术研究所 | Metadata storage system for cluster file system and metadata management method |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| US20140351490A1 (en) * | 2013-05-22 | 2014-11-27 | Industry-Academic Cooperation Foundation, Yonsei University | Method for updating inverted index of flash ssd |
| CN104267912A (en) * | 2014-09-19 | 2015-01-07 | 北京联创信安科技有限公司 | NAS (Network Attached Storage) accelerating method and system |
| CN105574104A (en) * | 2015-12-11 | 2016-05-11 | 上海爱数信息技术股份有限公司 | LogStructure storage system based on ObjectStore and data writing method thereof |
| CN106201911A (en) * | 2015-05-05 | 2016-12-07 | 苏州携云信息科技有限公司 | A kind of caching accelerated method based on solid state hard disc |
| CN106708443A (en) * | 2017-01-03 | 2017-05-24 | 北京百度网讯科技有限公司 | Data reading and writing method and device |
-
2018
- 2018-07-10 CN CN201810751082.1A patent/CN110703981A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| CN102364474A (en) * | 2011-11-17 | 2012-02-29 | 中国科学院计算技术研究所 | Metadata storage system for cluster file system and metadata management method |
| US20140351490A1 (en) * | 2013-05-22 | 2014-11-27 | Industry-Academic Cooperation Foundation, Yonsei University | Method for updating inverted index of flash ssd |
| CN104267912A (en) * | 2014-09-19 | 2015-01-07 | 北京联创信安科技有限公司 | NAS (Network Attached Storage) accelerating method and system |
| CN106201911A (en) * | 2015-05-05 | 2016-12-07 | 苏州携云信息科技有限公司 | A kind of caching accelerated method based on solid state hard disc |
| CN105574104A (en) * | 2015-12-11 | 2016-05-11 | 上海爱数信息技术股份有限公司 | LogStructure storage system based on ObjectStore and data writing method thereof |
| CN106708443A (en) * | 2017-01-03 | 2017-05-24 | 北京百度网讯科技有限公司 | Data reading and writing method and device |
Non-Patent Citations (2)
| Title |
|---|
| 王江涛等: "闪存数据库：现状、技术与展望", 《计算机学报》 * |
| 罗圣美等: "一种结合SSD特征的分布式文件系统元数据优化技术", 《小型微型计算机系统》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10303596B2 (en) | Read-write control method for memory, and corresponding memory and server | |
| CN110825748B (en) | High-performance and easily-expandable key value storage method by utilizing differentiated indexing mechanism | |
| US11301379B2 (en) | Access request processing method and apparatus, and computer device | |
| CN105574104B (en) | A kind of LogStructure storage system and its method for writing data based on ObjectStore | |
| CN102541968B (en) | Indexing method | |
| CN108459826B (en) | Method and device for processing IO (input/output) request | |
| US20140297603A1 (en) | Method and apparatus for deduplication of replicated file | |
| EP3316150B1 (en) | Method and apparatus for file compaction in key-value storage system | |
| WO2014015828A1 (en) | Data storage space processing method and processing system, and data storage server | |
| CN107329692B (en) | Data deduplication method and storage device | |
| KR20200122994A (en) | Key Value Append | |
| CN107391544B (en) | Processing method, device and equipment of column type storage data and computer storage medium | |
| US11030092B2 (en) | Access request processing method and apparatus, and computer system | |
| CN107135662B (en) | Differential data backup method, storage system and differential data backup device | |
| CN102567415B (en) | Control method and device of database | |
| CN109407985B (en) | Data management method and related device | |
| CN112346659B (en) | Storage method, equipment and storage medium for distributed object storage metadata | |
| CN110134551B (en) | Continuous data protection method and device | |
| CN102508790A (en) | Content-based cache method applied to content analysis storage | |
| CN111831691A (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
| CN110703981A (en) | Data reading and writing method, terminal and computer readable storage medium | |
| CN114647658A (en) | Data retrieval method, device, equipment and machine-readable storage medium | |
| CN117519612B (en) | Mass small file storage system and method based on index online splicing | |
| CN112597074B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200117 |