CN112799589A - Data reading method and device - Google Patents
Data reading method and device Download PDFInfo
- Publication number
- CN112799589A CN112799589A CN202110049279.2A CN202110049279A CN112799589A CN 112799589 A CN112799589 A CN 112799589A CN 202110049279 A CN202110049279 A CN 202110049279A CN 112799589 A CN112799589 A CN 112799589A
- Authority
- CN
- China
- Prior art keywords
- target data
- data
- read
- reading
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0862—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with prefetch
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The present disclosure relates to the field of data storage technologies, and in particular, to a data reading method and apparatus. The method is applied to the client in the distributed storage system, the distributed storage system comprises a plurality of clients and a plurality of servers, and the method comprises the following steps: receiving a data reading and writing request sent by a user, and judging whether the data reading and writing request is sequential reading or not; and if the read-write request is judged to be sequential read, determining address information of target data to be prefetched based on a preset prefetching algorithm, and sending the address information of the target data to a server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data to a local memory.
Description
Technical Field
The present disclosure relates to the field of data storage technologies, and in particular, to a data reading method and apparatus.
Background
In a storage system, storage performance is a very important index, in order to improve the reading performance of storage, a sequential flow identification algorithm is often adopted, once a sequential flow is identified, a background prefetches data from a disk to a Cache in advance, so that subsequent reading is guaranteed to directly hit in the Cache, and the reading performance is greatly improved.
However, in the distributed storage system, it is assumed that the distributed storage system includes a client: client1, Client 2; the method comprises the following steps: server1, Server2, Server 3. When multiple clients are running, if the Cache prefetches the latest data in the client1, but the client2 renews the data, but the client1 is not aware, so that the data that the client1 finally reads from the Cache is old data, and therefore the data that the user reads is not the latest data.
Disclosure of Invention
The application provides a data reading method and device, which are used for solving the problem that pre-fetched data is not the latest data when a plurality of clients in a distributed storage system run in the prior art.
In a first aspect, the present application provides a data reading method, which is applied to a client in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the method includes:
receiving a data reading and writing request sent by a user, and judging whether the data reading and writing request is sequential reading or not;
and if the read-write request is judged to be sequential read, determining address information of target data to be prefetched based on a preset prefetching algorithm, and sending the address information of the target data to a server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data to a local memory.
Optionally, the method further comprises:
and receiving a target data read-write request sent by the user, reading the target data from the memory of the server based on the target data read-write request, and returning the target data to the user.
In a second aspect, the present application provides another data reading method, which is applied to a server in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the method includes:
receiving address information of target data issued by a client, wherein when the client judges that a received data read-write request sent by a user is sequential read, the client determines the address information of the target data to be prefetched based on a preset prefetching algorithm;
and reading the target data from the disk based on the address information of the target data, and storing the target data to a local memory.
Optionally, the method further comprises:
receiving a target data read-write request issued by the client;
and reading the target data from a local memory based on the target data read-write request, and returning the target data to the client.
Optionally, the method further comprises:
and if the data cached in the local memory is judged to be larger than the first set value, deleting the data cached in the local memory, the data heat of which is smaller than the second set value.
In a third aspect, the present application provides a data reading apparatus, which is applied to a client in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the apparatus includes:
the receiving unit is used for receiving a data read-write request sent by a user;
the judging unit is used for judging whether the data reading and writing request is sequential reading or not;
and the determining unit determines the address information of target data to be prefetched based on a preset prefetching algorithm if the judging unit judges that the read-write request is sequential read, and sends the address information of the target data to the server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data to a local memory.
Optionally, the receiving unit is further configured to receive a target data read-write request sent by the user, read the target data from the memory of the server based on the target data read-write request, and return the target data to the user.
In a fourth aspect, the present application provides another data reading apparatus, which is applied to a server in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the apparatus includes:
the system comprises a receiving unit, a pre-fetching unit and a pre-fetching unit, wherein the receiving unit is used for receiving address information of target data issued by a client, and the client determines the address information of the target data to be pre-fetched based on a preset pre-fetching algorithm when the received data read-write request sent by a user is judged to be sequential read;
and the reading unit is used for reading the target data from the disk based on the address information of the target data and storing the target data to a local memory.
Optionally, the receiving unit is further configured to receive a target data read-write request issued by the client;
the reading unit is further configured to read the target data from a local memory based on the target data read-write request, and return the target data to the client.
Optionally, the apparatus further comprises:
and the deleting unit is used for deleting the data of which the data heat degree is less than a second set value cached in the local memory if the data cached in the local memory is judged to be more than the first set value.
In a fifth aspect, an embodiment of the present application provides a client, where the client includes:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the steps of the method according to any one of the above first aspects in accordance with the obtained program instructions.
In a sixth aspect, the present application further provides a computer-readable storage medium, which stores computer-executable instructions for causing the computer to perform the steps of the method according to any one of the above first aspects.
In a seventh aspect, an embodiment of the present application provides a server, where the server includes:
a memory for storing program instructions;
a processor for calling the program instructions stored in the memory and executing the steps of the method according to any one of the above second aspects in accordance with the obtained program instructions.
In an eighth aspect, the present embodiments also provide a computer-readable storage medium, which stores computer-executable instructions for causing the computer to perform the steps of the method according to any one of the above second aspects.
In summary, the data reading method provided in the embodiment of the present application is applied to a client in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the method includes: receiving a data reading and writing request sent by a user, and judging whether the data reading and writing request is sequential reading or not; and if the read-write request is judged to be sequential read, determining address information of target data to be prefetched based on a preset prefetching algorithm, and sending the address information of the target data to a server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data to a local memory.
By adopting the data reading method provided by the embodiment of the application, the sequential flow identification and pre-fetching algorithm is deployed at the client side in the distributed storage system, and the data cache is deployed at the server side, so that the problem of inconsistent cache data when multiple client sides in the distributed storage system operate is solved, the sequential flow identification and pre-fetching functions can be started under the scene of multiple client sides, and the performance of sequential read IO with the same distributed storage performance is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present application.
Fig. 1 is a detailed flowchart of a data reading method according to an embodiment of the present application;
FIG. 2 is a detailed flowchart of another data reading method according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a distributed storage system according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of another data reading apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a client according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
Exemplarily, referring to fig. 1, a detailed flowchart of a data reading method provided in an embodiment of the present application is shown, where the method is applied to a client in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the method includes the following steps:
step 100: receiving a data read-write request sent by a user, and judging whether the data read-write request is sequential read.
In the embodiment of the application, a sequential flow identification function and a prefetching function are deployed on each client of a distributed storage system, so that when any client receives a data read-write request sent by a user, a current data read-write request is identified based on a preset sequential flow identification method to determine whether the current data read-write request is sequential read.
It should be noted that the sequential flow recognition must be put to the client, because the client can see the whole RBD (which can be regarded as a logical disk), it is split according to 4M, and then it is respectively distributed to different object servers through pseudo-random algorithm, and the servers cannot see the whole view, so only the client can be selected to perform the sequential flow recognition.
Step 110: and if the read-write request is judged to be sequential read, determining the address information of target data to be prefetched based on a preset prefetching algorithm, and sending the address information of the target data to a server, so that the server prefetches the target data from a disk based on the address information of the target data and stores the target data to a local memory.
Specifically, since each client in the distributed storage system is deployed with a prefetching function, if the client determines that the current data read-write request is read sequentially, first, target data that needs to be read in advance and cached in the cache region is determined based on a preset prefetching algorithm (i.e., information such as address information of the target data and length of the target data is determined), and then, address information of the target data that needs to be prefetched is sent to a corresponding server, so that the server reads the target data from a corresponding OSD (e.g., a disk) based on the address information of the target data, and caches the read target data in the local memory.
In the embodiment of the application, the data caching function is deployed on the server, and the server directly stores the pre-fetched target data in the local memory, so that the target data is directly read from the memory and returned to the client when a target data reading request sent by the client is subsequently received.
As can be seen from the above, when a client writes data, firstly, data to be written is cached in a memory of the client, then, the cached data in the memory is downloaded to an OSD based on a preset rule, then, it is assumed that a client (e.g., client 1) reads data 1 and data 2 in advance, a server caches data 1 and data 2 in a local memory, and if the client2 performs a modification operation on data 1 at this time, firstly, modified data 1 is cached in the memory, and then, an operation of replacing original data 1 with modified data 1 is implemented, so when the client1 reads data 1 and data 2 from the memory of the client, finally, read data 1 is the newest data, and a situation that read data 1 is unmodified data 1 does not occur. The data cached in the memory of the server side are all the latest data, and the problem of data consistency is solved.
In the embodiment of the present application, a preferred implementation manner is that the prefetching function is deployed at the client, so that resource consumption of the server can be reduced, and thus data read-write performance is improved.
Further, if the client receives a target data read-write request sent by the user, the target data is read from the memory of the server based on the target data read-write request, and the target data is returned to the user.
Because the server caches the target data to be read subsequently by the user in the local memory through the pre-fetching calculation, when the client receives the target data read-write request sent by the user, the target data read-write request can be forwarded to the server, the server directly reads the target data from the local memory according to the target read-write request, and forwards the read target data to the client, and then the client forwards the target data to the user.
Exemplarily, referring to fig. 2, a detailed flowchart of another data reading method provided in an embodiment of the present application is shown, where the method is applied to a server in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, and the method includes the following steps:
step 200: and receiving address information of target data issued by a client, wherein when the client judges that the received data read-write request sent by a user is sequential read, the address information of the target data to be prefetched is determined based on a preset prefetching algorithm.
Step 210: and reading the target data from the disk based on the address information of the target data, and storing the target data to a local memory.
Further, in this embodiment of the application, the data reading method applied to the server may further include the following steps:
and if the data cached in the local memory is judged to be larger than the first set value, deleting the data cached in the local memory, the data heat of which is smaller than the second set value.
In the embodiment of the present application, the data caching function is deployed at the server and limited by the memory resource, and the configured space for caching data is fixed, so that the cache region is used up, and at this time, data with low heat in the cache region needs to be eliminated according to a preset rule, for example, based on an LRU algorithm, data with low heat in the cache region is eliminated to vacate a space for caching other data.
The following describes the structure of the distributed storage system provided in the embodiment of the present application in detail with reference to a specific application scenario. Exemplarily, referring to fig. 3, a schematic structural diagram of a distributed storage system provided in this embodiment is shown, where the distributed storage system includes a client1, a client2, a server1, a server2, and a server3, the client1 is respectively in communication connection with the client1, the client2, and the client 3, the client2 is respectively connected with the client1, the client2, and the client 3, each client is preconfigured with a sequential stream recognition function and a prefetching algorithm, when receiving a data read request sent by a user, the algorithm is preset to recognize whether the data read request is a sequential read operation, if so, the preset prefetching algorithm is adopted to calculate target data to be prefetched, and a calculation result is notified to the corresponding server, so that the server performs a prefetching operation of the target data, in this embodiment, each server is preconfigured with a data caching function, the server side directly obtains the target data from the local memory based on the target data reading request, sends the target data to the client side, and the client side directly returns the target data to the user.
Based on the same inventive concept as the above method applied to the client, for example, referring to fig. 4, a schematic structural diagram of a data reading apparatus provided in the embodiment of the present application is shown, where the apparatus is applied to a client in a distributed storage system, the distributed storage system includes a plurality of clients and a plurality of servers, and the apparatus includes:
a receiving unit 40, configured to receive a data read-write request sent by a user;
a judging unit 41, configured to judge whether the data read-write request is sequential read;
if the determining unit 42 determines that the read-write request is sequential read, based on a preset prefetching algorithm, the determining unit 42 determines address information of target data to be prefetched, and sends the address information of the target data to the server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data in a local memory.
Alternatively,
the receiving unit 40 is further configured to receive a target data read-write request sent by the user, read the target data from the memory of the server based on the target data read-write request, and return the target data to the user.
Based on the same inventive concept as the above-mentioned method applied to the server, exemplarily, refer to fig. 5, which is a schematic structural diagram of another data reading apparatus provided in the embodiment of the present application, the apparatus is applied to a server in a distributed storage system, the distributed storage system includes a plurality of clients and a plurality of servers, and the apparatus includes:
the receiving unit 50 is configured to receive address information of target data sent by a client, where the client determines, based on a preset prefetching algorithm, address information of the target data to be prefetched when it determines that a received data read-write request sent by a user is sequential read;
the reading unit 51 is configured to read target data from a disk based on the address information of the target data, and store the target data in a local memory.
Alternatively,
the receiving unit 50 is further configured to receive a target data read-write request issued by the client;
the reading unit 51 is further configured to read the target data from a local memory based on the target data read-write request, and return the target data to the client.
Optionally, the apparatus further comprises:
and the deleting unit is used for deleting the data of which the data heat degree is less than a second set value cached in the local memory if the data cached in the local memory is judged to be more than the first set value.
The above units may be one or more integrated circuits configured to implement the above methods, for example: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above units is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Further, an embodiment of the present application provides a client device, and from a hardware level, a hardware architecture diagram of the client device may be as shown in fig. 6, where the client device may include: a memory 60 and a processor 61, which,
the memory 60 is used to store program instructions; the processor 61 calls the program instructions stored in the memory 60 and executes the above-described method embodiment applied to the client according to the obtained program instructions. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the present application also provides a client comprising at least one processing element (or chip) for performing the above-mentioned method embodiment applied to the client.
Optionally, the present application also provides a program product, such as a computer-readable storage medium, storing computer-executable instructions for causing a computer to perform the above-described method embodiments applied to a client.
Further, in the embodiment of the present application, a server is provided, and from a hardware level, a hardware architecture diagram of the server may be shown in fig. 7, where the server may include: a memory 70 and a processor 71, which,
the memory 70 is used to store program instructions; the processor 71 calls the program instructions stored in the memory 70 and executes the above-mentioned method embodiment applied to the server according to the obtained program instructions. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the present application further provides a server, which includes at least one processing element (or chip) for executing the above method embodiment applied to the server.
Optionally, the present application also provides a program product, such as a computer-readable storage medium, which stores computer-executable instructions for causing the computer to execute the above-mentioned method embodiment applied to the server.
Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and so forth. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (10)
1. A data reading method is applied to a client in a distributed storage system, the distributed storage system comprises a plurality of clients and a plurality of servers, and the method comprises the following steps:
receiving a data reading and writing request sent by a user, and judging whether the data reading and writing request is sequential reading or not;
and if the read-write request is judged to be sequential read, determining address information of target data to be prefetched based on a preset prefetching algorithm, and sending the address information of the target data to a server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data to a local memory.
2. The method of claim 1, wherein the method further comprises:
and receiving a target data read-write request sent by the user, reading the target data from the memory of the server based on the target data read-write request, and returning the target data to the user.
3. A data reading method is applied to a server in a distributed storage system, wherein the distributed storage system comprises a plurality of clients and a plurality of servers, and the method comprises the following steps:
receiving address information of target data issued by a client, wherein when the client judges that a received data read-write request sent by a user is sequential read, the client determines the address information of the target data to be prefetched based on a preset prefetching algorithm;
and reading the target data from the disk based on the address information of the target data, and storing the target data to a local memory.
4. The method of claim 3, wherein the method further comprises:
receiving a target data read-write request issued by the client;
and reading the target data from a local memory based on the target data read-write request, and returning the target data to the client.
5. The method of claim 3 or 4, further comprising:
and if the data cached in the local memory is judged to be larger than the first set value, deleting the data cached in the local memory, the data heat of which is smaller than the second set value.
6. A data reading apparatus, applied to a client in a distributed storage system, where the distributed storage system includes a plurality of clients and a plurality of servers, the apparatus comprising:
the receiving unit is used for receiving a data read-write request sent by a user;
the judging unit is used for judging whether the data reading and writing request is sequential reading or not;
and the determining unit determines the address information of target data to be prefetched based on a preset prefetching algorithm if the judging unit judges that the read-write request is sequential read, and sends the address information of the target data to the server, so that the server prefetches the target data from a disk based on the address information of the target data, and stores the target data to a local memory.
7. The apparatus of claim 6,
the receiving unit is further configured to receive a target data read-write request sent by the user, read the target data from the memory of the server based on the target data read-write request, and return the target data to the user.
8. A data reading device is applied to a server in a distributed storage system, wherein the distributed storage system comprises a plurality of clients and a plurality of servers, and the device comprises:
the system comprises a receiving unit, a pre-fetching unit and a pre-fetching unit, wherein the receiving unit is used for receiving address information of target data issued by a client, and the client determines the address information of the target data to be pre-fetched based on a preset pre-fetching algorithm when the received data read-write request sent by a user is judged to be sequential read;
and the reading unit is used for reading the target data from the disk based on the address information of the target data and storing the target data to a local memory.
9. The apparatus of claim 8,
the receiving unit is further configured to receive a target data read-write request issued by the client;
the reading unit is further configured to read the target data from a local memory based on the target data read-write request, and return the target data to the client.
10. The apparatus of claim 8 or 9, wherein the apparatus further comprises:
and the deleting unit is used for deleting the data of which the data heat degree is less than a second set value cached in the local memory if the data cached in the local memory is judged to be more than the first set value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110049279.2A CN112799589B (en) | 2021-01-14 | 2021-01-14 | Data reading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110049279.2A CN112799589B (en) | 2021-01-14 | 2021-01-14 | Data reading method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112799589A true CN112799589A (en) | 2021-05-14 |
| CN112799589B CN112799589B (en) | 2023-07-14 |
Family
ID=75810813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110049279.2A Active CN112799589B (en) | 2021-01-14 | 2021-01-14 | Data reading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112799589B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113703666A (en) * | 2021-07-14 | 2021-11-26 | 新华三大数据技术有限公司 | Data reading and writing method and device |
| CN115481187A (en) * | 2022-09-28 | 2022-12-16 | 北京亚控科技发展有限公司 | Data reading and writing method and device and storage medium |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH1027069A (en) * | 1996-07-09 | 1998-01-27 | Hitachi Ltd | Storage device |
| CN101377788A (en) * | 2008-09-28 | 2009-03-04 | 中国科学院计算技术研究所 | Method and system of caching management in cluster file system |
| CN101388824A (en) * | 2008-10-15 | 2009-03-18 | 中国科学院计算技术研究所 | File reading method and system under sliced memory mode in cluster system |
| CN102111448A (en) * | 2011-01-13 | 2011-06-29 | 华为技术有限公司 | Data prefetching method of DHT memory system and node and system |
| CN102541985A (en) * | 2011-10-25 | 2012-07-04 | 曙光信息产业(北京)有限公司 | Organization method of client directory cache in distributed file system |
| US8667224B1 (en) * | 2007-12-20 | 2014-03-04 | Emc Corporation | Techniques for data prefetching |
| CN103635887A (en) * | 2013-09-23 | 2014-03-12 | 华为技术有限公司 | Data caching method and storage system |
| CN103795781A (en) * | 2013-12-10 | 2014-05-14 | 西安邮电大学 | Distributed cache model based on file prediction |
| CN105573667A (en) * | 2015-12-10 | 2016-05-11 | 华为技术有限公司 | Data reading method and storage server |
| US20170031823A1 (en) * | 2015-07-31 | 2017-02-02 | Oracle International Corporation | Systems and methods for prefetching data |
| US20180275919A1 (en) * | 2015-12-23 | 2018-09-27 | Hewlett Packard Enterprise Development Lp | Prefetching data in a distributed storage system |
| CN109918023A (en) * | 2019-02-22 | 2019-06-21 | 深圳忆联信息系统有限公司 | It is limited pre-reading method, device and the computer equipment of SSD based on memory |
| CN110008142A (en) * | 2019-04-16 | 2019-07-12 | 苏州浪潮智能科技有限公司 | Storing data management method, device, equipment and computer readable storage medium |
| CN110737388A (en) * | 2018-07-18 | 2020-01-31 | 中兴通讯股份有限公司 | Data pre-reading method, client, server and file system |
| CN110764708A (en) * | 2019-10-25 | 2020-02-07 | 北京浪潮数据技术有限公司 | Data reading method, device, equipment and storage medium |
| CN112328185A (en) * | 2020-12-28 | 2021-02-05 | 烽火通信科技股份有限公司 | Intelligent pre-reading method based on distributed storage |
-
2021
- 2021-01-14 CN CN202110049279.2A patent/CN112799589B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH1027069A (en) * | 1996-07-09 | 1998-01-27 | Hitachi Ltd | Storage device |
| US8667224B1 (en) * | 2007-12-20 | 2014-03-04 | Emc Corporation | Techniques for data prefetching |
| CN101377788A (en) * | 2008-09-28 | 2009-03-04 | 中国科学院计算技术研究所 | Method and system of caching management in cluster file system |
| CN101388824A (en) * | 2008-10-15 | 2009-03-18 | 中国科学院计算技术研究所 | File reading method and system under sliced memory mode in cluster system |
| CN102111448A (en) * | 2011-01-13 | 2011-06-29 | 华为技术有限公司 | Data prefetching method of DHT memory system and node and system |
| CN102541985A (en) * | 2011-10-25 | 2012-07-04 | 曙光信息产业(北京)有限公司 | Organization method of client directory cache in distributed file system |
| CN103635887A (en) * | 2013-09-23 | 2014-03-12 | 华为技术有限公司 | Data caching method and storage system |
| CN103795781A (en) * | 2013-12-10 | 2014-05-14 | 西安邮电大学 | Distributed cache model based on file prediction |
| US20170031823A1 (en) * | 2015-07-31 | 2017-02-02 | Oracle International Corporation | Systems and methods for prefetching data |
| CN105573667A (en) * | 2015-12-10 | 2016-05-11 | 华为技术有限公司 | Data reading method and storage server |
| US20180275919A1 (en) * | 2015-12-23 | 2018-09-27 | Hewlett Packard Enterprise Development Lp | Prefetching data in a distributed storage system |
| CN110737388A (en) * | 2018-07-18 | 2020-01-31 | 中兴通讯股份有限公司 | Data pre-reading method, client, server and file system |
| CN109918023A (en) * | 2019-02-22 | 2019-06-21 | 深圳忆联信息系统有限公司 | It is limited pre-reading method, device and the computer equipment of SSD based on memory |
| CN110008142A (en) * | 2019-04-16 | 2019-07-12 | 苏州浪潮智能科技有限公司 | Storing data management method, device, equipment and computer readable storage medium |
| CN110764708A (en) * | 2019-10-25 | 2020-02-07 | 北京浪潮数据技术有限公司 | Data reading method, device, equipment and storage medium |
| CN112328185A (en) * | 2020-12-28 | 2021-02-05 | 烽火通信科技股份有限公司 | Intelligent pre-reading method based on distributed storage |
Non-Patent Citations (3)
| Title |
|---|
| YANG HONGZHANG: "Research of Massive Small Files Reading Optimization Based on Parallel Network File System", 《IEEE XPLORE》 * |
| 方娟: "RLCP:异构的协同预取存储设计方案", 《计算机工程与设计》 * |
| 闫鹤: "分布式文件系统的流式数据预读", 《万方数据库》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113703666A (en) * | 2021-07-14 | 2021-11-26 | 新华三大数据技术有限公司 | Data reading and writing method and device |
| CN115481187A (en) * | 2022-09-28 | 2022-12-16 | 北京亚控科技发展有限公司 | Data reading and writing method and device and storage medium |
| CN115481187B (en) * | 2022-09-28 | 2023-11-21 | 北京亚控科技发展有限公司 | Data reading and writing method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112799589B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112799589B (en) | Data reading method and device | |
| CN107197359B (en) | Video file caching method and device | |
| US20210073134A1 (en) | Input/output patterns and data pre-fetch | |
| CN111324556B (en) | Method and system for prefetching a predetermined number of data items into a cache | |
| Kim et al. | Analysis of smartphone I/O characteristics—Toward efficient swap in a smartphone | |
| CN101404649B (en) | Data processing system based on CACHE and its method | |
| CN112000426A (en) | Data processing method and device | |
| CN111951275A (en) | Image transmission method and device | |
| CN114647658A (en) | Data retrieval method, device, equipment and machine-readable storage medium | |
| CN106649143B (en) | Cache access method and device and electronic equipment | |
| CN114442931A (en) | Data deduplication method and system, electronic device and storage medium | |
| US9971693B2 (en) | Prefetch tag for eviction promotion | |
| CN114416178A (en) | Data access method, device and non-transitory computer readable storage medium | |
| CN108874690B (en) | Data prefetching implementation method and processor | |
| CN113485642A (en) | Data caching method and device | |
| CN112199400B (en) | Method and device for data processing | |
| CN112256206B (en) | IO processing method and device | |
| CN115357779A (en) | Data processing method, device and system, electronic equipment and storage medium | |
| CN112596825B (en) | Cloud desktop starting method and device | |
| CN112115393B (en) | Page loading method, system, device and medium | |
| CN114637946A (en) | Resource data processing method and device and electronic equipment | |
| CN110825652B (en) | Method, device and equipment for eliminating cache data on disk block | |
| CN114139097A (en) | Webpage loading method and device | |
| CN112800123A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN113194118B (en) | Sequential flow identification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |