CN113254415B - Method and device for processing read request of distributed file system - Google Patents
Method and device for processing read request of distributed file system Download PDFInfo
- Publication number
- CN113254415B CN113254415B CN202110546268.5A CN202110546268A CN113254415B CN 113254415 B CN113254415 B CN 113254415B CN 202110546268 A CN202110546268 A CN 202110546268A CN 113254415 B CN113254415 B CN 113254415B
- Authority
- CN
- China
- Prior art keywords
- data
- host
- read request
- read
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 50
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000000977 initiatory effect Effects 0.000 claims abstract description 36
- 238000013507 mapping Methods 0.000 claims description 7
- 238000003672 processing method Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for processing a read request of a distributed file system. In the invention, if the data to be read has copy data in the host initiating the read request, the data is directly read from the host, and compared with the prior art, the data needs to be read across the network under the condition that the copy data exists in the host initiating the read request, thereby optimizing the read performance of the distributed file system.
Description
Technical Field
The present invention relates to the field of distributed system technologies, and in particular, to a method and an apparatus for processing read requests of a distributed file system.
Background
The Hyper Converged Infrastructure (HCI) is an architecture that integrates virtual computing resources and storage device Infrastructure information. One set of HCI equipment simultaneously has resources and technologies such as calculation, storage, network, virtualization and the like, and multiple sets of equipment can be easily expanded transversely to form a uniform resource pool. The main components of the super-convergence infrastructure comprise computing virtualization, network virtualization and storage virtualization.
The distributed file system is mainly used for virtualizing storage resources of equipment, performing unified management and distribution in a super-convergence infrastructure, and mainly can provide related storage services for the outside. Another main function of the distributed file system is to provide a system disk required for starting a virtual machine in the architecture or a data disk for read-write operation, so that correspondingly, the read-write performance of the distributed file system may affect the system performance to some extent, for example, the start speed or the read-write performance of the virtual machine may be affected.
Disclosure of Invention
The invention aims to provide a method and a device for processing a read request of a distributed file system, which optimize the read performance of the distributed file system.
In order to achieve the purpose, the invention provides the following technical scheme:
a method for processing read requests of a distributed file system comprises the following steps:
acquiring host information for initiating a read request;
obtaining host information of original data of data to be read in the read request and host information of copy data to form a host set;
and checking whether the host set contains a host initiating the reading request, and if so, reading data from the host according to the reading request.
Preferably, the obtaining the host information initiating the read request includes: and according to the reading request, obtaining host information for initiating the reading request by accessing a preset configuration item, wherein the preset configuration item is used for calculating the host information for initiating the request.
Preferably, if the host set includes a host initiating the read request, the process corresponding to the host and used for processing the request or the data is returned, so as to read the data through the process.
Preferably, if the host set does not include a host initiating the read request, the data is read from the host where the original data of the data to be read is located.
Preferably, if the host set does not include the host initiating the read request, a process for processing the request or the data corresponding to the host where the original data of the data to be read is located is returned, so as to read the data through the process.
Preferably, the method specifically comprises the following steps: and obtaining the host information of the original data of the data to be read and the host information of the copy data according to the mapping formed by the data and the directory for storing the data and the mapping formed by the directory for storing the data and the host for storing the data.
A distributed file system read request processing device is used for executing the distributed file system read request processing method.
Preferably, the method comprises the following steps:
the first processing layer is used for creating or deleting a channel for communicating with the server;
and the second processing layer is used for acquiring host information for initiating the read request according to the read request, acquiring host information of original data and host information of copy data of data to be read by the read request, and sending the read request to the server through a channel.
Preferably, the second processing layer is further configured to determine whether the server returns data to be read by the read request, and if not, send the read request to the server again, and if the server has not returned the data to be read by the read request within a preset time period after the read request is sent to the server for the first time, stop sending the read request to the server.
Preferably, the device is located at a client.
According to the technical scheme, the method and the device for processing the read request of the distributed file system firstly acquire the host information initiating the read request, acquire the host information of the original data of the data to be read in the read request and the host information of the duplicate data to form a host set, further check whether the host set comprises the host initiating the read request, and if so, read the data from the host according to the read request.
According to the method and the device for processing the read request of the distributed file system, if the data to be read has the copy data in the host initiating the read request, the data is directly read from the host.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a method for processing a read request of a distributed file system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a distributed file system in an embodiment of the invention;
fig. 3 is a schematic layout diagram of a distributed file system read request processing apparatus according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a read request processing method for a distributed file system according to this embodiment, and it can be seen from the figure that the read request processing method for the distributed file system includes the following steps:
s10: host information for initiating a read request is obtained.
Optionally, configuration items for computing host information for initiating requests may be added. When a read request is received, host information for initiating the read request is obtained by accessing a preset configuration item according to the read request, wherein the preset configuration item is used for calculating the host information for initiating the request.
For example, the preset configuration item may be described as:
[client]crush_location＝‘host＝hostname root＝default’。
s11: and obtaining host information of original data and host information of copy data of the data to be read in the read request to form a host set.
In a distributed file system, raw data is stored on respective hosts and duplicate data is stored on each host separately. In this step, according to the read request, host information where original data of data to be read by the read request is located and host information where duplicate data of the data to be read is located are obtained, and each host information is obtained to form a set.
Optionally, in the distributed file system, the data and the directory for storing the data form a map, the directory for storing the data and the host for storing the data form a map, and each host corresponds to a process for processing a request or data. And the read/write operation command of the data falls into the process of the corresponding host through two layers of mapping.
In the method, the host information of the original data of the data to be read and the host information of the copy data can be obtained according to the mapping formed by the data and the catalogue for storing the data and the mapping formed by the catalogue for storing the data and the host for storing the data. Correspondingly, according to the obtained host set, the method can correspondingly obtain the processes for processing the requests or the data, which correspond to the hosts in the host set.
S12: checking whether a host initiating the read request is included in the set of hosts. If yes, the process proceeds to step S13.
S13: and reading data from the host according to the read request.
If the host set comprises the host initiating the read request, which indicates that the original data or the copy data of the data to be read is stored in the host initiating the read request, the data is directly read from the host according to the read request, and in this case, the data does not need to be read across the network.
Referring to fig. 2, fig. 2 is a schematic diagram of a distributed file system in this embodiment, in the prior art, when reading data in the distributed file system, only a process of a host where original data is located, that is, a host process of the data, provides a service, when initiating a read request, the host where the original data of the data to be read by the read request is located is obtained, and a read operation is executed through a process corresponding to the host. Therefore, if the original data of the data to be read is not local to the host, the data of other physical nodes needs to be read across the network, which increases the transmission of the data between networks, and causes serious network load, consumption of bandwidth resources, influence on the starting speed of the virtual machine and influence on the reading and writing performance of the virtual machine when a large number of virtual machines are started. According to the method for processing the read request of the distributed file system, whether the hosts initiating the read request include the host initiating the read request or not is checked by acquiring the host information of the original data of the data to be read in the read request and the host information of the copy data, and if the copy data exists in the host initiating the read request, the data is directly read from the host.
Specifically, if the host set includes a host initiating the read request, a process for processing the request or the data corresponding to the host is returned, so as to read the data through the process. And reading data from the host through the corresponding process of the host according to the read request.
In a further method of this embodiment, if the host set does not include a host that initiates the read request, step S14 is performed, and step S14: and reading data from the host where the original data of the data to be read is located.
Specifically, if the host set does not include the host initiating the read request, a process for processing the request or the data corresponding to the host where the original data of the data to be read is located is returned, so that the data is read through the process. And reading the data from the host by the corresponding process of the host where the original data of the data to be read is located according to the reading request.
The method for processing the read request of the distributed file system can directly read data from the host if the data to be read has copy data in the host initiating the read request under the condition of not changing the host process of each physical node, thereby optimizing the read performance of the distributed file system.
Correspondingly, the embodiment of the invention also provides a device for processing the read request of the distributed file system, which is used for executing the method for processing the read request of the distributed file system.
The read request processing apparatus of the distributed file system of this embodiment first obtains host information that initiates a read request, and obtains host information of original data and host information of replica data of data to be read in the read request, to form a host set, and further checks whether the host set includes a host that initiates the read request, and if so, reads data from the host according to the read request.
In the read request processing apparatus of the distributed file system of this embodiment, if there is duplicate data in the host initiating the read request, the data is directly read from the host, and compared with the prior art, it is avoided that data needs to be read across a network under such a situation, so that the read performance of the distributed file system is optimized.
Optionally, referring to fig. 3, fig. 3 is a schematic diagram of a read request processing apparatus of a distributed file system provided in this embodiment, as can be seen from the figure, the apparatus includes a first processing layer 200 and a second processing layer 201, where the first processing layer 200 is configured to create or delete a channel used for communicating with the server 21; the second processing layer 201 is configured to obtain, according to the read request, host information that initiates the read request, obtain host information that is located by original data of data to be read by the read request and host information that is located by duplicate data, and send the read request to the server 21 through a channel.
The first processing layer 200 and the second processing layer 201 may be located at a lower layer of the client 20, and a channel for communicating with the server 21 can be created or deleted by the first processing layer 200 providing an interface for establishing a communication connection with the server 21. Optionally, the second processing layer 201 may be configured to encapsulate and form a request, including a write request or a read request, and send the request to the server 21 through a channel.
Optionally, the second processing layer 201 is further configured to determine whether the server 21 returns data to be read by the read request, and if not, send the read request to the server 21 again, so as to improve reliability of performing the service. The second processing layer 201 is further configured to stop sending the read request to the server 21 if the server 21 has not returned the data to be read by the read request within a preset time period after the read request is sent to the server 21 for the first time. And if the data returned by the server side is not received within a certain time period, stopping requesting the reading operation.
The method and the device for processing the read request of the distributed file system provided by the invention are described in detail above. The principles and embodiments of the present invention have been described herein using specific examples, which are presented only to assist in understanding the method and its core concepts of the present invention. It should be noted that, for those skilled in the art, without departing from the principle of the present invention, it is possible to make various improvements and modifications to the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (9)
1. A method for processing a read request of a distributed file system is characterized by comprising the following steps:
acquiring host information for initiating a read request;
obtaining host information of original data of data to be read by the read request and host information of duplicate data to form a host set;
checking whether the host set contains a host initiating the read request, if so, reading data from the host according to the read request;
the obtaining of the host information where the original data of the data to be read by the read request is located and the host information where the copy data is located includes:
and obtaining the host information of the original data of the data to be read and the host information of the copy data according to the mapping formed by the data and the directory for storing the data and the mapping formed by the directory for storing the data and the host for storing the data.
2. The method of claim 1, wherein obtaining host information that initiates the read request comprises: and according to the reading request, obtaining host information for initiating the reading request by accessing a preset configuration item, wherein the preset configuration item is used for calculating the host information for initiating the request.
3. The method for processing the read request of the distributed file system according to claim 1, wherein if the host set includes a host initiating the read request, a process for processing the request or the data corresponding to the host is returned, so as to read the data through the process.
4. The method according to claim 1, wherein if the host set does not include a host that initiates the read request, reading data from a host in which original data of the data to be read is located.
5. The method for processing the read request of the distributed file system according to claim 4, wherein if the host set does not include a host initiating the read request, a process for processing the request or the data corresponding to the host where the original data of the data to be read is located is returned, so that the data is read through the process.
6. A distributed file system read request processing apparatus, configured to execute the distributed file system read request processing method according to any one of claims 1 to 5.
7. The distributed file system read request processing apparatus of claim 6, comprising:
the first processing layer is used for creating or deleting a channel for communicating with the server;
and the second processing layer is used for acquiring host information for initiating the read request according to the read request, acquiring host information of original data of data to be read by the read request and host information of copy data, and sending the read request to the server through a channel.
8. The device for processing the read request of the distributed file system according to claim 7, wherein the second processing layer is further configured to determine whether the server returns data to be read by the read request, and if not, send the read request to the server again, and if the server does not return the data to be read by the read request within a preset time period after the read request is sent to the server for the first time, stop sending the read request to the server.
9. The distributed file system read request processing apparatus of claim 6, wherein the apparatus is located at a client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110546268.5A CN113254415B (en) | 2021-05-19 | 2021-05-19 | Method and device for processing read request of distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110546268.5A CN113254415B (en) | 2021-05-19 | 2021-05-19 | Method and device for processing read request of distributed file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113254415A CN113254415A (en) | 2021-08-13 |
| CN113254415B true CN113254415B (en) | 2022-11-04 |
Family
ID=77182782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110546268.5A Active CN113254415B (en) | 2021-05-19 | 2021-05-19 | Method and device for processing read request of distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113254415B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102982105A (en) * | 2012-11-07 | 2013-03-20 | 江苏乐买到网络科技有限公司 | Method for reading and writing files in distributed file system (DFS) |
| CN105872040A (en) * | 2016-03-30 | 2016-08-17 | 华中科技大学 | Method for optimizing write performance of distributed block storage system by utilizing cache of gateway nodes |
| CN106161503A (en) * | 2015-03-27 | 2016-11-23 | 中兴通讯股份有限公司 | File reading in a kind of distributed memory system and service end |
| CN108959519A (en) * | 2018-06-28 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method, apparatus and computer readable storage medium reading data |
| CN109691065A (en) * | 2018-08-23 | 2019-04-26 | 袁振南 | Distributed memory system and its data read-write method, storage terminal and storage medium |
| CN112130758A (en) * | 2020-09-04 | 2020-12-25 | 苏州浪潮智能科技有限公司 | Data reading request processing method and system, electronic equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653684B (en) * | 2015-12-29 | 2020-03-03 | 曙光云计算集团有限公司 | Pre-reading method and device of distributed file system |
| CN111881096B (en) * | 2020-07-24 | 2022-06-17 | 北京浪潮数据技术有限公司 | File reading method, device, equipment and storage medium |
-
2021
- 2021-05-19 CN CN202110546268.5A patent/CN113254415B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102982105A (en) * | 2012-11-07 | 2013-03-20 | 江苏乐买到网络科技有限公司 | Method for reading and writing files in distributed file system (DFS) |
| CN106161503A (en) * | 2015-03-27 | 2016-11-23 | 中兴通讯股份有限公司 | File reading in a kind of distributed memory system and service end |
| CN105872040A (en) * | 2016-03-30 | 2016-08-17 | 华中科技大学 | Method for optimizing write performance of distributed block storage system by utilizing cache of gateway nodes |
| CN108959519A (en) * | 2018-06-28 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method, apparatus and computer readable storage medium reading data |
| CN109691065A (en) * | 2018-08-23 | 2019-04-26 | 袁振南 | Distributed memory system and its data read-write method, storage terminal and storage medium |
| WO2020037625A1 (en) * | 2018-08-23 | 2020-02-27 | 袁振南 | Distributed storage system and data read-write method therefor, and storage terminal and storage medium |
| CN112130758A (en) * | 2020-09-04 | 2020-12-25 | 苏州浪潮智能科技有限公司 | Data reading request processing method and system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113254415A (en) | 2021-08-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10776174B2 (en) | Managing hosted resources across different virtualization platforms | |
| US20120102072A1 (en) | Distributed management monitoring system, monitoring method and creating method thereof | |
| JP2005535019A (en) | Storage management bridge | |
| CN103078898B (en) | File system, interface service device and data storage service supplying method | |
| CN106648838B (en) | Resource pool management configuration method and device | |
| CN113395340A (en) | Information updating method, device, equipment, system and readable storage medium | |
| WO2016101662A1 (en) | Data processing method and relevant server | |
| CN113535068A (en) | Data reading method and system | |
| US20240126847A1 (en) | Authentication method and apparatus, and storage system | |
| CN113885798A (en) | Data operation method, device, equipment and medium | |
| WO2021226822A1 (en) | Log write method and apparatus, electronic device, and storage medium | |
| CN111600755B (en) | Internet access behavior management system and method | |
| CN113254415B (en) | Method and device for processing read request of distributed file system | |
| CN115987974A (en) | Lightweight FTP service method and system | |
| US20190121899A1 (en) | Apparatus and method for managing integrated storage | |
| CN110347656A (en) | The management method and device requested in document storage system | |
| CN107526530B (en) | Data processing method and device | |
| CN115238006A (en) | Retrieval data synchronization method, device, equipment and computer storage medium | |
| CN116594551A (en) | Data storage method and device | |
| US11526446B1 (en) | Modifying caching amongst services from a history of requests and responses | |
| US11386072B1 (en) | Automatic consistency for database write forwarding | |
| US11886439B1 (en) | Asynchronous change data capture for direct external transmission | |
| WO2024066904A1 (en) | Container creation method, system, and node | |
| US11656957B1 (en) | Managing nodes of a DBMS | |
| US12007954B1 (en) | Selective forwarding for multi-statement database transactions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |