CN110781137A - Directory reading method and device for distributed system, server and storage medium - Google Patents
Directory reading method and device for distributed system, server and storage medium Download PDFInfo
- Publication number
- CN110781137A CN110781137A CN201911030300.3A CN201911030300A CN110781137A CN 110781137 A CN110781137 A CN 110781137A CN 201911030300 A CN201911030300 A CN 201911030300A CN 110781137 A CN110781137 A CN 110781137A
- Authority
- CN
- China
- Prior art keywords
- directory
- reading
- directory reading
- subtasks
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention discloses a directory reading method of a distributed system, which is applied to a local server and comprises the following steps: receiving a directory reading request sent by a client; generating a directory read task based on the directory read request; dividing the directory reading task into N subtasks based on a preset rule; asynchronously executing the N subtasks, and generating a group of directory reading results after each subtask is executed; and based on the execution progress of the N subtasks, sequentially sending the directory reading result to the client according to the time sequence of result generation. The invention also provides a directory reading device, a server and a storage medium of the distributed system. The reading task of the client is divided into N subtasks to be asynchronously executed, so that the reading time is shortened, and meanwhile, the reading tasks are sequentially returned to the client according to the time sequence of the completion of the subtasks, so that the problem that the whole reading task fails due to the failure of reading part of the directory is avoided.
Description
Technical Field
The embodiment of the invention relates to a technology for reading a super-large directory in a processing file interface by using a metadata service of a cluster file system, in particular to a directory reading method, a device, a server and a storage medium of a distributed system.
Background
The cluster file system can be divided into a shared storage type cluster file system and a distributed cluster file system according to the access mode of the storage space, wherein the shared storage type cluster file system is a shared file system which is called a shared file system, wherein a plurality of computers identify the same storage space and coordinate with one another to jointly manage files on the shared storage type cluster file system; the latter is that each computer provides its own storage space and coordinates and manages the files in all the computer nodes. The large-scale cluster file system belongs to the distributed cluster file system without exception. The distributed cluster file system is more extensible.
In the prior art, when a super-large directory executes column directory operation, the situation that the time for listing the directory or the column directory is very long cannot be effectively realized, the super-large directory loading time is very long, a large amount of time is consumed, and when the super-large directory loading fails, the directory cannot be read out, and the reading fails.
The reading task of the client is divided into N subtasks to be asynchronously executed, so that the reading time is shortened, and meanwhile, the reading tasks are sequentially returned to the client according to the time sequence of the completion of the subtasks, so that the problem that the whole reading task fails due to the failure of reading part of the directory is avoided.
Disclosure of Invention
The invention provides a directory reading method of a distributed system, which is applied to a local server to shorten the reading time and avoid the effect that the whole reading task fails due to the failure of reading part of directories.
In a first aspect, an embodiment of the present invention provides a directory reading method for a distributed system, including:
receiving a directory reading request sent by a client;
generating a directory read task based on the directory read request;
dividing the directory reading task into N subtasks based on a preset rule;
asynchronously executing the N subtasks, and generating a group of directory reading results after each subtask is executed;
and based on the execution progress of the N subtasks, sequentially sending the directory reading result to the client according to the time sequence of result generation.
In a second aspect, an embodiment of the present invention further provides a directory reading apparatus for a distributed system, including the following modules:
the receiving module is used for receiving a directory reading request sent by a client;
the task generation module is used for generating a directory reading task based on the directory reading request;
the grouping module is used for dividing the directory reading task into N subtasks based on a preset rule;
the execution module is used for asynchronously executing the N subtasks, and each subtask generates a group of directory reading results after being executed;
and the first sending module is used for sending the directory reading result to the client side in sequence according to the time sequence of result generation based on the execution progress of the N subtasks.
In a third aspect, an embodiment of the present invention further provides a server, including a memory, a processor, and a program stored in the memory and executable on the processor, where the processor executes the program to implement the directory reading method for the distributed system as described in any one of the above.
In a fourth aspect, an embodiment of the present invention further provides a terminal-readable storage medium, on which a program is stored, where the program, when executed by a processor, can implement the directory reading method for the distributed system as described in any one of the above.
The reading task of the client is divided into N subtasks to be asynchronously executed, so that the reading time is shortened, and meanwhile, the reading tasks are sequentially returned to the client according to the time sequence of the completion of the subtasks, so that the problem that the whole reading task fails due to the failure of reading part of the directory is avoided.
Drawings
Fig. 1 is a flowchart of a directory reading method of a distributed system according to a first embodiment of the present invention.
Fig. 2 is a flowchart of a directory reading method of a distributed system according to a second embodiment of the present invention.
Fig. 3 is a flowchart of a directory reading method of a distributed system in the third embodiment of the present invention.
Fig. 4 is a flow chart of steps in a third alternative embodiment of the present invention.
Fig. 5 is a structural diagram of a directory reading apparatus of a distributed system according to a fourth embodiment of the present invention.
Fig. 6 is a block diagram of a directory reading apparatus of a distributed system according to a fourth alternative embodiment of the present invention.
Fig. 7 is a schematic structural diagram of a server in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
Example one
As shown in fig. 1, a first embodiment of the present invention provides a directory reading method for a distributed system, where the distributed cluster file system Ceph is taken as an example for description in this embodiment and the following embodiments, and the method specifically includes the following steps:
s101, receiving a catalog reading request sent by a client.
In a distributed file system, a read operation for a directory generally refers to a metadata operation, such as a column directory obtaining information of all files in the directory. The process of reading the directory by the file interface of the open source distributed cluster file system Ceph is that the client sends a request for reading the file information under the directory, the metadata management service generates a directory reading task, the directory reading task is executed to inquire to obtain the total number N of the file information under the directory, all the metadata information of the N files is obtained at one time, and the obtained metadata information is returned to the client together.
And S102, generating a directory reading task based on the directory reading request.
In a big data environment, the size of the directory is very large, and the access performance of the directory is the key of the performance of the whole distributed file system. In this step, after receiving the read request message, the server of the distributed storage system concurrently reads directory entries of a plurality of directory shard objects in the target directory from the metadata server. The directory reading task refers to an operation of reading metadata of each piece of description data in a directory and delivering the read directory entry to the client.
S103, dividing the directory reading task into N subtasks based on a preset rule.
In the step, the server obtains the number of files of the directory to be read by scanning the record of the data file directory stored in the file system, and divides the directory reading task into N subtasks based on the number of the files. After receiving a target directory request sent by a client, a client of the distributed file system requests a metadata server for a directory size of a target directory, where the directory size may be represented by the number of directory entries contained in the directory, where a directory entry refers to data describing an attribute of a specific file, that is, data describing data, including attributes of the file, such as time of file creation, modification, and access, file owner, file permissions, mapping information, and the like, and may also be referred to as metadata.
Preferably, the directory reading request includes the number of directory files to be read, and the preset number of files in each group is determined according to the number of the directory files to be read. In this embodiment, preferably, the number M of files per group is the same, and the value of the number M of files per group is such that the number N of grouped subtasks is a positive integer. Optionally, the value setting of the number M of files and the number N of subtasks in each group is related to the remaining IOPS and the average IOPS of the file system.
S104, asynchronously executing the N subtasks, and generating a group of directory reading results after each subtask is executed.
This step is performed using asynchrony. Asynchronous execution means that the execution order of each statement is not necessarily the same as the execution order of the task in the asynchronous execution mode. In the operation of this embodiment, the server executes the N subtasks for reading the directory file, each subtask reads the file directory in the metadata, and directly returns the task result to the client after the execution is completed, without waiting for the server to merge the query results and then return the query results to the client together.
And S105, sequentially sending the directory reading result to the client according to the time sequence of result generation based on the execution progress of the N subtasks.
Due to the fact that the size and the number of the read directory files are different, the execution end time of the N sub-tasks executed asynchronously is different. And returning the directory reading result generated firstly to the client according to the time sequence, and returning the directory reading result generated subsequently to the client.
In this step, the file system processes N subtasks, and since the directory file of the distributed file system sometimes has a large capacity, if the processed N subtasks are returned to the client after being merged, time consumption may be too long. Meanwhile, if a subtask with a reading failure occurs in the N subtasks, the processing results cannot be merged and returned to the client, and if one subtask fails, the reading failure is caused.
In the embodiment, the reading task of the client is divided into N subtasks to be asynchronously executed, so that the reading time is shortened, and meanwhile, the reading tasks are sequentially returned to the client according to the time sequence of the completion of the subtasks, so that the failure of the whole reading task caused by the failure of reading part of the directory is avoided.
Example two
As shown in fig. 2, in this embodiment, on the basis of the above embodiment, the step of dividing the directory read task into N subtasks is subdivided as follows:
s201, receiving a directory reading request sent by a client.
And S202, generating a directory reading task based on the directory reading request.
S2031, scanning the directory file to be read according to the directory reading task, and acquiring the number of the directory files of the directory file to be read.
And S2032, dividing the directory files into N groups based on the preset number of files in each group, wherein the number N of the groups is the number of the directory files/the preset number of the files in each group.
In the step, the number of the preset files in each group can be set by parameter customization, and dynamic adjustment can be performed according to the use condition of the actual environment so as to achieve the optimal number. The grouping number N is obtained by rounding up the number of directory files/the number of preset files in each group.
S2033, dividing the directory reading task into N subtasks based on the grouping number N.
S204, the N subtasks are executed asynchronously, and a group of directory reading results are generated after each subtask is executed.
In this step, each subtask execution reads metadata information of the task assigned by itself from the server.
And S205, sequentially sending the directory reading result to the client according to the time sequence of result generation based on the execution progress of the N subtasks.
In this step, the time for finishing the execution of the N sub-tasks executed asynchronously is different because the size and the number of the read directory files are different. And returning the directory reading result generated firstly to the client according to the time sequence, and returning the directory reading result generated subsequently to the client. Optionally, when the reading of one or more subtasks fails and a directory reading result cannot be generated, returning corresponding subtask reading failure information to the client.
The embodiment determines the grouping number based on the preset number of each group of files, so that the grouping number is an integer, the processing of the subtasks is more convenient, and the problem that the whole reading task fails due to the fact that part of directories are failed to read is avoided.
EXAMPLE III
As shown in fig. 3, in this embodiment, a caching step is added on the basis of the above embodiment, so that the processed directory read result is cached in the client of the distributed file system, and the specific steps are as follows:
s301, receiving a directory reading request sent by a client.
S302, judging whether the cached directory reading result exists.
In this step, in the history operation, the local cache may store the directory entry that needs to be sent to the service end in the history operation. If the reading task of the client is acquired each time, the reading task is executed, and then the directory file is returned to the client, the time for generating the reading task, the server splitting task and the server executing the directory entry to read the directory file is needed, and the cache task is directly read, so that the scheme does not need waiting time, and the reading speed of the directory on the metadata server is accelerated.
When a directory reading request initiated by the client is acquired, the server judges whether the cached directory exists in the local cache, if so, the server indicates that the part of directory information is read in the history operation, and can directly acquire a directory reading result in the local cache, and step S303 is executed; if not, generating a directory reading task and executing the step S304.
And S303, if so, sending the cached directory reading result to a client.
S304, if not, generating a directory reading task based on the directory reading request.
In this step, if there is no cache, it indicates that the directory file in the part has not been read in the history operation, and needs to be read again, and after the time of generating the read task, the server splitting task, and the server executing the directory entry to read the directory file, the following step S305 is executed.
S305, generating a directory reading task based on the directory reading request.
S306, dividing the directory reading task into N subtasks based on a preset rule.
S307, the N subtasks are executed asynchronously, and a group of directory reading results are generated after each subtask is executed.
And S308, sequentially caching the generated directory reading result.
In this step, the N subtasks are sequentially executed and completed, and a directory read result is generated. And storing the directory reading result to a local cache of the distributed file system, and opening up a cache space locally by the server so as to store the directory entry generated after reading and returned to the client.
In this embodiment, the caching the generated directory reading result may be: caching in a local server, caching in other clients mounted on a file system, caching in a metadata server, and the like, but not limited to the above caching methods. Exemplarily, the cache is cached in other clients mounted in the file system, and specifically includes: and when the directory reading result is returned, the directory reading result is sent to the client initiating the reading request, and the directory reading result is distributed to other mounted clients in the distributed file system.
309. And sending the directory reading result to a metadata server and/or other local servers.
In a distributed file system, a file system that allows files to be shared across multiple hosts over a network may allow multiple users on multiple machines to share files and storage space. As described above, the directory read results may be cached to a metadata server and/or other local servers.
In this step, it should be noted that the storage spaces of the metadata server and the local server are both limited, and if the capacity of the directory read result is too large, the storage spaces of the metadata server and/or the local server may be occupied, which may cause an imbalance in resource allocation of the distributed file system. Therefore, in this step, preferably, a quota threshold is set for the cached directory read result, and if the quota threshold is exceeded, it indicates that there is no margin in the space of the metadata server and/or the local server, the directory read result is not cached. Optionally, the global quota space in the metadata server can dynamically adjust the quota space of the local server, and therefore, if the local server cannot be cached due to insufficient space, a quota adjustment request can be sent to the metadata server to dynamically allocate an adjustment space to the local server, and a directory reading result is cached, so that time is saved for the next directory reading operation.
And S310, sequentially sending the directory reading result to the client according to the time sequence of result generation based on the execution progress of the N subtasks.
As shown in fig. 4, in an alternative embodiment, after step S308, the method further includes:
s311, judging whether a file read-write request sent by a client is received, wherein the file read-write request comprises a file directory to be written.
The step S308 caches the directory reading result in the history operation, and when the directory reading result is cached and sent to other clients and/or servers, the directory reading result generates a current timestamp to record the generation time information of the directory reading result. When the client or other clients write or delete the directory after the timestamp, the content of the directory file changes, and all cached directory read results will automatically fail.
In this step, it is determined whether a file read-write request of the client is received after the directory reading result is cached, where the file read-write request includes operations of writing, deleting, modifying, and the like on file data. The file read-write request also comprises a file directory to be written, and the server judges whether cache data of a directory read result exist in the file directory to be written in the historical operation so as to determine whether the cache data needs to be deleted.
S312, if so, judging whether the file directory corresponds to the cached directory reading result.
If a write request from the client is received and the file content is modified, the directory read result changes, the cached data in the history operation is invalid, and step S313 is executed. If the write request is not received, no operation is performed.
S313, if so, deleting the cached directory reading result.
According to the embodiment, the executed subtasks are cached in the local server and the metadata server of the file system, so that the client can firstly obtain data through the cache when needing to read the directory next time, the reading efficiency is improved, and the time waste is avoided. Meanwhile, after the read-write operation of the client on the file data is obtained, the cache data is deleted, the condition that the cache data is inconsistent with the current file condition can be avoided, and the reading error is avoided.
Example four
As shown in fig. 5, the present embodiment provides a directory reading apparatus 4 of a distributed system, which includes the following modules:
the receiving module 401 is configured to receive a directory read request sent by a client.
A task generating module 402, configured to generate a directory read task based on the directory read request.
A grouping module 403, configured to divide the directory reading task into N subtasks based on a preset rule.
And the execution module 404 is configured to execute the N subtasks asynchronously, where each subtask generates a group of directory read results after being executed.
And a first sending module 405, configured to send the directory reading result to the client according to the time sequence of result generation in sequence based on the execution progress of the N subtasks.
As shown in fig. 6, in an alternative embodiment, the grouping module 403 further includes:
and the scanning unit 4031 is configured to scan the directory file to be read according to the directory reading task, and obtain the number of directory files of the directory file to be read.
The group number determining unit 4032 is configured to divide the directory files into N groups based on a preset number of files in each group, where the group number N is the number of directory files/the preset number of files in each group. The preset number of files in each group is determined according to the number of the directory files to be read
A subtask splitting unit 4033, configured to split the directory reading task into N subtasks based on the grouping number N.
In an alternative embodiment, further comprising:
the caching module 406 is configured to sequentially cache the generated directory read results before the directory read results are sequentially sent to the client according to the time sequence of result generation based on the execution progress of the N subtasks.
In an alternative embodiment, further comprising:
the cache reading module 407 is configured to determine whether a cached directory reading result exists after receiving a directory reading request sent by a client; if yes, sending the cached directory reading result to a client; and if not, generating a directory reading task based on the directory reading request.
A second sending module 408, configured to send the directory read result to a metadata server and/or other local servers.
The determining module 409 determines whether a file read-write request sent by a client is received, where the file read-write request includes a file directory to be written; if so, judging whether the file directory corresponds to the cached directory reading result; and if so, deleting the cached directory reading result.
The product can execute the method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
The present embodiment provides a schematic structural diagram of a server, as shown in fig. 7, the server includes a processor 501, a memory 502, an input device 503, and an output device 504; the number of the processors 501 in the server may be one or more, and one processor 501 is taken as an example in the figure; the processor 501, the memory 502, the input device 503 and the output device 504 in the device/terminal/server may be connected by a bus or other means, and fig. 7 illustrates the connection by a bus as an example.
The memory 502 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules (e.g., the receiving module 401, the task generating module 402, etc.) corresponding to the theme update method in the embodiment of the present invention. The processor 501 executes various functional applications of the device/terminal/server and data processing by running software programs, instructions, and modules stored in the memory 502, that is, implements the directory reading method of the distributed system described above.
The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 502 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 502 may further include memory located remotely from processor 501, which may be connected to devices/terminals/servers through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 503 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the device/terminal/server. The output device 504 may include a display device such as a display screen.
Fifth, the embodiments of the present invention provide a server, which can execute the directory reading method of the distributed system provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
EXAMPLE six
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a directory reading method for a distributed system according to any embodiment of the present invention, where the method may include:
receiving a directory reading request sent by a client;
generating a directory read task based on the directory read request;
dividing the directory reading task into N subtasks based on a preset rule;
asynchronously executing the N subtasks, and generating a group of directory reading results after each subtask is executed;
and based on the execution progress of the N subtasks, sequentially sending the directory reading result to the client according to the time sequence of result generation.
The computer-readable storage media of embodiments of the invention may take any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a storage medium may be transmitted over any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A directory reading method of a distributed system is applied to a local server, and is characterized by comprising the following steps:
receiving a directory reading request sent by a client;
generating a directory read task based on the directory read request;
dividing the directory reading task into N subtasks based on a preset rule;
asynchronously executing the N subtasks, and generating a group of directory reading results after each subtask is executed;
and based on the execution progress of the N subtasks, sequentially sending the directory reading result to the client according to the time sequence of result generation.
2. The directory reading method of the distributed system according to claim 1, wherein the dividing the directory reading task into N subtasks based on a preset rule includes:
scanning the directory file to be read according to the directory reading task, and acquiring the number of directory files of the directory file to be read;
dividing the directory files into N groups based on the preset number of files in each group, wherein the group number N is the number of the directory files/the preset number of the files in each group;
and dividing the directory reading task into N subtasks based on the grouping number N.
3. The directory reading method of the distributed system according to claim 2, wherein the directory reading request includes the number of directory files to be read, and the preset number of files in each group is determined according to the number of directory files to be read.
4. The directory reading method of the distributed system according to claim 1, wherein before the executing schedule based on the N subtasks sequentially sends the directory reading result to the client according to the time sequence of the result generation, the method further includes: and sequentially caching the generated directory reading results.
5. The directory reading method of the distributed system according to claim 4, wherein after receiving the directory reading request sent by the client, the method further includes:
judging whether the cached directory reading result exists or not;
if yes, sending the cached directory reading result to a client;
and if not, generating a directory reading task based on the directory reading request.
6. The directory reading method of the distributed system according to claim 4, wherein after the sequentially caching the generated directory reading results, the method further includes:
and sending the directory reading result to a metadata server and/or other local servers.
7. The directory reading method of the distributed system according to claim 4, wherein after the sequentially caching the generated directory reading results, the method further includes:
judging whether a file read-write request sent by a client is received, wherein the file read-write request comprises a file directory to be written;
if so, judging whether the file directory corresponds to the cached directory reading result;
and if so, deleting the cached directory reading result.
8. The directory reading device of the distributed system is characterized by comprising the following modules:
the receiving module is used for receiving a directory reading request sent by a client;
the task generation module is used for generating a directory reading task based on the directory reading request;
the grouping module is used for dividing the directory reading task into N subtasks based on a preset rule;
the execution module is used for asynchronously executing the N subtasks, and each subtask generates a group of directory reading results after being executed;
and the first sending module is used for sending the directory reading result to the client side in sequence according to the time sequence of result generation based on the execution progress of the N subtasks.
9. A server comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor implements the directory reading method of the distributed system as claimed in any one of claims 1 to 7 when executing the program.
10. A terminal-readable storage medium on which a program is stored, the program being capable of implementing a directory reading method of a distributed system according to any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911030300.3A CN110781137A (en) | 2019-10-28 | 2019-10-28 | Directory reading method and device for distributed system, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911030300.3A CN110781137A (en) | 2019-10-28 | 2019-10-28 | Directory reading method and device for distributed system, server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110781137A true CN110781137A (en) | 2020-02-11 |
Family
ID=69386858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911030300.3A Pending CN110781137A (en) | 2019-10-28 | 2019-10-28 | Directory reading method and device for distributed system, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781137A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625357A (en) * | 2020-05-22 | 2020-09-04 | 柏科数据技术（深圳）股份有限公司 | Directory reading method and device based on Feiteng platform, server and storage medium |
| CN116701299A (en) * | 2022-12-09 | 2023-09-05 | 荣耀终端有限公司 | Catalog management method and related device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120226872A1 (en) * | 2011-03-03 | 2012-09-06 | Zoran Rajic | Prefetching content of a directory by executing a directory access command |
| CN104331255A (en) * | 2014-11-17 | 2015-02-04 | 中国科学院声学研究所 | Embedded file system-based reading method for streaming data |
| CN106777180A (en) * | 2016-12-22 | 2017-05-31 | 北京京东金融科技控股有限公司 | The method of high-performance distributed data conversion, apparatus and system |
| CN107491545A (en) * | 2017-08-25 | 2017-12-19 | 郑州云海信息技术有限公司 | The catalogue read method and client of a kind of distributed memory system |
| CN108319634A (en) * | 2017-12-15 | 2018-07-24 | 创新科存储技术（深圳）有限公司 | The directory access method and apparatus of distributed file system |
| CN110019024A (en) * | 2019-04-11 | 2019-07-16 | 苏州浪潮智能科技有限公司 | A kind of directory method, system and electronic equipment and storage medium |
-
2019
- 2019-10-28 CN CN201911030300.3A patent/CN110781137A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120226872A1 (en) * | 2011-03-03 | 2012-09-06 | Zoran Rajic | Prefetching content of a directory by executing a directory access command |
| CN104331255A (en) * | 2014-11-17 | 2015-02-04 | 中国科学院声学研究所 | Embedded file system-based reading method for streaming data |
| CN106777180A (en) * | 2016-12-22 | 2017-05-31 | 北京京东金融科技控股有限公司 | The method of high-performance distributed data conversion, apparatus and system |
| CN107491545A (en) * | 2017-08-25 | 2017-12-19 | 郑州云海信息技术有限公司 | The catalogue read method and client of a kind of distributed memory system |
| CN108319634A (en) * | 2017-12-15 | 2018-07-24 | 创新科存储技术（深圳）有限公司 | The directory access method and apparatus of distributed file system |
| CN110019024A (en) * | 2019-04-11 | 2019-07-16 | 苏州浪潮智能科技有限公司 | A kind of directory method, system and electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 黄文海 著: "《Java多线程编程实战指南 核心篇》", 30 April 2017, 电子工业出版社 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625357A (en) * | 2020-05-22 | 2020-09-04 | 柏科数据技术（深圳）股份有限公司 | Directory reading method and device based on Feiteng platform, server and storage medium |
| CN116701299A (en) * | 2022-12-09 | 2023-09-05 | 荣耀终端有限公司 | Catalog management method and related device |
| CN116701299B (en) * | 2022-12-09 | 2024-03-26 | 荣耀终端有限公司 | Catalog management method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107391653B (en) | Distributed NewSQL database system and picture data storage method | |
| US10896102B2 (en) | Implementing secure communication in a distributed computing system | |
| JP6778795B2 (en) | Methods, devices and systems for storing data | |
| CN111078147B (en) | Processing method, device and equipment for cache data and storage medium | |
| US10235047B2 (en) | Memory management method, apparatus, and system | |
| CN111506269B (en) | Disk storage space allocation method, device, equipment and storage medium | |
| US9836516B2 (en) | Parallel scanners for log based replication | |
| KR101634403B1 (en) | Approaches to reducing lock communications in a shared disk database system | |
| CN103729247A (en) | Data acquisition request processing method and system and server | |
| CN104881466A (en) | Method and device for processing data fragments and deleting garbage files | |
| CN114138776A (en) | Method, system, apparatus and medium for graph structure and graph attribute separation design | |
| CN110781137A (en) | Directory reading method and device for distributed system, server and storage medium | |
| CN110781159B (en) | Ceph directory file information reading method and device, server and storage medium | |
| CN113051102B (en) | File backup method, device, system, storage medium and computer equipment | |
| CN113885780A (en) | Data synchronization method, device, electronic equipment, system and storage medium | |
| CN110706148A (en) | Face image processing method, device, equipment and storage medium | |
| CN104517067A (en) | Method, device and system for data access | |
| WO2017113277A1 (en) | Data processing method, device, and system | |
| US11055262B1 (en) | Extensible streams on data sources | |
| CN113051221A (en) | Data storage method, device, medium, equipment and distributed file system | |
| CN113760638A (en) | Log service method and device based on kubernets cluster | |
| CN117056123A (en) | Data recovery method, device, medium and electronic equipment | |
| US11176097B2 (en) | Accelerated deduplication block replication | |
| CN114116655A (en) | Parallel file storage method, device, equipment and storage medium based on virtual machine | |
| CN112559568A (en) | Virtual article determination method and device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200211 |