CN110781159B - Ceph directory file information reading method and device, server and storage medium - Google Patents
Ceph directory file information reading method and device, server and storage medium Download PDFInfo
- Publication number
- CN110781159B CN110781159B CN201911030299.4A CN201911030299A CN110781159B CN 110781159 B CN110781159 B CN 110781159B CN 201911030299 A CN201911030299 A CN 201911030299A CN 110781159 B CN110781159 B CN 110781159B
- Authority
- CN
- China
- Prior art keywords
- file information
- directory file
- processing unit
- reading
- directory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Abstract
The embodiment of the invention discloses a method, a device, a server and a storage medium for reading Ceph directory file information, wherein the method comprises the following steps: a first processing unit acquires a reading request of directory file information; the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information; the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule; the second processing unit executes a plurality of the subtasks to read the directory file information. The reading of the directory file information is realized, the reading speed of the directory file information is improved, and the reading time of the directory file information is reduced.
Description
Technical Field
The embodiment of the invention relates to the field of cluster file systems, in particular to a method and a device for reading information of a Ceph directory file, a server and a storage medium.
Background
The distributed file cluster system is called a Ceph file system for short, and refers to a file system which runs among a plurality of computers, communicates with each other in a certain mode so as to integrate and virtualize all storage space resources in a cluster and provide file access service to the outside.
Users often need to read a file directory through the Ceph file system, and the reading operation for the directory generally refers to metadata operation, such as obtaining information of all files in the directory. Generally, a client query end sends a request for reading file information in a directory, a metadata management service queries to obtain the total number of files in the directory, a CPU (central processing unit) sequentially obtains related file information according to the total number of files until the total number of files is obtained, and then the result is returned to the client query end.
The directories belong to main metadata servers in the metadata server cluster, and the rest metadata servers take over the function of the main metadata servers only when the main metadata servers are in failure, but only one metadata server establishes the directories at the same time, so that the load of the metadata servers is unbalanced, the performance of the metadata server cluster cannot be exerted, and when the directories are too large, the time for reading directory file information is prolonged.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a server, and a storage medium for reading Ceph directory file information, so as to improve the reading speed of the directory file information and reduce the reading time of the directory file information.
In a first aspect, an embodiment of the present invention provides a method for reading Ceph directory file information, including:
a first processing unit acquires a reading request of directory file information;
the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information;
the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule;
the second processing unit executes a plurality of the subtasks to read the directory file information.
Further, the generating, by the first processing unit, a reading task of the directory file information according to the reading request of the directory file information includes:
the first processing unit acquires the total number of file information corresponding to the directory file information according to the reading request of the directory file information;
and the first processing unit generates a reading task of the directory file information according to the total number of the file information corresponding to the directory file information.
Further, the splitting, by the second processing unit, the task of reading the directory file information into a plurality of sub-tasks according to a preset rule includes:
and the second processing unit averagely divides the reading task of the directory file information into a plurality of subtasks according to the total number of the file information and the number of preset single subtask file information.
Further, the second processing unit executing the plurality of subtasks to read the directory file information includes:
the second processing unit adopts a plurality of threads to simultaneously execute a plurality of subtasks to obtain a plurality of subdirectories of the directory file information;
and the second processing unit recombines a plurality of subdirectories of the directory file information to obtain the directory file information.
Further, after the second processing unit executes a plurality of the sub tasks to read the directory file information, the method includes:
the second processing unit sends the directory file information to the first processing unit;
and the first processing unit returns the directory file information to the inquiry end.
Further, the first processing unit is a metadata management module.
Further, the second processing unit is a GPU operation module.
In a second aspect, an embodiment of the present invention provides a Ceph directory file information reading apparatus, including:
the first processing unit is used for acquiring a reading request of the directory file information, generating a reading task of the directory file information according to the reading request of the directory file information, and sending the reading task of the directory file information to the second processing unit;
and the second processing unit is used for splitting the reading task of the directory file information into a plurality of subtasks according to a preset rule and executing the plurality of subtasks to read the directory file information.
In a third aspect, an embodiment of the present invention provides a server, including:
one or more treatments;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processes, the one or more processes implement the Ceph directory file information reading method provided by any embodiment of the present invention.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is processed and executed, the computer program implements the Ceph directory file information reading method provided in any embodiment of the present invention.
The embodiment of the invention obtains a reading request of the directory file information through a first processing unit; the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information; the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule; the second processing unit executes a plurality of the subtasks to read the directory file information. The reading of the directory file information is realized, the reading speed of the directory file information is improved, and the reading time of the directory file information is reduced.
Drawings
Fig. 1 is a schematic flowchart of a method for reading Ceph directory file information according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a method for reading Ceph directory file information according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a Ceph directory file information reading apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a server according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
Furthermore, the terms "first," "second," and the like may be used herein to describe various orientations, actions, steps, elements, or the like, but the orientations, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish one direction, action, step or element from another direction, action, step or element. For example, a first processing unit may be referred to as a second processing unit, and similarly, a second processing unit may be referred to as a first processing unit, without departing from the scope of the present application. The first processing unit and the second processing unit are both processing units, but they are not the same processing unit. The terms "first", "second", etc. are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Example one
Fig. 1 is a schematic flowchart of a method for reading Ceph directory file information according to an embodiment of the present invention, which is applicable to reading directory file information using a file interface of Ceph. The method can be realized by the Ceph directory file information reading device provided by any embodiment of the invention, and the device can be realized in a software and/or hardware mode and can be integrated in a server. As shown in fig. 1, a method for reading Ceph directory file information according to an embodiment of the present invention includes:
s110, the first processing unit obtains a reading request of the directory file information.
Specifically, in order to realize "access by name", a corresponding relationship between file names and physical addresses in a storage space needs to be established, and a data structure representing the corresponding relationship is called a file directory. The file information of the directory refers to information of files contained in the directory, for example, a plurality of Word files are contained in one directory, and the file information of the directory includes, but is not limited to, a storage location of each Word document, a document size, a document attribute, a document type, and the like. When a user needs to read the directory file information, the query end initiates a reading request of the directory file information, and the first processing unit can execute relevant operations by acquiring the reading request of the directory file information, so that the requirement of reading the relevant information is met.
Metadata (Metadata), also called intermediary data and relay data, is data (data about data) describing data, mainly information describing data property (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like. In the Ceph file system, the file information of the directory is usually recorded using metadata information and stored in a plurality of metadata servers, and the metadata management module manages the metadata servers, so the first processing unit may be the metadata management module of the Ceph file system.
S120, the first processing unit generates a reading task of the directory file information according to the reading request of the directory file information.
Specifically, the first processing unit generates a reading task of the corresponding directory file information after acquiring the reading request of the directory file information, and performs the reading operation of the directory file information according to the reading task of the directory file information.
S130, the first processing unit sends the reading task of the directory file information to a second processing unit, so that the second processing unit divides the reading task of the directory file information into a plurality of sub-tasks according to a preset rule.
Specifically, after generating the read task of the directory file information, the first processing unit does not directly perform the read operation of the directory file information, but sends the read task to the second processing unit, where the computational power of the second processing unit is higher than that of the first processing unit, and the second processing unit may be a GPU operation module.
The second processing unit splits the reading task of the directory file information into a plurality of subtasks according to a preset rule, where the preset rule is a splitting rule of the reading task of the directory file information, and the splitting rule may be performed according to the number of file information included in the directory file information, or may be performed according to the directory file information, which is not limited in this embodiment.
S140, the second processing unit executes a plurality of subtasks to read the directory file information.
Specifically, the second processing unit may execute a plurality of subtasks simultaneously, each subtask may read a part of the file information of the directory file information, and when all subtasks are executed, all file information of the directory file information is read.
In the first embodiment of the invention, a first processing unit obtains a reading request of directory file information; the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information; the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule; the second processing unit executes a plurality of the subtasks to read the directory file information. The reading of the directory file information is realized, the reading speed of the directory file information is improved, and the reading time of the directory file information is reduced.
Example two
Fig. 2 is a schematic flow chart of a Ceph directory file information reading method according to a second embodiment of the present invention, which is a refinement of the above embodiment. As shown in fig. 2, a method for reading Ceph directory file information according to a second embodiment of the present invention includes:
s210, the first processing unit obtains a reading request of the directory file information.
Specifically, in order to realize "access by name", a corresponding relationship between file names and physical addresses in a storage space needs to be established, and a data structure representing the corresponding relationship is called a file directory. The file information of the directory refers to information of files contained in the directory, for example, a plurality of Word files are contained in one directory, and the file information of the directory includes, but is not limited to, a storage location of each Word document, a document size, a document attribute, a document type, and the like. When a user needs to read the directory file information, the query end initiates a reading request of the directory file information, and the first processing unit can execute relevant operations by acquiring the reading request of the directory file information, so that the requirement of reading the relevant information is met.
Metadata (Metadata), also called intermediary data and relay data, is data (data about data) describing data, mainly information describing data property (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like. In the Ceph file system, the file information of the directory is usually recorded using metadata information and stored in a plurality of metadata servers, and the metadata management module manages the metadata servers, so the first processing unit may be the metadata management module of the Ceph file system.
S220, the first processing unit obtains the total number of the file information corresponding to the directory file information according to the reading request of the directory file information.
Specifically, the total number of file information indicates the number of all file information included in the directory, for example, the total number of file information obtained by the metadata management module according to the read request of the directory file information is 1000.
And S230, the first processing unit generates a reading task of the directory file information according to the total number of the file information corresponding to the directory file information.
Specifically, the reading task of the directory file information is generated according to the total number of the file information, so that the number of the file information needing to be read can be known, for example, the reading task of the directory file information generated by the metadata management module is to read 1000 pieces of file information of the directory.
S240, the first processing unit sends the reading task of the directory file information to a second processing unit, and the second processing unit averagely divides the reading task of the directory file information into a plurality of subtasks according to the total number of the file information and the number of preset single subtask file information.
Specifically, after generating the read task of the directory file information, the first processing unit does not directly perform the read operation of the directory file information, but sends the read task to the second processing unit, and the computational power of the second processing unit is higher than that of the first processing unit.
The total number of file information is N, the number of preset single sub-task file information is M, the GPU operation module averagely divides the reading task of the directory file information into L sub-tasks according to the L which is N/M, and when the L obtained through calculation is a non-integer, the L adopts an upward rounding method. For example, the total number of file information is 1000, and the number of preset single subtask file information is 50, the GPU operation module divides the read task with the total number of file information of 1000 into 20 subtasks with the number of read file information of 50. For another example, the total number of the file information is 1020, and the number of the file information of a single subtask is preset to be 50, then the GPU operation module divides the read task with the total number of the file information of 1050 into 20 subtasks with the number of the read file information of 50 and 1 subtask with the number of the read file information of 20.
In an alternative embodiment, the number of the subtasks may be determined first, and then the number of the file information to be read allocated to each subtask may be calculated according to the total number of the file information and the number of the subtasks. For example, the number of the subtasks is 25, when the total number of the file information is 1000, the number of the file information to be read by each subtask is 40, and when the total number of the file information is 1200, the number of the file information to be read by each subtask is 48.
And S250, the second processing unit adopts a plurality of threads to simultaneously execute a plurality of subtasks to obtain a plurality of subdirectories of the directory file information.
Specifically, one thread executes one subtask, and the GPU operation module may execute a plurality of threads at the same time, and may execute a plurality of subtasks at the same time by using a plurality of threads. Each subtask executes to obtain a part of the directory file information, the part of the directory file information is called a subdirectory of the directory file information, and a plurality of subtasks execute simultaneously to obtain a plurality of subdirectories of the directory file information simultaneously.
And S260, the second processing unit recombines the plurality of subdirectories of the directory file information to obtain the directory file information.
Specifically, the GPU operation module rearranges the plurality of subdirectories of the directory file information according to the sorting rule of the directory file information, so that complete directory file information can be obtained.
S270, the second processing unit sends the directory file information to the first processing unit.
Specifically, the GPU operation module sends the read directory file information to the metadata management module.
S280, the first processing unit returns the directory file information to the inquiry end.
Specifically, the metadata management module returns the directory file information to the user query end, and the user end obtains the corresponding directory file information.
In the embodiment of the invention, the read task of the directory file information is sent to the GPU operation module for execution, and the GPU operation module divides the read task into a plurality of subtasks to be simultaneously executed, so that the read speed of the directory file information is improved, and the read time of the directory file information is reduced.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a Ceph directory file information reading apparatus according to a third embodiment of the present invention, which is applicable to reading directory file information using a file interface of a Ceph. The device can be realized in a software and/or hardware mode, can be integrated in a server, can realize the Ceph directory file information reading method provided by any embodiment of the invention, and has corresponding functions and modules of the execution method.
As shown in fig. 3, a Ceph directory file information reading apparatus according to a third embodiment of the present invention includes: the first processing unit 310 is configured to obtain a reading request of the directory file information, generate a reading task of the directory file information according to the reading request of the directory file information, and send the reading task of the directory file information to the second processing unit; the second processing unit 320 is configured to split the task of reading the directory file information into a plurality of subtasks according to a preset rule, and execute the plurality of subtasks to read the directory file information.
Further, the first processing unit 310 is further configured to:
acquiring the total number of file information corresponding to the directory file information according to the reading request of the directory file information;
and generating a reading task of the directory file information according to the total number of the file information corresponding to the directory file information.
Further, the second processing unit 320 is further configured to: and averagely dividing the reading task of the directory file information into a plurality of subtasks according to the total number of the file information and the number of preset single subtask file information.
Further, the second processing unit 320 is further configured to:
executing a plurality of subtasks by adopting a plurality of threads to obtain a plurality of subdirectories of the directory file information;
and recombining the plurality of subdirectories of the directory file information to obtain the directory file information.
Further, the second processing unit 320 sends the directory file information to the first processing unit 310, and the first processing unit 310 returns the directory file information to the querying end.
The embodiment of the invention is also provided with a first processing unit, a second processing unit and a processing unit, wherein the first processing unit is used for acquiring a reading request of the directory file information, generating a reading task of the directory file information according to the reading request of the directory file information and sending the reading task of the directory file information to the second processing unit; and the second processing unit is used for splitting the reading task of the directory file information into a plurality of subtasks according to a preset rule and executing the plurality of subtasks to read the directory file information. The reading of the directory file information is realized, the reading speed of the directory file information is improved, and the reading time of the directory file information is reduced.
Example four
Fig. 4 is a schematic structural diagram of a server according to a fourth embodiment of the present invention. FIG. 4 illustrates a block diagram of an exemplary server 412 suitable for use in implementing embodiments of the present invention. The server 412 shown in fig. 4 is only an example and should not bring any limitations to the function and scope of use of the embodiments of the present invention.
As shown in FIG. 4, server 412 is in the form of a general purpose server. Components of server 412 may include, but are not limited to: one or more processors 416, a storage device 428, and a bus 418 that couples the various system components including the storage device 428 and the processors 416.
A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in storage 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 442 generally perform the functions and/or methodologies of the described embodiments of the invention.
The server 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing server, display 424, etc.), with one or more servers that enable a user to interact with the server 412, and/or with any servers (e.g., network card, modem, etc.) that enable the server 412 to communicate with one or more other computing servers. Such communication may occur via input/output (I/O) interfaces 422. Further, server 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), Wide Area Network (WAN), and/or a public Network, such as the Internet) via Network adapter 420. As shown in FIG. 4, network adapter 420 communicates with the other modules of server 412 via bus 418. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the server 412, including but not limited to: microcode, server drives, Redundant processors, external disk drive Arrays, disk array (RAID) systems, tape drives, and data backup storage systems, to name a few.
The processor 416 executes various functional applications and data processing by running programs stored in the storage device 428, for example, implementing a Ceph directory file information reading method provided by any embodiment of the present invention, where the method may include:
a first processing unit acquires a reading request of directory file information;
the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information;
the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule;
the second processing unit executes a plurality of the subtasks to read the directory file information.
EXAMPLE five
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a Ceph directory file information reading method according to any embodiment of the present invention, where the method includes:
a first processing unit acquires a reading request of directory file information;
the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information;
the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule;
the second processing unit executes a plurality of the subtasks to read the directory file information.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (7)
1. A method for reading information of a Ceph directory file is characterized by comprising the following steps:
a first processing unit acquires a reading request of directory file information, wherein the first processing unit is a metadata management module of a Ceph file system;
the first processing unit generates a reading task of the directory file information according to a reading request of the directory file information;
the first processing unit sends the reading task of the directory file information to a second processing unit so that the second processing unit divides the reading task of the directory file information into a plurality of subtasks according to a preset rule; the computing power of the second processing unit is higher than that of the first processing unit;
the second processing unit executes a plurality of the subtasks to read the directory file information;
the second processing unit executing the plurality of subtasks to read the directory file information includes:
the second processing unit adopts a plurality of threads to simultaneously execute a plurality of subtasks to obtain a plurality of subdirectories of the directory file information;
the second processing unit recombines a plurality of subdirectories of the directory file information to obtain the directory file information;
after the second processing unit executes the plurality of subtasks to read the directory file information, the method includes:
the second processing unit sends the directory file information to the first processing unit;
and the first processing unit returns the directory file information to the inquiry end.
2. The method of claim 1, wherein the generating, by the first processing unit, the read task of the directory file information according to the read request of the directory file information comprises:
the first processing unit acquires the total number of file information corresponding to the directory file information according to the reading request of the directory file information;
and the first processing unit generates a reading task of the directory file information according to the total number of the file information corresponding to the directory file information.
3. The method of claim 2, wherein the second processing unit splitting the task of reading the directory file information into a plurality of subtasks according to a preset rule comprises:
and the second processing unit averagely divides the reading task of the directory file information into a plurality of subtasks according to the total number of the file information and the number of preset single subtask file information.
4. The method of any of claims 1-3, wherein the second processing unit is a GPU operation module.
5. A Ceph directory file information reading apparatus, comprising:
the first processing unit is used for acquiring a reading request of the directory file information, generating a reading task of the directory file information according to the reading request of the directory file information, and sending the reading task of the directory file information to the second processing unit; the first processing unit is a metadata management module of a Ceph file system;
the second processing unit is used for splitting a reading task of the directory file information into a plurality of subtasks according to a preset rule and executing the plurality of subtasks to read the directory file information; the computing power of the second processing unit is higher than that of the first processing unit;
the second processing unit is further configured to: executing a plurality of subtasks by adopting a plurality of threads to obtain a plurality of subdirectories of the directory file information; recombining a plurality of subdirectories of the directory file information to obtain the directory file information;
after the second processing unit executes the plurality of subtasks to read the directory file information, the method further includes: the second processing unit sends the directory file information to the first processing unit;
and the first processing unit returns the directory file information to the inquiry end.
6. A server, comprising:
one or more treatments;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processes, the one or more processes are caused to implement the Ceph directory file information reading method according to any one of claims 1 to 4.
7. A computer-readable storage medium on which a computer program is stored, the program being configured to implement, when executed, the Ceph directory file information reading method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911030299.4A CN110781159B (en) | 2019-10-28 | 2019-10-28 | Ceph directory file information reading method and device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911030299.4A CN110781159B (en) | 2019-10-28 | 2019-10-28 | Ceph directory file information reading method and device, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781159A CN110781159A (en) | 2020-02-11 |
| CN110781159B true CN110781159B (en) | 2021-02-02 |
Family
ID=69386997
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911030299.4A Active CN110781159B (en) | 2019-10-28 | 2019-10-28 | Ceph directory file information reading method and device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781159B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
| CN111625357A (en) * | 2020-05-22 | 2020-09-04 | 柏科数据技术（深圳）股份有限公司 | Directory reading method and device based on Feiteng platform, server and storage medium |
| CN112035501B (en) * | 2020-09-01 | 2023-09-19 | 中国银行股份有限公司 | User identifier updating method, device and equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104166538A (en) * | 2013-05-16 | 2014-11-26 | 北大方正集团有限公司 | Data task processing method and system |
| CN107590278A (en) * | 2017-09-28 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of pre-reading method of files and relevant apparatus based on CEPH |
| CN109597681A (en) * | 2018-10-22 | 2019-04-09 | 平安科技（深圳）有限公司 | Cloud control method, device, computer equipment and storage medium |
| CN109873868A (en) * | 2019-03-01 | 2019-06-11 | 深圳市网心科技有限公司 | A kind of computing capability sharing method, system and relevant device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331255B (en) * | 2014-11-17 | 2018-04-17 | 中国科学院声学研究所 | A kind of stream data read method based on embedded file system |
| KR101918806B1 (en) * | 2015-06-30 | 2018-11-14 | 전자부품연구원 | Cache Management Method for Optimizing the Read Performance of Distributed File System |
| CN105677892B (en) * | 2016-01-29 | 2018-12-25 | 华为技术有限公司 | A kind of method and device reading catalogue subitem metadata |
| CN106777180B (en) * | 2016-12-22 | 2020-09-01 | 北京京东金融科技控股有限公司 | Method, device and system for high-performance distributed data conversion |
| CN108319634B (en) * | 2017-12-15 | 2021-08-06 | 深圳创新科技术有限公司 | Directory access method and device for distributed file system |
| CN109766183A (en) * | 2018-12-28 | 2019-05-17 | 郑州云海信息技术有限公司 | A kind of method and system of cluster GPU multiplexing and intelligent load |
-
2019
- 2019-10-28 CN CN201911030299.4A patent/CN110781159B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104166538A (en) * | 2013-05-16 | 2014-11-26 | 北大方正集团有限公司 | Data task processing method and system |
| CN107590278A (en) * | 2017-09-28 | 2018-01-16 | 郑州云海信息技术有限公司 | A kind of pre-reading method of files and relevant apparatus based on CEPH |
| CN109597681A (en) * | 2018-10-22 | 2019-04-09 | 平安科技（深圳）有限公司 | Cloud control method, device, computer equipment and storage medium |
| CN109873868A (en) * | 2019-03-01 | 2019-06-11 | 深圳市网心科技有限公司 | A kind of computing capability sharing method, system and relevant device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781159A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6764989B2 (en) | Requirement processing technology | |
| CN110781159B (en) | Ceph directory file information reading method and device, server and storage medium | |
| MX2007014899A (en) | Back-off mechanism for search. | |
| US9063918B2 (en) | Determining a virtual interrupt source number from a physical interrupt source number | |
| US20160283522A1 (en) | Matching untagged data sources to untagged data analysis applications | |
| US20180225333A1 (en) | Data write/import performance in a database through distributed memory | |
| US20150007046A1 (en) | Management of an application for an electronic device | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| WO2022199206A1 (en) | Memory sharing method and device for virtual machines | |
| CN113886353B (en) | Data configuration recommendation method and device for hierarchical storage management software and storage medium | |
| US9298487B2 (en) | Managing virtual machine images in a distributed computing environment | |
| CN113127430A (en) | Mirror image information processing method and device, computer readable medium and electronic equipment | |
| US20220229858A1 (en) | Multi-cloud object store access | |
| US20190258736A1 (en) | Dynamic Execution of ETL Jobs Without Metadata Repository | |
| CN110753136B (en) | Domain name resolution method, device, equipment and storage medium | |
| Wang et al. | ODDS: Optimizing data-locality access for scientific data analysis | |
| CN111475279B (en) | System and method for intelligent data load balancing for backup | |
| US10169076B2 (en) | Distributed batch job promotion within enterprise computing environments | |
| CN112905596A (en) | Data processing method and device, computer equipment and storage medium | |
| US10528400B2 (en) | Detecting deadlock in a cluster environment using big data analytics | |
| CN114153620B (en) | Optimal allocation method and device for Hudi operating environment resources | |
| CN111008074B (en) | File processing method, device, equipment and medium | |
| CN112783925B (en) | Paging retrieval method and device | |
| CN111753017B (en) | Method and device for processing dimension table based on Kylin system, electronic equipment and storage medium | |
| US20210096763A1 (en) | Method, device, and computer program product for managing storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: No.3333 Liuxian Avenue, Fuguang community, Taoyuan Street, Nanshan District, Shenzhen, Guangdong 518000 Patentee after: BAIKE DATA TECHNOLOGY (SHENZHEN) Co.,Ltd. Address before: 518000 room 701-c, 7th floor, bike technology, No.9, Keke Road, high tech Zone, Nanshan District, Shenzhen City, Guangdong Province Patentee before: BAIKE DATA TECHNOLOGY (SHENZHEN) Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |