CN113076281B - Method, system, equipment and medium for communication of Ceph kernel client - Google Patents
Method, system, equipment and medium for communication of Ceph kernel client Download PDFInfo
- Publication number
- CN113076281B CN113076281B CN202110340423.8A CN202110340423A CN113076281B CN 113076281 B CN113076281 B CN 113076281B CN 202110340423 A CN202110340423 A CN 202110340423A CN 113076281 B CN113076281 B CN 113076281B
- Authority
- CN
- China
- Prior art keywords
- size
- data
- sent
- ceph
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
- G06F15/163—Interprocessor communication
- G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
- G06F15/17306—Intercommunication techniques
- G06F15/17331—Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/1827—Management specifically adapted to NAS
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a method, a system, equipment and a storage medium for communication of a Ceph kernel client, wherein the method comprises the following steps: establishing communication between a Ceph kernel client and Ceph distributed storage based on a RoCE protocol, and establishing and initializing a sending completion queue and a receiving completion queue; in response to the message being sent to the Ceph distributed storage and the type of the message being a non-blocking message, determining a first size of data to be sent according to the message; calculating a second size of data which can be sent at this time according to the size of the current idle cache, and judging whether the second size is smaller than the first size; responding to the second size smaller than the first size, acquiring a request sent to a sending completion queue, and releasing a cache corresponding to the request to update the current idle cache; and copying the data to be sent to the updated current idle buffer, submitting a request to a sending completion queue to send the data to be sent, updating the first size based on the size of the copied data, and sending the data again until the first size is zero.
Description
Technical Field
The present invention relates to the field of distributed file systems, and more particularly, to a method, a system, a computer device, and a readable medium for communication between Ceph kernel clients.
Background
The Ceph is a distributed file system, has the characteristics of high performance, high reliability and high expandability, and can provide object storage, block device storage and file system storage. When the Ceph provides file system storage, a user can access the file system through the NAS client, and when the client is a linux system, the user can also access the file system through an fs module in a linux kernel, the kernel client for the linux system has access to the Ceph storage file system currently, and a tcp/ip communication mode is adopted in network implementation. In the prior art, the communication delay of a Ceph kernel client is high, the CPU occupancy rate is high, and the communication efficiency is low.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, a computer device, and a computer-readable storage medium for communication by a Ceph kernel client, where the communication by the Ceph kernel client is performed based on an RoCE protocol, and the advantages of low latency, high bandwidth, and lower cpu occupation of the RoCE protocol can be fully exerted, so as to improve the read-write performance of the Ceph kernel client.
Based on the above object, an aspect of the embodiments of the present invention provides a method for enabling a Ceph core client to perform communication, including the following steps: establishing communication between a Ceph kernel client and a Ceph distributed storage based on a RoCE protocol, and establishing and initializing a sending completion queue and a receiving completion queue; responding to a message sent by a Ceph kernel client to the Ceph distributed storage, judging whether the type of the message is a non-blocking message, responding to the message of which the type is the non-blocking message, and determining a first size of data to be sent according to the message; acquiring a current idle cache, calculating a second size of data which can be sent at this time according to the size of the current idle cache, and judging whether the second size is smaller than the first size; responding to the second size smaller than the first size, acquiring a request sent to the sending completion queue, and releasing a cache corresponding to the request to update the current idle cache; and copying the data to be sent to the updated current idle buffer, submitting a request to the sending completion queue to send the data to be sent, updating the first size based on the size of the copied data, and sending the data again until the first size is zero.
In some embodiments, the establishing communication between the Ceph kernel client and the Ceph distributed storage based on the RoCE protocol comprises: resolving the address and the route according to the IP address, and creating and initializing a remote direct memory access connection context; and initiating a remote direct memory access connection, responsive to a successful connection, setting a state of the socket to connected.
In some embodiments, the method further comprises: a protection domain is created and event handlers and completion handlers of the send and receive completion queues are registered.
In some embodiments, the method further comprises: and in response to that the second size is not smaller than the first size and a request for sending to the sending completion queue is not obtained, writing the data to be sent of the first size into the current idle cache.
In some embodiments, the copying the data to be sent to the current idle buffer after updating includes: and obtaining indexes of the released cache and the current idle cache, and copying data to be sent to a single or a plurality of memory mapping blocks of the current idle cache after updating according to the indexes.
In some embodiments, the method further comprises: in response to receiving the message sent by the Ceph distributed storage, judging whether available data exists in a receiving cache; and in response to the available data existing in the receiving buffer, copying the available data and sending a request to the receiving completion queue.
In some embodiments, the method further comprises: responding to the absence of available data in a receiving cache, acquiring a completion request from the receiving completion queue, and judging whether the completion request meets a preset requirement; and responding to the completion request which does not accord with the preset requirement, finishing the receiving and prompting that an error occurs.
In another aspect of the embodiments of the present invention, a system for enabling a Ceph kernel client to perform communication is provided, including: the system comprises a creating module, a sending module and a receiving module, wherein the creating module is configured to establish communication between a Ceph kernel client and a Ceph distributed storage based on a RoCE protocol, and create and initialize a sending completion queue and a receiving completion queue; the first judgment module is configured to respond to a message sent by a Ceph kernel client to the Ceph distributed storage, judge whether the type of the message is a non-blocking message, respond to the message with the type of the non-blocking message, and determine a first size of data to be sent according to the message; the second judgment module is configured to acquire a current idle cache, calculate a second size of data which can be sent this time according to the size of the current idle cache, and judge whether the second size is smaller than the first size; the cache module is configured to respond to the second size being smaller than the first size, acquire a request sent to the sending completion queue, and release a cache corresponding to the request to update the current idle cache; and the execution module is configured to copy the data to be sent to the updated current idle cache, submit a request to the sending completion queue to send the data to be sent, update the first size based on the size of the copied data, and send the data again until the first size is zero.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method as above.
In another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has the following beneficial technical effects: the communication of the Ceph kernel client is carried out based on the RoCE protocol, the advantages of low delay, high bandwidth and lower CPU occupation of the RoCE protocol can be fully exerted, and therefore the read-write performance of the Ceph kernel client is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a schematic diagram of an embodiment of a method for enabling a Ceph kernel client to communicate according to the present invention;
fig. 2 is a flowchart illustrating a connection between a Ceph kernel client and a Ceph distributed storage according to an embodiment of the present invention;
fig. 3 is a flowchart of data transmission of a Ceph kernel client according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating the reception of data by the Ceph kernel client according to an embodiment of the present invention;
fig. 5 is a schematic hardware structure diagram of an embodiment of a computer device in which a Ceph kernel client communicates according to the present invention;
fig. 6 is a schematic diagram of an embodiment of a computer storage medium in which a Ceph kernel client communicates according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In view of the foregoing, a first aspect of the embodiments of the present invention provides an embodiment of a method for enabling a Ceph kernel client to perform communication. Fig. 1 is a schematic diagram illustrating an embodiment of a method for enabling a Ceph kernel client to communicate according to the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, establishing communication between a Ceph kernel client and a Ceph distributed storage based on a RoCE protocol, and establishing and initializing a sending completion queue and a receiving completion queue;
s2, responding to a message sent by a Ceph kernel client to the Ceph distributed storage, judging whether the type of the message is a non-blocking message or not, responding to the message with the type of the non-blocking message, and determining a first size of data to be sent according to the message;
s3, acquiring a current idle cache, calculating a second size of data which can be sent at this time according to the size of the current idle cache, and judging whether the second size is smaller than the first size;
s4, responding to the fact that the second size is smaller than the first size, obtaining a request sent to the sending completion queue, and releasing a cache corresponding to the request to update the current idle cache; and
and S5, copying the data to be sent to the updated current idle buffer, submitting a request to a sending completion queue to send the data to be sent, updating the first size based on the size of the copied data, and sending the data again until the first size is zero.
RoCE (RDMA over Converged Ethernet) denotes RDMA protocol over Ethernet, RDMA (Remote Direct Memory Access) denotes Remote Direct Memory Access.
And establishing communication between the Ceph core client and the Ceph distributed storage based on the RoCE protocol, and creating and initializing a sending completion queue and a receiving completion queue.
In some embodiments, the establishing communication between the Ceph kernel client and the Ceph distributed storage based on the RoCE protocol comprises: resolving the address and route according to the IP address, and creating and initializing a remote direct memory access connection context; and initiating a remote direct memory access connection, responsive to a successful connection, setting a state of the socket to connected.
In some embodiments, the method further comprises: a protection domain is created and event handlers and completion handlers of the send and receive completion queues are registered.
Fig. 2 is a flowchart illustrating a Ceph kernel client establishing a connection with a Ceph distributed storage according to an embodiment of the present invention. As shown in fig. 2, an icfs _ connection structure is initialized, and the socket state is set to NEW; initializing a rdma _ socket structure, setting the socket state to be UNCONNECTED, and setting the sending/receiving cache size; creating cm _ id, and setting the socket state as CONNECTING (in connection); a registered cm event processing function (icfs _ cm _ handler) used for processing state switching in the RDMA connection process; setting a cm _ id service type; according to the far-end IP address, performing address resolution, and waiting for the completion of the address resolution (synchronous waiting is realized based on the icfs _ cm _ handler); analyzing the route and waiting for the completion of the analysis (synchronous waiting is realized based on the icfs _ cm _ handler); creating and initializing an RDMA connection context, creating a PD (protection domain), initializing a sending work queue and a receiving work queue, allocating an initialization sending and receiving buffer, completing registration, initializing a sending completion queue and a receiving completion queue, and registering a related completion handler and an event handler; creating QP (queue pair), submitting request to the receiving completion queue, and making ready for receiving; initiating the rdma connection and waiting for the connection to complete, if the connection is successful, setting the socket state to CONNECTED.
And in response to the message sent by the Ceph kernel client to the Ceph distributed storage, judging whether the type of the message is a non-blocking message, and in response to the message being the non-blocking message, determining the first size of the data to be sent according to the message. And acquiring the current idle cache, calculating the second size of the data which can be sent at this time according to the size of the current idle cache, and judging whether the second size is smaller than the first size. And responding to the second size smaller than the first size, acquiring a request sent to the sending completion queue, and releasing a cache corresponding to the request to update the current idle cache. For example, it is determined that the first size is 100G according to the message, and if the second size of the data that can be sent this time is 50G, the request sent to the sending completion queue is acquired, and the buffer corresponding to the request is released, for example, 40G is released, and then the data to be sent may be sent according to the latest second size of 90G.
In some embodiments, the method further comprises: and in response to that the second size is not smaller than the first size and that a request for sending to the sending completion queue is not acquired, writing the data to be sent of the first size into the current idle cache. For example, the first size is determined to be 100G according to the message, and the second size of the data that can be sent this time is 120G, then the data to be sent is directly written in the idle buffer.
Copying the data to be sent to the updated current idle buffer, submitting a request to a sending completion queue to send the data to be sent, updating the first size based on the size of the copied data, and sending the data again until the first size is zero.
In some embodiments, the copying the data to be sent to the current idle buffer after updating includes: and obtaining indexes of the released cache and the current idle cache, and copying data to be sent to a single or a plurality of memory mapping blocks of the current idle cache after updating according to the indexes.
Fig. 3 is a flowchart illustrating data transmission by a Ceph kernel client in the embodiment of the present invention. As shown in fig. 3, it is determined whether the message is a non-blocking message, if so, it is checked whether there is an available sending buffer, and if there is an available sending buffer, the number and size of the sending buffers are obtained; if no available transmission buffer memory exists, the request which is successfully transmitted is obtained, the buffer memory of the corresponding request is released, and the state of the buffer memory is set to be available. Calculating the size of the data which can be sent at this time according to the size and the number of the available caches, judging whether the data still remain to be sent, if the data which remains to be sent and the available caches exist, acquiring the index of the available caches, copying the data into the sending caches, submitting a sending request to a sending completion queue, judging whether the data still remain to be sent again until the data which remains to be sent does not exist, and ending the whole process.
In some embodiments, the method further comprises: in response to receiving the message sent by the Ceph distributed storage, judging whether available data exists in a receiving cache; and responding to the available data in the receiving buffer, copying the available data and sending a request to the receiving completion queue.
In some embodiments, the method further comprises: responding to the fact that no available data exists in a receiving cache, obtaining a completion request from the receiving completion queue, and judging whether the completion request meets a preset requirement or not; and responding to the completion request which does not accord with the preset requirement, finishing the receiving and prompting that an error occurs.
Fig. 4 is a flow chart illustrating the receipt of Ceph kernel client data in an embodiment of the present invention. As shown in fig. 4, it is checked whether there is data available in the receive buffer; and if the available data exists, copying the data in the receiving buffer into the incoming buffer until the data in the receiving buffer is completely copied, submitting a recv request to a receiving completion queue, and then ending. If no data is available, obtaining a completion request from the receive completion queue; judging whether the request is a correct request or not, and if so, marking the cache corresponding to the request as an available cache; if the completion request is not a correct request, the receiving is finished, and an error is returned. Whether the completion request is a correct request or not can be judged according to a preset standard, and if the completion request meets the preset standard, the completion request is a correct request.
It should be particularly noted that, the steps in the embodiments of the method for enabling a Ceph core client to communicate described above may be intersected, replaced, added, or deleted, and therefore, these methods for enabling a Ceph core client to communicate through reasonable permutation and combination transformation also belong to the scope of the present invention, and the scope of the present invention should not be limited to the embodiments.
In view of the above, a second aspect of the embodiments of the present invention provides a system for enabling a Ceph kernel client to perform communication, including: the system comprises a creating module, a sending module and a receiving module, wherein the creating module is configured to establish communication between a Ceph kernel client and a Ceph distributed storage based on a RoCE protocol, and create and initialize a sending completion queue and a receiving completion queue; the first judgment module is configured to respond to a message sent by a Ceph kernel client to the Ceph distributed storage, judge whether the type of the message is a non-blocking message, respond to the message with the type of the non-blocking message, and determine a first size of data to be sent according to the message; the second judgment module is configured to acquire a current idle cache, calculate a second size of data which can be sent this time according to the size of the current idle cache, and judge whether the second size is smaller than the first size; the cache module is configured to respond to the second size being smaller than the first size, acquire a request sent to the sending completion queue, and release a cache corresponding to the request to update the current idle cache; and the execution module is configured to copy the data to be sent to the updated current idle cache, submit a request to the sending completion queue to send the data to be sent, update the first size based on the size of the copied data, and send the data again until the first size is zero.
In some embodiments, the creation module is configured to: resolving the address and route according to the IP address, and creating and initializing a remote direct memory access connection context; and initiating a remote direct memory access connection, responsive to a successful connection, setting a state of the socket to connected.
In some embodiments, the system further comprises a registration module configured to: a protection domain is created, and event handlers and completion handlers of the send completion queue and the receive completion queue are registered.
In some embodiments, the system further comprises a second caching module configured to: and in response to that the second size is not smaller than the first size and that a request for sending to the sending completion queue is not acquired, writing the data to be sent of the first size into the current idle cache.
In some embodiments, the execution module is configured to: and obtaining indexes of the released cache and the current idle cache, and copying data to be sent to a single or a plurality of memory mapping blocks of the current idle cache after updating according to the indexes.
In some embodiments, the system further comprises a third determining module configured to: responding to the received message sent by the Ceph distributed storage, and judging whether available data exists in a receiving cache or not; and in response to the available data existing in the receiving buffer, copying the available data and sending a request to the receiving completion queue.
In some embodiments, the system further comprises a termination module configured to: responding to the absence of available data in a receiving cache, acquiring a completion request from the receiving completion queue, and judging whether the completion request meets a preset requirement; and responding to the completion request which does not accord with the preset requirement, finishing the receiving and prompting that an error occurs.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, establishing communication between a Ceph kernel client and a Ceph distributed storage based on a RoCE protocol, and establishing and initializing a sending completion queue and a receiving completion queue; s2, responding to a message sent by a Ceph kernel client to the Ceph distributed storage, judging whether the type of the message is a non-blocking message or not, responding to the message with the type of the non-blocking message, and determining a first size of data to be sent according to the message; s3, acquiring a current idle cache, calculating a second size of data which can be sent at this time according to the size of the current idle cache, and judging whether the second size is smaller than the first size; s4, responding to the fact that the second size is smaller than the first size, obtaining a request sent to the sending completion queue, and releasing a cache corresponding to the request to update the current idle cache; and S5, copying the data to be sent to the updated current idle buffer, submitting a request to a sending completion queue to send the data to be sent, updating the first size based on the size of the copied data, and sending the data again until the first size is zero.
In some embodiments, the establishing communication between the Ceph kernel client and the Ceph distributed storage based on the RoCE protocol comprises: resolving the address and the route according to the IP address, and creating and initializing a remote direct memory access connection context; and initiating a remote direct memory access connection, responsive to a successful connection, setting a state of the socket to connected.
In some embodiments, the steps further comprise: a protection domain is created and event handlers and completion handlers of the send and receive completion queues are registered.
In some embodiments, the steps further comprise: and in response to that the second size is not smaller than the first size and a request for sending to the sending completion queue is not obtained, writing the data to be sent of the first size into the current idle cache.
In some embodiments, the copying the data to be sent to the current idle buffer after updating includes: and obtaining indexes of the released cache and the current idle cache, and copying data to be sent to a single or a plurality of memory mapping blocks of the current idle cache after updating according to the indexes.
In some embodiments, the steps further comprise: in response to receiving the message sent by the Ceph distributed storage, judging whether available data exists in a receiving cache; and responding to the available data in the receiving buffer, copying the available data and sending a request to the receiving completion queue.
In some embodiments, the steps further comprise: responding to the fact that no available data exists in a receiving cache, obtaining a completion request from the receiving completion queue, and judging whether the completion request meets a preset requirement or not; and responding to the completion request which does not accord with the preset requirement, finishing the receiving and prompting that an error occurs.
Fig. 5 is a schematic hardware structure diagram of an embodiment of the computer device for communicating with the Ceph kernel client according to the present invention.
Taking the apparatus shown in fig. 5 as an example, the apparatus includes a processor 201 and a memory 202, and may further include: an input device 203 and an output device 204.
The processor 201, the memory 202, the input device 203 and the output device 204 may be connected by a bus or other means, and fig. 5 illustrates the connection by a bus as an example.
The memory 202, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for enabling the Ceph kernel client to communicate in the embodiments of the present application. The processor 201 executes various functional applications and data processing of the server by running the nonvolatile software programs, instructions and modules stored in the memory 202, that is, the method for enabling the Ceph kernel client to communicate in the above method embodiments.
The memory 202 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of a method of communication by the Ceph kernel client, and the like. Further, the memory 202 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 202 may optionally include memory located remotely from processor 201, which may be connected to local modules via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 203 may receive information such as a user name and a password that are input. The output device 204 may include a display device such as a display screen.
Program instructions/modules corresponding to one or more methods for enabling a Ceph core client to communicate are stored in the memory 202, and when executed by the processor 201, perform the method for enabling a Ceph core client to communicate in any of the above-described method embodiments.
Any embodiment of the computer device executing the method for enabling the Ceph kernel client to communicate can achieve the same or similar effects as any corresponding method embodiment.
The invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the method as above.
Fig. 6 is a schematic diagram of an embodiment of a computer storage medium for communicating with the Ceph kernel client according to the present invention. Taking the computer storage medium as shown in fig. 6 as an example, the computer readable storage medium 3 stores a computer program 31 which, when executed by a processor, performs the method as described above.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and the program of the method for enabling a Ceph kernel client to communicate can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments corresponding thereto.
The foregoing are exemplary embodiments of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant only to be exemplary, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (9)
1. A method for communication of a Ceph kernel client is characterized by comprising the following steps:
establishing communication between a Ceph kernel client and Ceph distributed storage based on a RoCE protocol, and establishing and initializing a sending completion queue and a receiving completion queue;
responding to the message sent by the client side of the Ceph kernel to the Ceph distributed storage, judging whether the type of the message is a non-blocking message, responding to the message with the type of the non-blocking message, and determining a first size of data to be sent according to the message;
acquiring a current idle cache, calculating a second size of data which can be sent at this time according to the size of the current idle cache, and judging whether the second size is smaller than the first size;
responding to the second size smaller than the first size, acquiring a request sent to the sending completion queue, and releasing a cache corresponding to the request to update the current idle cache; and
copying data to be sent to the current idle cache after updating, submitting a request to the sending completion queue to send the data to be sent, updating the first size based on the size of the copied data and sending the data again until the first size is zero,
the method further comprises the following steps:
in response to receiving the message sent by the Ceph distributed storage, judging whether available data exists in a receiving cache; and
and in response to the available data in the receiving buffer, copying the available data and sending a request to the receiving completion queue.
2. The method of claim 1, wherein establishing communication between a Ceph kernel client and a Ceph distributed storage based on the RoCE protocol comprises:
resolving the address and route according to the IP address, and creating and initializing a remote direct memory access connection context; and
a remote direct memory access connection is initiated, and in response to a successful connection, the state of the socket is set to connected.
3. The method of claim 2, further comprising:
a protection domain is created and event handlers and completion handlers of the send and receive completion queues are registered.
4. The method of claim 1, further comprising:
and in response to that the second size is not smaller than the first size and that a request for sending to the sending completion queue is not acquired, writing the data to be sent of the first size into the current idle cache.
5. The method of claim 1, wherein copying the data to be sent to the current idle buffer after updating comprises:
and obtaining indexes of the released cache and the current idle cache, and copying data to be sent to a single or a plurality of memory mapping blocks of the current idle cache after updating according to the indexes.
6. The method of claim 1, further comprising:
responding to the absence of available data in a receiving cache, acquiring a completion request from the receiving completion queue, and judging whether the completion request meets a preset requirement;
and responding to the completion request which does not accord with the preset requirement, finishing the receiving and prompting that an error occurs.
7. A system for communication of a Ceph kernel client, comprising:
the system comprises a creating module, a sending module and a receiving module, wherein the creating module is configured to establish communication between a Ceph kernel client and a Ceph distributed storage based on a RoCE protocol, and create and initialize a sending completion queue and a receiving completion queue;
the first judging module is configured to respond to the message sent by the Ceph kernel client to the Ceph distributed storage, judge whether the type of the message is a non-blocking message, and respond to the message with the type of the non-blocking message, and determine a first size of data to be sent according to the message;
the second judgment module is configured to acquire a current idle cache, calculate a second size of data which can be sent this time according to the size of the current idle cache, and judge whether the second size is smaller than the first size;
the cache module is configured to respond to the second size being smaller than the first size, acquire a request sent to the sending completion queue, and release a cache corresponding to the request to update the current idle cache; and
an execution module configured to copy data to be transmitted into the updated current idle buffer, submit a request to the transmission completion queue to transmit the data to be transmitted, update the first size based on the size of the copied data and transmit the data again until the first size is zero,
the system further comprises a third determination module configured to: in response to receiving the message sent by the Ceph distributed storage, judging whether available data exists in a receiving cache; and in response to the available data existing in the receiving buffer, copying the available data and sending a request to the receiving completion queue.
8. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 6.
9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110340423.8A CN113076281B (en) | 2021-03-30 | 2021-03-30 | Method, system, equipment and medium for communication of Ceph kernel client |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110340423.8A CN113076281B (en) | 2021-03-30 | 2021-03-30 | Method, system, equipment and medium for communication of Ceph kernel client |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113076281A CN113076281A (en) | 2021-07-06 |
| CN113076281B true CN113076281B (en) | 2022-11-04 |
Family
ID=76611450
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110340423.8A Active CN113076281B (en) | 2021-03-30 | 2021-03-30 | Method, system, equipment and medium for communication of Ceph kernel client |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113076281B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918101B (en) * | 2021-12-09 | 2022-03-15 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium for writing data cache |
| CN114817090B (en) * | 2022-06-09 | 2023-06-02 | 远峰科技股份有限公司 | MCU communication management method and system with low RAM consumption |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111381988A (en) * | 2020-03-24 | 2020-07-07 | 北京奇艺世纪科技有限公司 | Request speed limiting method and device, electronic equipment and storage medium |
| CN111756811A (en) * | 2020-05-29 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for actively pushing distributed system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103345368B (en) * | 2013-07-18 | 2015-05-06 | 内蒙古科技大学 | Data caching method in buffer storage |
| KR102310188B1 (en) * | 2015-08-10 | 2021-10-08 | 한국전자기술연구원 | Receiver device for transmitting/receiving data based on rdma and method for transmitting/receiving data thereof |
| EP3358456A4 (en) * | 2016-12-05 | 2018-08-08 | Huawei Technologies Co., Ltd. | Control method, storage device and system for data read/write command in nvme over fabric architecture |
| CN107948334B (en) * | 2018-01-09 | 2019-06-07 | 无锡华云数据技术服务有限公司 | Data processing method based on distributed memory system |
| CN111324305B (en) * | 2020-02-16 | 2021-02-02 | 西安奥卡云数据科技有限公司 | Data writing/reading method in distributed storage system |
-
2021
- 2021-03-30 CN CN202110340423.8A patent/CN113076281B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111381988A (en) * | 2020-03-24 | 2020-07-07 | 北京奇艺世纪科技有限公司 | Request speed limiting method and device, electronic equipment and storage medium |
| CN111756811A (en) * | 2020-05-29 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for actively pushing distributed system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113076281A (en) | 2021-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6564960B2 (en) | Networking technology | |
| KR102429904B1 (en) | Method and system for maximizing PCI-express bandwidth of peer-to-peer(P2P) connections | |
| US10592464B2 (en) | Methods for enabling direct memory access (DMA) capable devices for remote DMA (RDMA) usage and devices thereof | |
| CN109690510B (en) | Multicast apparatus and method for distributing data to multiple receivers in high performance computing networks and cloud-based networks | |
| CN113076281B (en) | Method, system, equipment and medium for communication of Ceph kernel client | |
| US8544025B2 (en) | Efficient data transfer on local network connections using a pseudo socket layer | |
| WO2017028399A1 (en) | Communication data transmission method and system | |
| WO2017032152A1 (en) | Method for writing data into storage device and storage device | |
| WO2022017475A1 (en) | Data access method and related device | |
| CN110798520B (en) | Service processing method, system, device and readable storage medium | |
| CN112822300B (en) | RDMA (remote direct memory Access) -based data transmission method and device and electronic equipment | |
| JP2806778B2 (en) | Conversion index buffer clear instruction processing method | |
| CN111131470B (en) | Terminal device, data processing method thereof and data processing system | |
| CN113422793A (en) | Data transmission method and device, electronic equipment and computer storage medium | |
| CN111026532B (en) | Message queue management method for voice data | |
| CN111913817A (en) | Data transmission method, system and equipment and computer readable storage medium | |
| CN112822299B (en) | RDMA (remote direct memory Access) -based data transmission method and device and electronic equipment | |
| KR20050112912A (en) | System and method for relaying data by use of socket applicaton program | |
| CN113326151A (en) | Inter-process communication method, device, equipment, system and storage medium | |
| CN109710183B (en) | Data synchronization method and device | |
| CN115509435A (en) | Data reading and writing method, device, equipment and medium | |
| CN111580987A (en) | Remote procedure calling method and device, storage medium and terminal equipment | |
| CN110519242A (en) | Data transmission method and device | |
| CN111049754B (en) | Data communication method, device, equipment and computer readable storage medium | |
| CN117938844A (en) | Information transmission method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |