WO2015139470A1 - Data write/read method and apparatus in rdma and network system - Google Patents
Data write/read method and apparatus in rdma and network system Download PDFInfo
- Publication number
- WO2015139470A1 WO2015139470A1 PCT/CN2014/091263 CN2014091263W WO2015139470A1 WO 2015139470 A1 WO2015139470 A1 WO 2015139470A1 CN 2014091263 W CN2014091263 W CN 2014091263W WO 2015139470 A1 WO2015139470 A1 WO 2015139470A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- rdma
- memory
- specified address
- address
- data
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0661—Format or protocol conversion arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
Definitions
- the present invention relates to the field of communications, and in particular, to a method, apparatus, and network system for reading and writing data under remote direct memory access (RDMA).
- RDMA remote direct memory access
- RDMA refers to direct memory access from the memory of one computer to the memory of another computer, without involving the operating system of any computer. RDMA allows for high-throughput, low-latency network connections, which is especially useful in large-scale parallel computer clusters.
- RDMA technology supports three data read and write modes: send/receive, RDMA read, and RDMA write. Among them, in the case where the client reads or writes the distributed memory address to the server, the RDMA read and the RDMA write operation may not be performed normally.
- the embodiment of the invention provides a method, a device and a network system for reading and writing RDMA write data, which can support RDMA read operation and RDMA write operation when the client reads or writes a distributed memory address to the server. .
- a method for remote data access to read data under RDMA comprising:
- the first end sends an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is a plurality of addresses dispersed in the memory area of the second end ;
- the first end receives data corresponding to the specified address sent by the second end.
- the method before the sending, by the first end, the RDMA read request message to the second end, the method further includes:
- the first end acquires memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
- the memory area of the second end includes a plurality of distributed memory areas, and the first end acquires a memory area of the second end
- the memory information includes:
- the first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
- the specified address includes multiple addresses that are distributed in the multiple distributed memory regions
- the receiving, by the first end, the data corresponding to the specified address sent by the second end includes:
- the first end receives data corresponding to each address in the specified address sent by the second end.
- the method before the sending, by the first end, the RDMA read request message to the second end, the method further includes:
- the first end registers the network card with its own memory area.
- a method for remote data access to read data under RDMA comprising:
- the second end receives the RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is a plurality of distributed areas within the memory area of the second end address;
- the second end reads data corresponding to the specified address
- the second end sends data corresponding to the specified address to the first end.
- the reading, by the second end, the data corresponding to the specified address includes: the second end reading, corresponding to each address in the specified address data;
- a plurality of distributed memory regions exist in the second end, where the specified address includes multiple addresses distributed in the plurality of distributed memory regions, then
- the reading, by the second end, the data corresponding to the specified address is: the second end reads data corresponding to each address in the specified address;
- the sending, by the second end, the data corresponding to the specified address to the first end includes: the second The terminal sends data corresponding to each address in the specified address to the first end.
- the method before the receiving, by the second end, the RDMA read request message sent by the first end, the method further includes:
- the second end registers the network card with its own memory area.
- a method for remote data access to data writes under RDMA comprising:
- the first end encapsulates the data corresponding to the specified address and the specified address in the RDMA write data packet, where the specified address is a plurality of addresses dispersed within the memory area of the second end;
- the first end sends the RDMA write data message to the second end.
- the method further includes:
- the first end acquires memory information of the memory area of the second end, where the memory information of the memory area includes: address information and/or length information of the memory area.
- the memory area of the second end includes a plurality of distributed memory areas, and the first end acquires the memory area of the second end
- the memory information includes:
- the first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
- the specified address includes a plurality of addresses distributed in the plurality of scatter memory regions.
- the method further includes:
- the first end receives an acknowledgement message sent by the second end indicating successful writing.
- a method for remote data access to data writes under RDMA comprising:
- the second end receives the RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is the Multiple addresses dispersed within the memory area of the second end;
- the second end writes data corresponding to the specified address at the specified address.
- a plurality of distributed memory regions exist in the second end, where the specified address includes multiple addresses distributed in the plurality of distributed memory regions,
- Writing, by the second end, the corresponding data at the specified address includes: the second end writing data corresponding to the address in each of the specified addresses.
- an apparatus for data reading under RDMA comprising:
- a sending module configured to send an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed in the memory area of the second end Address
- the receiving module is configured to receive data corresponding to the specified address sent by the second end.
- the device further includes:
- an obtaining module configured to acquire, after the sending module sends an RDMA read request message to the second end, the memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
- the memory area of the second end includes a plurality of scatter memory areas
- the acquiring module is specifically configured to:
- the specified address includes a plurality of addresses distributed in the plurality of distributed memory regions
- the receiving module is specifically configured to:
- the device further includes:
- the registration module is configured to register the network area of the memory module with the VR card before sending the RDMA read request message to the second end.
- an apparatus for data reading under RDMA comprising:
- a receiving module configured to receive an RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed within a memory area of the second end Multiple addresses;
- a processing module configured to read data corresponding to the specified address
- a sending module configured to send data corresponding to the specified address to the first end.
- the processing module is specifically configured to: read data corresponding to each address in the specified address;
- the sending module is specifically configured to: send data corresponding to each address in the specified address to the first end.
- a plurality of distributed memory regions exist in the second end, and the specified address includes multiple addresses distributed in the plurality of distributed memory regions, then
- the processing module is specifically configured to: read data corresponding to each address in the specified address;
- the sending module is specifically configured to: send data corresponding to each address in the specified address to the first end.
- the device further includes:
- the registration module is configured to register the memory area of the first end with the network card before receiving the RDMA read request message sent by the first end.
- an apparatus for data writing under RDMA comprising:
- a processing module configured to encapsulate data corresponding to the specified address and the specified address in the RDMA write data packet, where the specified address is a plurality of addresses dispersed within the memory area of the second end ;
- a sending module configured to send the RDMA write data message to the second end.
- the device further includes:
- An acquiring module configured to acquire, after the processing module encapsulates the data corresponding to the data to be written and the data corresponding to the specified address, in the RDMA write data packet, the memory information of the memory area of the second end, where The memory information of the memory area includes: address information and/or length information of the memory area.
- the memory area of the second end includes a plurality of scatter memory areas, and the acquiring module is specifically configured to:
- the specified address includes a plurality of addresses distributed in the plurality of distributed memory regions.
- the device further includes:
- the receiving module is configured to: after the sending module sends the RDMA write data message to the second end, receive an acknowledgement message sent by the second end indicating successful writing.
- an apparatus for data writing under RDMA comprising:
- a receiving module configured to receive an RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is a plurality of addresses dispersed within a memory area of the second end;
- a processing module configured to write data corresponding to the specified address at the specified address.
- a plurality of distributed memory regions exist in the second end, and the specified address includes multiple addresses distributed in the plurality of distributed memory regions,
- the processing module is specifically configured to: write data corresponding to the address in each of the specified addresses.
- a network system comprising any one of the apparatus for reading RDMA data mentioned in the above fifth aspect, and the apparatus for reading any of the RDMA data mentioned in the sixth aspect above .
- a network system comprising any one of the apparatus for writing RDMA data mentioned in the seventh aspect, and any apparatus for writing RDMA data mentioned in the above eighth aspect .
- the method, device and network system for reading and writing data under RDMA when the first end performs RDMA read and write operations, can read data in the memory area of the second end.
- a plurality of distributed addresses are sent to the second end, and the second end performs reading and writing of data of the distributed address to implement data read and write operations under RDMA.
- FIG. 1 is a flowchart of a method for reading data under RDMA according to an embodiment of the present invention
- FIG. 3 is a flowchart of a method for writing data under RDMA according to an embodiment of the present invention
- FIG. 5 is a schematic diagram of a method for reading data under RDMA according to an embodiment of the present invention.
- FIG. 6 is a schematic diagram of a method for writing data under RDMA according to an embodiment of the present invention.
- FIGS. 7a-7c are structural block diagrams of an apparatus for reading data under RDMA according to an embodiment of the present invention.
- FIGS. 8a-8b are structural block diagrams of an apparatus for reading data under an RD MA according to an embodiment of the present invention.
- 9a-9c are block diagrams showing the structure of an apparatus for writing data under RDMA according to an embodiment of the present invention.
- FIG. 10 is a structural block diagram of an apparatus for writing data under RDMA according to an embodiment of the present invention.
- a method for reading data under RDMA according to an embodiment of the present invention includes:
- the first end sends an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is dispersed in the memory area of the second end. Addresses.
- first end and the second end of the embodiment of the present invention are equal to each other, and they are distinguished only for convenience of description.
- the first end may be a client, and the second end may be a server; otherwise, the first end may also be a server, and the second end may be a client.
- the method for reading the data in the RDMA may further include: the first end acquiring the second Memory information of the memory area of the end.
- the memory information of the memory area may include: address information and/or length information of the memory area.
- the acquiring, by the first end, the memory information of the memory area of the second end may include:
- the first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
- the method for obtaining the memory information of the memory area of the second end is not limited, and any manner of obtaining the memory information of the memory area of the second end by using the interaction between the first end and the second end may be adopted.
- the first end may also register the network card with its own memory area.
- the first end receives data corresponding to the specified address sent by the second end.
- the receiving, by the first end, the data corresponding to the specified address sent by the second end may include: the first end receiving each of the specified addresses sent by the second end The data corresponding to the address.
- each of the specified addresses may exist in different dispersed memory areas, and at this time, data corresponding to each address in the specified address comes from Data in different disparate memory regions.
- the RDMA read request message sent by the first end to the second end may carry the memory information of the dispersed memory area corresponding to each address in the specified address, in addition to the specified address. In this way, after the second end knows the memory information of the dispersed memory area corresponding to each address in the specified address, the data corresponding to the address can be read in the corresponding distributed memory area.
- the first end when the first end performs the RDMA read operation, the first end can send a plurality of distributed addresses in the memory area of the second end that need to read data to the second end.
- the data of each distributed address is read by the second end to implement the data reading operation under RDMA.
- a method for writing data under RDMA according to an embodiment of the present invention includes:
- the second end receives the RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is dispersed within the memory area of the second end. Multiple addresses.
- the RDMA read request message further carries the memory information of the dispersed memory area corresponding to each address in the specified address.
- the memory information of the dispersed memory area may include address information and/or length information of the dispersed memory area.
- the first end may also register the network card with its own memory area.
- the second end reads data corresponding to the specified address.
- the reading, by the second end, the data corresponding to the specified address, when the plurality of addresses are in the same memory area of the second end, may include: the second end reads the Designation The data corresponding to each address in the address.
- step 22 may be specifically: the second end reads the The data corresponding to each address in the specified address, wherein the data corresponding to each address may be located in a different distributed memory area.
- the second end may directly read the address corresponding to the address in the scatter memory area corresponding to each address. data.
- the second end sends data corresponding to the specified address to the first end.
- the sending, by the second end, the data corresponding to the specified address to the first end by the second end may include: the second end sending, to the first end, data corresponding to each address in the specified address.
- the step 23 may be specifically: the second end is The first end sends data corresponding to each address in the specified address.
- the data corresponding to each address may be located in different dispersed memory areas.
- the method for reading data in the RDMA provided by the embodiment of the present invention, after the first end acquires the memory information of the dispersed memory area at the second end, the RDMA read operation can be performed in the range of the dispersed memory area of the second end.
- the address of the read data is sent to the second end, and the data of the address is read by the second end to implement the data reading operation under the RDMA.
- FIG. 3 is a flowchart of a method for writing data under RDMA according to an embodiment of the present invention.
- a method for writing data under RDMA according to an embodiment of the present invention includes:
- the first end encapsulates the data corresponding to the data to be written and the data corresponding to the specified address in an RDMA write data packet, where the specified address is a plurality of addresses dispersed in the memory area of the second end. .
- the first end may acquire the data of the second end of the first end before the first end of the data is encapsulated in the RDMA write data packet.
- the memory information of the memory area may include: address information and/or length information of the memory area.
- the memory area of the second end may include a plurality of distributed memory areas.
- the acquiring, by the first end, the memory information of the memory area of the second end may include:
- the first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
- the present invention is not limited to how to obtain the memory information of the memory area of the second end, and any manner of obtaining the memory information of the memory area of the second end by using the interaction between the first end and the second end may be adopted.
- the specified address may include multiple addresses distributed in the plurality of distributed memory regions.
- the RDMA write data packet may further carry the memory information of the distributed memory area corresponding to each address in the specified address.
- the first end can also register the network card with its own memory area.
- the first end sends the RDMA write data packet to the second end.
- the first end may receive an acknowledgement message sent by the second end indicating successful writing.
- the first end when the first end performs the RDMA write operation, the first end can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
- a method for writing data under RDMA according to an embodiment of the present invention includes:
- the second end receives the RDMA write data message sent by the first end, and the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is A plurality of addresses dispersed within the memory area of the second end.
- the second end may register its own memory area before the second end receives the RDMA write data message sent by the first end.
- the second end writes data corresponding to the specified address at the specified address.
- the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas
- the second end is in the specified address in step 42
- Writing the corresponding data may include: the second end writing data corresponding to the address in each of the specified addresses.
- the data corresponding to each address may be located in different dispersed memory areas.
- the RDMA write data packet may further carry the memory information of the distributed memory area corresponding to each address in the specified address.
- the memory information of the distributed memory area may include: address information and/or length information of the dispersed memory area.
- the first end when the first end performs the RDMA write operation, the first end can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
- the present invention is further illustrated by the specific embodiment by taking the client to read or write the distributed memory address to the server.
- FIG. 5 is a schematic diagram of a method for reading data under RDMA according to an embodiment of the present invention.
- a method for reading data under RDMA according to an embodiment of the present invention includes:
- the client uses the ibv_reg_mr interface to specify the addr parameter, and registers the memory area to the network card; the server registers the memory area to the network card in the same manner.
- the client acquires memory information of a memory area of the server.
- the memory area of the server includes a distributed memory area and a contiguous memory area, and the memory information of the memory area includes: address information and/or length information of the memory area.
- the client may obtain the memory information of the memory area of the server in various different manners, including but not limited to, in the process of establishing an RDMA communication connection between the client and the server, the client The memory information of the memory area in the server is obtained through a private channel.
- the client sends an RDMA read request message to the server, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is a plurality of scattered within the memory area of the server. address.
- the specified address may include multiple addresses distributed in the plurality of distributed memory regions.
- the RDMA read request message may further include memory information of the dispersed memory area corresponding to the specified address.
- the server receives the RDMA read request message sent by the client, reads the data corresponding to the specified address, and sends the data corresponding to the specified address to the client. Specifically, the server may read data corresponding to each address in the specified address, and send data corresponding to each address in the specified address to the client.
- the specified address may include a plurality of addresses distributed in the plurality of discrete memory regions.
- the data that the server reads the specified address may be specifically The server reads data corresponding to each address in the specified address.
- the sending, by the server, the data corresponding to the specified address to the client may be: the server sends the data corresponding to each address in the specified address to the client.
- the protocol header of the RDMA read request message can be as follows:
- each SrcTO corresponds to a scatter memory address.
- SinkSTag the memory space identifier of the data receiving end
- SinkTO the offset of the starting address when the data is stored in the memory space of the receiving end
- RDMARDSZ RDMA read message size
- SrcSTag data source memory space identifier
- SrcTO The offset of the starting address when the data is read from the source memory space.
- any two servers connected through the network can modify the RDMA read protocol header field in the RDMA protocol and solidify the capability to the RDMA-capable network card.
- the client can read the server once by RDMA. Data of the decentralized memory address is read to the local memory space.
- FIG. 6 is a schematic diagram of a method for data writing under RDMA according to an embodiment of the present invention.
- a method for reading data under RDMA according to an embodiment of the present invention includes:
- the client uses the ibv_reg_mr interface to specify the addr parameter, and registers the memory area to the network card; the server registers the memory area to the network card in the same manner.
- the client acquires memory information of a memory area of the server.
- the memory area of the server includes a distributed memory area and a contiguous memory area, and the memory information of the memory area includes: address information and/or length information of the memory area.
- the client may obtain the memory information of the memory area of the server in various different manners, including but not limited to, in the process of establishing an RDMA communication connection between the client and the server, the client The memory information of the dispersed memory area in the server may be obtained through a private channel.
- the client encapsulates the data corresponding to the specified address and the specified address in an RDMA write data packet, and sends the RDMA write data packet to the server.
- the designated address is a plurality of distributed addresses in which a data needs to be written in a range of a memory area of the server.
- the specified address may include multiple addresses distributed in the plurality of distributed memory regions.
- the RDMA write data message may further include memory information of the dispersed memory area corresponding to the specified address.
- the server receives an RDMA write data message sent by the client, and writes data corresponding to the specified address at the specified address.
- the specified address may include multiple addresses distributed in the plurality of distributed memory areas.
- the server may write the corresponding data at the specified address, where the server may write data corresponding to the address in each of the specified addresses.
- the protocol header of the RDMA write data packet can be as follows:
- each Data Sink Tagged offset corresponds to a scatter memory address.
- Data Sink STag data space identifier of the data receiving end
- Data Sink Tagged offset The offset of the starting address when the data is stored in the memory space of the receiving end;
- RDMA write ULP Payload Data payload.
- any two servers connected through the network can modify the RDMA protocol protocol header field in the RDMA protocol and solidify the capability to the RDMA-capable network card.
- the client can write the client once by RDMA write.
- the data of the decentralized memory address is written to the server memory space.
- the method for reading and writing data under RDMA provided by the present invention has the following advantages:
- the conventional RDMA read operation does not support reading the server-side distributed memory address data into the client's continuous or decentralized memory address, and only supports reading the server-side contiguous memory address into the client memory address.
- the present invention supports the read server. Distributes memory addresses and is compatible with traditional read contiguous memory addresses.
- the traditional RDMA write operation does not support writing client memory data to the server.
- the memory address only supports writing to the server contiguous memory address.
- the present invention supports the function of writing the server-side distributed memory address and being compatible with the traditional write server contiguous memory address.
- the described scenario of the present invention supports any platform and operating system such as a virtual machine environment, a physical machine environment, a Linux system, and a Windows system.
- server and “client” in the embodiments of the present invention are relatively speaking, and the distinction between them in the text is only for convenience of description, and is not intended to be limiting, for example, in the present invention.
- the server can be a client
- the client can also be a server.
- an embodiment of the present invention further provides an apparatus for reading data under RDMA.
- the device 70 for data reading under RDMA provided by the embodiment of the present invention includes a sending module 71 and a receiving module 72. among them:
- the sending module 71 is configured to send an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed within a range of the memory area of the second end Multiple addresses;
- the receiving module 72 is configured to receive data corresponding to the specified address sent by the second end.
- the device 70 further includes an acquisition module 73. among them:
- the obtaining module 73 is configured to acquire, after the sending module 71 sends an RDMA read request message to the second end, the memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
- the obtaining module 73 may be specifically configured to:
- the receiving module 72 may be specifically configured to:
- the apparatus further includes: a registration module 74, configured to: forward the memory area to the network card before the sending module sends the RDMA read request message to the second end Sign up.
- a registration module 74 configured to: forward the memory area to the network card before the sending module sends the RDMA read request message to the second end Sign up.
- the apparatus for reading data under RDMA provided by the embodiment of the present invention, the first end is performing RDMA During the read operation, a plurality of distributed addresses in the memory area of the second end that need to be read data can be sent to the second end, and the data of each distributed address is read by the second end to implement data reading under RDMA. operating.
- FIG. 8 is a structural block diagram of another apparatus for reading RDMA data according to an embodiment of the present invention.
- the device 80 for data reading under RDMA provided by the embodiment of the present invention includes a receiving module 81, a processing module 82, and a sending module 83. among them:
- the receiving module 81 is configured to receive an RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is dispersed within a range of the memory area of the second end Multiple addresses;
- the processing module 82 is configured to read data corresponding to the specified address
- the sending module 83 is configured to send data corresponding to the specified address to the first end.
- the processing module 82 is specifically configured to: read data corresponding to each address in the specified address.
- the sending module 83 is specifically configured to: send, to the first end, data corresponding to each address in the specified address.
- the specified address includes multiple addresses distributed in the plurality of distributed memory regions
- the processing module 82 is specifically configured to: read data corresponding to each address in the specified address;
- the sending module 83 is specifically configured to: send, to the first end, data corresponding to each address in the specified address.
- the apparatus 80 further includes:
- the registration module 84 is configured to register the memory area of the user with the network card before the receiving module 81 receives the RDMA read request message sent by the first end.
- the apparatus for reading data under the RDMA provided by the embodiment of the present invention, after the first end acquires the memory information of the dispersed memory area of the second end, and performs an RDMA read operation, the second end of the dispersed memory area can be required.
- the address of the read data is sent to the second end, and the data of the address is read by the second end to implement the data reading operation under the RDMA.
- FIG. 9 is a structural block diagram of another apparatus for writing RDMA data according to an embodiment of the present invention.
- the apparatus 90 for writing data under RDMA includes a processing module 91 and a sending module 92. among them:
- the processing module 91 is configured to encapsulate, in the RDMA write data packet, data corresponding to the specified address that needs to be written into the data, and the specified address is a plurality of distributed in the memory area of the second end address;
- the sending module 92 is configured to send the RDMA write data message to the second end.
- the device 90 further includes:
- the obtaining module 93 is configured to acquire the memory information of the memory area of the second end before the processing module 91 encapsulates the data corresponding to the specified address and the specified address to be written in the RDMA write data packet.
- the memory information of the memory area includes: address information and/or length information of the memory area.
- the obtaining module 93 may be specifically configured to: acquire memory information of each of the dispersed memory areas in the memory area of the second end.
- the specified address in the embodiment of the present invention may include multiple addresses distributed in the plurality of distributed memory regions.
- the device 90 further includes:
- the receiving module 94 is configured to receive, after the sending module sends the RDMA write data message to the second end, an acknowledgement message sent by the second end indicating successful writing.
- the device for writing data under RDMA when the first end performs an RDMA write operation, can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
- FIG. 10 is a structural block diagram of another apparatus for writing RDMA data according to an embodiment of the present invention.
- an apparatus 100 for data writing under RDMA includes a receiving module 101 and a processing module 102. among them:
- the receiving module 101 is configured to receive an RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is a plurality of addresses dispersed within a memory area of the second end;
- the processing module 102 is configured to write data corresponding to the specified address at the specified address.
- the specified address may include multiple addresses distributed in the plurality of distributed memory regions.
- the processing module 102 has The body is configured to: write data corresponding to the address in each of the specified addresses.
- the first end when the first end performs the RDMA write operation, the first end can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
- embodiments of the present invention also provide a network system that can include the RDMA reading device shown in any of Figures 7a-7c and the RDMA reading device shown in Figure 8a or Figure 8b.
- the embodiment of the present invention further provides another network system, which may include the RDMA writing device shown in any of FIGS. 9a to 9c and the RDMA writing device shown in FIG.
- the apparatus for reading data under RDMA and the apparatus for writing data under RDMA provided by the above embodiments are merely exemplified by the division of the above functional modules. In actual applications, the functions may be allocated according to needs. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
- the device for reading and writing data under the RDMA provided by the above embodiment is the same as the method for reading and writing the RDMA data. For the specific implementation process, refer to the method embodiment, and details are not described herein again.
- a person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium.
- the storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Bus Control (AREA)
Abstract
The present invention relates to the field of computers. Provided is a data write/read method in remote direct memory access (RDMA), so that in a case in which a client reads discrete memory addresses from or writes discrete memory addresses into a serving end, RDMA read operations and RDMA write operations are supported. The data read method in RDMA comprises: a first end sending an RDMA read request message to a second end, the RDMA read request message carrying specified addresses of data that needs to be read, and the specified addresses being discrete multiple addresses inside a memory area range of the second end; and the first end receiving data that is sent by the second end and corresponds to the specified addresses. The present invention further provides a corresponding apparatus.
Description
本发明涉及通信领域,特别涉及一种远程直接存储器访问(remote direct memory access,RDMA)下数据读写的方法、装置和网络系统。The present invention relates to the field of communications, and in particular, to a method, apparatus, and network system for reading and writing data under remote direct memory access (RDMA).
在电脑运算中,RDMA是指从一台电脑的存储器到另一台电脑的存储器的直接存储器访问,而不涉及到任一台电脑的操作系统。RDMA允许高通量、低延时的网络连接,这在大规模并行电脑群集中尤为有用。In computer computing, RDMA refers to direct memory access from the memory of one computer to the memory of another computer, without involving the operating system of any computer. RDMA allows for high-throughput, low-latency network connections, which is especially useful in large-scale parallel computer clusters.
目前RDMA技术支持三种数据读写方式:发送/接收(send/receive)、RDMA读取(RDMA read)、RDMA写入(RDMA write)。其中,在客户端向服务端读取或写入分散内存地址的情况下,RDMA读取和RDMA写入操作有可能无法正常进行。Currently RDMA technology supports three data read and write modes: send/receive, RDMA read, and RDMA write. Among them, in the case where the client reads or writes the distributed memory address to the server, the RDMA read and the RDMA write operation may not be performed normally.
发明内容Summary of the invention
本发明实施例提供了一种RDMA写数据读写的方法、装置和网络系统,在客户端向服务端读取或写入分散内存地址的情况下,能够支持RDMA读取操作和RDMA写入操作。The embodiment of the invention provides a method, a device and a network system for reading and writing RDMA write data, which can support RDMA read operation and RDMA write operation when the client reads or writes a distributed memory address to the server. .
第一方面,提供一种远程直接存储器访问RDMA下数据读取的方法,所述方法包括:In a first aspect, a method for remote data access to read data under RDMA is provided, the method comprising:
第一端向第二端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The first end sends an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is a plurality of addresses dispersed in the memory area of the second end ;
所述第一端接收所述第二端发送的、所述指定地址对应的数据。The first end receives data corresponding to the specified address sent by the second end.
结合第一方面,在第一方面的第一种实现方式中,在所述第一端向第二端发送RDMA读取请求消息之前,所述方法还包括:In conjunction with the first aspect, in a first implementation manner of the first aspect, before the sending, by the first end, the RDMA read request message to the second end, the method further includes:
所述第一端获取所述第二端的内存区域的内存信息,其中,所述内存信息包括:地址信息和/或长度信息。
The first end acquires memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
结合第一方面的第一种实现方式,在第一方面的第二种实现方式中,所述第二端的内存区域包括多个分散内存区域,所述第一端获取所述第二端的内存区域的内存信息包括:With reference to the first implementation of the first aspect, in a second implementation manner of the first aspect, the memory area of the second end includes a plurality of distributed memory areas, and the first end acquires a memory area of the second end The memory information includes:
所述第一端获取所述第二端的内存区域中的每个分散内存区域的内存信息。The first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
结合第一方面的第二种实现方式,在第一方面的第三种实现方式中,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the specified address includes multiple addresses that are distributed in the multiple distributed memory regions,
所述第一端接收所述第二端发送的、所述指定地址对应的数据包括:The receiving, by the first end, the data corresponding to the specified address sent by the second end includes:
所述第一端接收所述第二端发送的、所述指定地址中的每个地址对应的数据。The first end receives data corresponding to each address in the specified address sent by the second end.
结合第一方面,在第一方面的第四种实现方式中,在所述第一端向第二端发送RDMA读取请求消息之前,所述方法还包括:In conjunction with the first aspect, in a fourth implementation manner of the first aspect, before the sending, by the first end, the RDMA read request message to the second end, the method further includes:
所述第一端对自身的内存区域向网卡进行注册。The first end registers the network card with its own memory area.
第二方面,提供一种远程直接存储器访问RDMA下数据读取的方法,所述方法包括:In a second aspect, a method for remote data access to read data under RDMA is provided, the method comprising:
第二端接收第一端发送的RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The second end receives the RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is a plurality of distributed areas within the memory area of the second end address;
所述第二端读取所述指定地址对应的数据;The second end reads data corresponding to the specified address;
所述第二端向所述第一端发送所述指定地址对应的数据。The second end sends data corresponding to the specified address to the first end.
结合第二方面,在第二方面的第一种实现方式中,所述第二端读取所述指定地址对应的数据包括:所述第二端读取所述指定地址中每个地址对应的数据;With reference to the second aspect, in a first implementation manner of the second aspect, the reading, by the second end, the data corresponding to the specified address includes: the second end reading, corresponding to each address in the specified address data;
所述第二端向所述第一端发送所述指定地址对应的数据包括:所述第二端向所述第一端发送所述指定地址中每个地址对应的数据。The sending, by the second end, the data corresponding to the specified address to the first end, the second end sends, to the first end, data corresponding to each address in the specified address.
结合第二方面,在第二方面的第二种实现方式中,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则With reference to the second aspect, in a second implementation manner of the second aspect, a plurality of distributed memory regions exist in the second end, where the specified address includes multiple addresses distributed in the plurality of distributed memory regions, then
所述第二端读取所述指定地址对应的数据包括:所述第二端读取所述指定地址中每个地址对应的数据;The reading, by the second end, the data corresponding to the specified address, is: the second end reads data corresponding to each address in the specified address;
所述第二端向所述第一端发送所述指定地址对应的数据包括:所述第二
端向所述第一端发送所述指定地址中每个地址对应的数据。The sending, by the second end, the data corresponding to the specified address to the first end includes: the second
The terminal sends data corresponding to each address in the specified address to the first end.
结合第二方面,在第二方面的第三种实现方式中,在所述第二端接收第一端发送的RDMA读取请求消息之前,所述方法还包括:With reference to the second aspect, in a third implementation manner of the second aspect, before the receiving, by the second end, the RDMA read request message sent by the first end, the method further includes:
所述第二端对自身的内存区域向网卡进行注册。The second end registers the network card with its own memory area.
第三方面,提供一种远程直接存储器访问RDMA下数据写入的方法,所述方法包括:In a third aspect, a method for remote data access to data writes under RDMA is provided, the method comprising:
第一端将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The first end encapsulates the data corresponding to the specified address and the specified address in the RDMA write data packet, where the specified address is a plurality of addresses dispersed within the memory area of the second end;
所述第一端向所述第二端发送所述RDMA写入数据报文。The first end sends the RDMA write data message to the second end.
结合第三方面,在第三方面的第一种实现方式中,在所述第一端将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中之前,所述方法还包括:With reference to the third aspect, in a first implementation manner of the third aspect, before the first end end, the data corresponding to the specified address to be written data and the specified address is encapsulated in the RDMA write data message, The method further includes:
所述第一端获取所述第二端的内存区域的内存信息,其中,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。The first end acquires memory information of the memory area of the second end, where the memory information of the memory area includes: address information and/or length information of the memory area.
结合第三方面的第一种实现方式,在第三方面的第二种实现方式中,所述第二端的内存区域包括多个分散内存区域,所述第一端获取所述第二端的内存区域的内存信息包括:With reference to the first implementation manner of the third aspect, in a second implementation manner of the third aspect, the memory area of the second end includes a plurality of distributed memory areas, and the first end acquires the memory area of the second end The memory information includes:
所述第一端获取所述第二端的内存区域中的每个分散内存区域的内存信息。The first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
结合第三方面的第二种实现方式,在第三方面的第三种实现方式中,所述指定地址包括分布于所述多个分散内存区域中的多个地址。In conjunction with the second implementation of the third aspect, in a third implementation of the third aspect, the specified address includes a plurality of addresses distributed in the plurality of scatter memory regions.
结合第三方面,在第三方面的第四种实现方式中,在所述第一端向所述第二端发送所述RDMA写入数据报文之后,所述方法还包括:With the third aspect, in a fourth implementation manner of the third aspect, after the first end sends the RDMA write data packet to the second end, the method further includes:
所述第一端接收所述第二端发送的指示成功写入的确认消息。The first end receives an acknowledgement message sent by the second end indicating successful writing.
第四方面,提供一种远程直接存储器访问RDMA下数据写入的方法,所述方法包括:In a fourth aspect, a method for remote data access to data writes under RDMA is provided, the method comprising:
第二端接收第一端发送的RDMA写入数据报文,所述RDMA写入数据报文中携带需要写入数据的指定地址和所述指定地址对应的数据,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The second end receives the RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is the Multiple addresses dispersed within the memory area of the second end;
所述第二端在所述指定地址写入所述指定地址对应的数据。
The second end writes data corresponding to the specified address at the specified address.
结合第四方面,在第四方面的第一种实现方式中,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,With reference to the fourth aspect, in a first implementation manner of the fourth aspect, a plurality of distributed memory regions exist in the second end, where the specified address includes multiple addresses distributed in the plurality of distributed memory regions,
所述第二端在所述指定地址写入对应的数据包括:所述第二端在所述指定地址中的每一个地址写入该地址对应的数据。Writing, by the second end, the corresponding data at the specified address includes: the second end writing data corresponding to the address in each of the specified addresses.
第五方面,提供一种RDMA下数据读取的装置,所述装置包括:In a fifth aspect, an apparatus for data reading under RDMA is provided, the apparatus comprising:
发送模块,用于向第二端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a sending module, configured to send an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed in the memory area of the second end Address
接收模块,用于接收所述第二端发送的、所述指定地址对应的数据。The receiving module is configured to receive data corresponding to the specified address sent by the second end.
结合第五方面,在第五方面的第一种实现方式中,所述装置还包括:With reference to the fifth aspect, in a first implementation manner of the fifth aspect, the device further includes:
获取模块,用于在所述发送模块向第二端发送RDMA读取请求消息之前,获取所述第二端的内存区域的内存信息,其中,所述内存信息包括:地址信息和/或长度信息。And an obtaining module, configured to acquire, after the sending module sends an RDMA read request message to the second end, the memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
结合第五方面的第一种实现方式,在第五方面的第二种实现方式中,所述第二端的内存区域包括多个分散内存区域,则所述获取模块具体用于:With reference to the first implementation manner of the fifth aspect, in a second implementation manner of the fifth aspect, the memory area of the second end includes a plurality of scatter memory areas, and the acquiring module is specifically configured to:
获取所述第二端的内存区域中的每个分散内存区域的内存信息。Obtaining memory information of each of the scattered memory areas in the memory area of the second end.
结合第五方面的第二种实现方式,在第五方面的第三种实现方式中,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则With reference to the second implementation manner of the fifth aspect, in a third implementation manner of the fifth aspect, the specified address includes a plurality of addresses distributed in the plurality of distributed memory regions,
所述接收模块具体用于:The receiving module is specifically configured to:
接收所述第二端发送的、所述指定地址中的每个地址对应的数据。Receiving data corresponding to each of the specified addresses sent by the second end.
结合第五方面,在第五方面的第四种实现方式中,所述装置还包括:With reference to the fifth aspect, in a fourth implementation manner of the fifth aspect, the device further includes:
注册模块,用于在所述发送模块向第二端发送RDMA读取请求消息之前,对自身的内存区域向网卡进行注册。The registration module is configured to register the network area of the memory module with the VR card before sending the RDMA read request message to the second end.
第六方面,提供一种RDMA下数据读取的装置,其特征在于,所述装置包括:In a sixth aspect, an apparatus for data reading under RDMA is provided, wherein the apparatus comprises:
接收模块,用于接收第一端发送的RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a receiving module, configured to receive an RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed within a memory area of the second end Multiple addresses;
处理模块,用于读取所述指定地址对应的数据;a processing module, configured to read data corresponding to the specified address;
发送模块,用于向所述第一端发送所述指定地址对应的数据。
And a sending module, configured to send data corresponding to the specified address to the first end.
结合第六方面,在第六方面的第一种实现方式中,所述处理模块具体用于:读取所述指定地址中每个地址对应的数据;With reference to the sixth aspect, in a first implementation manner of the sixth aspect, the processing module is specifically configured to: read data corresponding to each address in the specified address;
所述发送模块具体用于:向所述第一端发送所述指定地址中每个地址对应的数据。The sending module is specifically configured to: send data corresponding to each address in the specified address to the first end.
结合第六方面,在第六方面的第二种实现方式中,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则With reference to the sixth aspect, in a second implementation manner of the sixth aspect, a plurality of distributed memory regions exist in the second end, and the specified address includes multiple addresses distributed in the plurality of distributed memory regions, then
所述处理模块具体用于:读取所述指定地址中每个地址对应的数据;The processing module is specifically configured to: read data corresponding to each address in the specified address;
所述发送模块具体用于:向所述第一端发送所述指定地址中每个地址对应的数据。The sending module is specifically configured to: send data corresponding to each address in the specified address to the first end.
结合第六方面,在第六方面的第三种实现方式中,所述装置还包括:In conjunction with the sixth aspect, in a third implementation manner of the sixth aspect, the device further includes:
注册模块,在所述接收模块接收第一端发送的RDMA读取请求消息之前,用于对自身的内存区域向网卡进行注册。The registration module is configured to register the memory area of the first end with the network card before receiving the RDMA read request message sent by the first end.
第七方面,提供一种RDMA下数据写入的装置,所述装置包括:In a seventh aspect, an apparatus for data writing under RDMA is provided, the apparatus comprising:
处理模块,用于将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a processing module, configured to encapsulate data corresponding to the specified address and the specified address in the RDMA write data packet, where the specified address is a plurality of addresses dispersed within the memory area of the second end ;
发送模块,用于向所述第二端发送所述RDMA写入数据报文。And a sending module, configured to send the RDMA write data message to the second end.
结合第七方面,在第七方面的第一种实现方式中,所述装置还包括:In conjunction with the seventh aspect, in a first implementation manner of the seventh aspect, the device further includes:
获取模块,用于在所述处理模块将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中之前,获取所述第二端的内存区域的内存信息,其中,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。An acquiring module, configured to acquire, after the processing module encapsulates the data corresponding to the data to be written and the data corresponding to the specified address, in the RDMA write data packet, the memory information of the memory area of the second end, where The memory information of the memory area includes: address information and/or length information of the memory area.
结合第七方面的第一种实现方式,在第七方面的第二种实现方式中,所述第二端的内存区域包括多个分散内存区域,所述获取模块具体用于:With reference to the first implementation manner of the seventh aspect, in a second implementation manner of the seventh aspect, the memory area of the second end includes a plurality of scatter memory areas, and the acquiring module is specifically configured to:
获取所述第二端的内存区域中的每个分散内存区域的内存信息。Obtaining memory information of each of the scattered memory areas in the memory area of the second end.
结合第七方面的第二种实现方式,在第七方面的第三种实现方式中,所述指定地址包括分布于所述多个分散内存区域中的多个地址。With reference to the second implementation manner of the seventh aspect, in a third implementation manner of the seventh aspect, the specified address includes a plurality of addresses distributed in the plurality of distributed memory regions.
结合第七方面,在第七方面的第四种实现方式中,所述装置还包括:With reference to the seventh aspect, in a fourth implementation manner of the seventh aspect, the device further includes:
接收模块,用于在所述发送模块向所述第二端发送所述RDMA写入数据报文之后,接收所述第二端发送的指示成功写入的确认消息。
The receiving module is configured to: after the sending module sends the RDMA write data message to the second end, receive an acknowledgement message sent by the second end indicating successful writing.
第八方面,提供一种RDMA下数据写入的装置,所述装置包括:In an eighth aspect, an apparatus for data writing under RDMA is provided, the apparatus comprising:
接收模块,用于接收第一端发送的RDMA写入数据报文,所述RDMA写入数据报文中携带需要写入数据的指定地址和所述指定地址对应的数据,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a receiving module, configured to receive an RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is a plurality of addresses dispersed within a memory area of the second end;
处理模块,用于在所述指定地址写入所述指定地址对应的数据。And a processing module, configured to write data corresponding to the specified address at the specified address.
结合第八方面,在第八方面的第一种实现方式中,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,With reference to the eighth aspect, in a first implementation manner of the eighth aspect, a plurality of distributed memory regions exist in the second end, and the specified address includes multiple addresses distributed in the plurality of distributed memory regions,
所述处理模块具体用于:在所述指定地址中的每一个地址写入该地址对应的数据。The processing module is specifically configured to: write data corresponding to the address in each of the specified addresses.
第九方面,提供一种网络系统,所述网络系统包括上述第五方面中提到的任一种RDMA数据读取的装置和上述第六方面中提到的任一种RDMA数据读取的装置。According to a ninth aspect, a network system is provided, the network system comprising any one of the apparatus for reading RDMA data mentioned in the above fifth aspect, and the apparatus for reading any of the RDMA data mentioned in the sixth aspect above .
第十方面,提供一种网络系统,所述网络系统包括上述第七方面中提到的任一种RDMA数据写入的装置和上述第八方面中提到的任一种RDMA数据写入的装置。According to a tenth aspect, a network system is provided, the network system comprising any one of the apparatus for writing RDMA data mentioned in the seventh aspect, and any apparatus for writing RDMA data mentioned in the above eighth aspect .
采用上述技术方案后,本发明实施例提供的RDMA下数据读写的方法、装置和网络系统,第一端在进行RDMA读写操作时,能够将第二端的内存区域范围内需要读取数据的多个分散地址发送给第二端,由第二端执行对分散地址的数据的读写,实现RDMA下数据读写操作。After the foregoing technical solution, the method, device and network system for reading and writing data under RDMA provided by the embodiment of the present invention, when the first end performs RDMA read and write operations, can read data in the memory area of the second end. A plurality of distributed addresses are sent to the second end, and the second end performs reading and writing of data of the distributed address to implement data read and write operations under RDMA.
为了更清楚地说明本发明实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Other drawings may also be obtained from those of ordinary skill in the art in light of the inventive work.
图1是本发明实施例提供的一种RDMA下数据读取的方法的流程图;1 is a flowchart of a method for reading data under RDMA according to an embodiment of the present invention;
图2是本发明实施例提供的另一种RDMA下数据读取的方法的流程图;2 is a flowchart of another method for data reading under RDMA according to an embodiment of the present invention;
图3是本发明实施例提供的一种RDMA下数据写入的方法的流程图;3 is a flowchart of a method for writing data under RDMA according to an embodiment of the present invention;
图4是本发明实施例提供的另一种RDMA下数据写入的方法的流程图;
4 is a flowchart of another method for data writing under RDMA according to an embodiment of the present invention;
图5是本发明一实施例提供的RDMA下数据读取的方法的示意图;FIG. 5 is a schematic diagram of a method for reading data under RDMA according to an embodiment of the present invention; FIG.
图6是本发明一实施例提供的RDMA下数据写入的方法的示意图;6 is a schematic diagram of a method for writing data under RDMA according to an embodiment of the present invention;
图7a-7c是本发明实施例提供的RDMA下数据读取的装置的结构框图;7a-7c are structural block diagrams of an apparatus for reading data under RDMA according to an embodiment of the present invention;
图8a-8b是本发明实施例提供的RD MA下数据读取的装置的结构框图;8a-8b are structural block diagrams of an apparatus for reading data under an RD MA according to an embodiment of the present invention;
图9a-9c是本发明实施例提供的RDMA下数据写入的装置的结构框图;9a-9c are block diagrams showing the structure of an apparatus for writing data under RDMA according to an embodiment of the present invention;
图10是本发明实施例提供的RDMA下数据写入的装置的结构框图。FIG. 10 is a structural block diagram of an apparatus for writing data under RDMA according to an embodiment of the present invention.
为使本发明的目的、技术方案和优点更加清楚,下面将结合附图对本发明实施方式作进一步地详细描述。The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
图1是本发明实施例提供的一种RDMA下数据读取的方法的流程图,参照图1,本发明实施例提供的RDMA下数据读取的方法包括:1 is a flowchart of a method for reading data under RDMA according to an embodiment of the present invention. Referring to FIG. 1, a method for reading data under RDMA according to an embodiment of the present invention includes:
11、第一端向第二端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址。The first end sends an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is dispersed in the memory area of the second end. Addresses.
需要指出的是,本发明实施例中的第一端与第二端的地位对等,对它们进行区分只是为了便于描述。在本发明实施例中,所述第一端可以为客户端,所述第二端可以为服务端;反之,所述第一端也可以为服务端,所述第二端可以为客户端。It should be noted that the first end and the second end of the embodiment of the present invention are equal to each other, and they are distinguished only for convenience of description. In the embodiment of the present invention, the first end may be a client, and the second end may be a server; otherwise, the first end may also be a server, and the second end may be a client.
可选地,在本步骤11第一端向第二端发送RDMA读取请求消息之前,本发明实施例提供的RDMA下数据读取的方法还可包括:所述第一端获取所述第二端的内存区域的内存信息。其中,所述内存区域的内存信息可包括:内存区域的地址信息和/或长度信息。Optionally, before the sending, by the first end, the RDMA read request message to the second end, the method for reading the data in the RDMA according to the embodiment of the present invention may further include: the first end acquiring the second Memory information of the memory area of the end. The memory information of the memory area may include: address information and/or length information of the memory area.
进一步地,在所述第二端的内存区域包括多个分散内存区域时,所述第一端获取所述第二端的内存区域的内存信息可包括:Further, when the memory area of the second end includes a plurality of distributed memory areas, the acquiring, by the first end, the memory information of the memory area of the second end may include:
所述第一端获取所述第二端的内存区域中的每个分散内存区域的内存信息。The first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
其中,本发明对具体如何获取所述第二端的内存区域的内存信息不作限定,任何通过第一端与第二端的交互获取所述第二端的内存区域的内存信息的方式均可以采用。
The method for obtaining the memory information of the memory area of the second end is not limited, and any manner of obtaining the memory information of the memory area of the second end by using the interaction between the first end and the second end may be adopted.
同时,在本步骤所述第一端向第二端发送RDMA读取请求消息之前,所述第一端还可以对自身的内存区域向网卡进行注册。At the same time, before the first end sends the RDMA read request message to the second end, the first end may also register the network card with its own memory area.
12、所述第一端接收所述第二端发送的、所述指定地址对应的数据。12. The first end receives data corresponding to the specified address sent by the second end.
其中,本步骤所述第一端接收所述第二端发送的、所述指定地址对应的数据可包括:所述第一端接收所述第二端发送的、所述指定地址中的每个地址对应的数据。The receiving, by the first end, the data corresponding to the specified address sent by the second end may include: the first end receiving each of the specified addresses sent by the second end The data corresponding to the address.
当所述第二端中存在多个分散内存区域时,所述指定地址中的各个地址可能分别存在于不同的分散内存区域中,此时,所述指定地址中的每个地址对应的数据来自于各个不同分散内存区域中的数据。When a plurality of distributed memory areas exist in the second end, each of the specified addresses may exist in different dispersed memory areas, and at this time, data corresponding to each address in the specified address comes from Data in different disparate memory regions.
此外,第一端向第二端发送的RDMA读取请求消息除了可携带所述指定地址外,还可以携带所述指定地址中每个地址对应的分散内存区域的内存信息。这样,第二端在获知所述指定地址中每个地址对应的分散内存区域的内存信息后,即可在相应的分散内存区域读取该地址对应的数据。In addition, the RDMA read request message sent by the first end to the second end may carry the memory information of the dispersed memory area corresponding to each address in the specified address, in addition to the specified address. In this way, after the second end knows the memory information of the dispersed memory area corresponding to each address in the specified address, the data corresponding to the address can be read in the corresponding distributed memory area.
本发明实施例提供的RDMA下数据读取的方法,第一端在进行RDMA读取操作时,能够将所述第二端的内存区域范围内需要读取数据的多个分散地址发送给第二端,由第二端读取每个分散地址的数据,实现RDMA下数据读取操作。In the RDMA data reading method provided by the embodiment of the present invention, when the first end performs the RDMA read operation, the first end can send a plurality of distributed addresses in the memory area of the second end that need to read data to the second end. The data of each distributed address is read by the second end to implement the data reading operation under RDMA.
图2是本发明实施例提供的另一种RDMA下数据读取的方法的流程图,参照图2,本发明实施例提供的RDMA下数据写入的方法包括:2 is a flowchart of another method for data reading under RDMA according to an embodiment of the present invention. Referring to FIG. 2, a method for writing data under RDMA according to an embodiment of the present invention includes:
21、第二端接收第一端发送的RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址。The second end receives the RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is dispersed within the memory area of the second end. Multiple addresses.
其中,可选地,所述RDMA读取请求消息中还携带所述指定地址中每个地址对应的分散内存区域的内存信息。所述分散内存区域的内存信息可包括:分散内存区域的地址信息和/或长度信息。Optionally, the RDMA read request message further carries the memory information of the dispersed memory area corresponding to each address in the specified address. The memory information of the dispersed memory area may include address information and/or length information of the dispersed memory area.
同时,在步骤21所述第二端接收第一端发送的RDMA读取请求消息之前,所述第一端还可以对自身的内存区域向网卡进行注册。At the same time, before the second end receives the RDMA read request message sent by the first end, the first end may also register the network card with its own memory area.
22、所述第二端读取所述指定地址对应的数据。22. The second end reads data corresponding to the specified address.
其中,当所述分散的多个地址存在于所述第二端中的同一内存区域时,所述第二端读取所述指定地址对应的数据可包括:所述第二端读取所述指定
地址中每个地址对应的数据。The reading, by the second end, the data corresponding to the specified address, when the plurality of addresses are in the same memory area of the second end, may include: the second end reads the Designation
The data corresponding to each address in the address.
在所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址时,步骤22可具体为:所述第二端读取所述指定地址中每个地址对应的数据,其中各个地址对应的数据可位于不同的分散内存区域中。In the second end, when there are a plurality of distributed memory areas, where the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas, step 22 may be specifically: the second end reads the The data corresponding to each address in the specified address, wherein the data corresponding to each address may be located in a different distributed memory area.
在所述RDMA读取请求消息中还携带所述指定地址中每个地址对应的分散内存区域的内存信息时,第二端可直接在每个地址对应的分散内存区域中读取该地址对应的数据。When the RDMA read request message further carries the memory information of the scatter memory area corresponding to each address in the specified address, the second end may directly read the address corresponding to the address in the scatter memory area corresponding to each address. data.
23、所述第二端向所述第一端发送所述指定地址对应的数据。23. The second end sends data corresponding to the specified address to the first end.
其中,步骤23所述第二端向所述第一端发送所述指定地址对应的数据可包括:所述第二端向所述第一端发送所述指定地址中每个地址对应的数据。The sending, by the second end, the data corresponding to the specified address to the first end by the second end may include: the second end sending, to the first end, data corresponding to each address in the specified address.
在所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址时,本步骤23可具体为:所述第二端向所述第一端发送所述指定地址中每个地址对应的数据。其中,各个地址对应的数据可位于不同的分散内存区域中。In the second end, when there are a plurality of singular memory areas, where the specified address includes a plurality of addresses distributed in the plurality of scattered memory areas, the step 23 may be specifically: the second end is The first end sends data corresponding to each address in the specified address. The data corresponding to each address may be located in different dispersed memory areas.
本发明实施例提供的RDMA下数据读取的方法,第一端在获取第二端的分散内存区域的内存信息后,进行RDMA读取操作时,能够将所述第二端的分散内存区域范围内需要读取数据的地址发送给第二端,由第二端读取该地址的数据,实现RDMA下数据读取操作。The method for reading data in the RDMA provided by the embodiment of the present invention, after the first end acquires the memory information of the dispersed memory area at the second end, the RDMA read operation can be performed in the range of the dispersed memory area of the second end. The address of the read data is sent to the second end, and the data of the address is read by the second end to implement the data reading operation under the RDMA.
图3是本发明实施例提供的一种RDMA下数据写入的方法的流程图,参照图3,本发明实施例提供的RDMA下数据写入的方法包括:FIG. 3 is a flowchart of a method for writing data under RDMA according to an embodiment of the present invention. Referring to FIG. 3, a method for writing data under RDMA according to an embodiment of the present invention includes:
31、第一端将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中,所述指定地址为所述第二端的内存区域范围内分散的多个地址。The first end encapsulates the data corresponding to the data to be written and the data corresponding to the specified address in an RDMA write data packet, where the specified address is a plurality of addresses dispersed in the memory area of the second end. .
其中,在本步骤31所述第一端将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中之前,所述第一端可获取所述第二端的内存区域的内存信息。其中,所述内存区域的内存信息可包括:内存区域的地址信息和/或长度信息。The first end may acquire the data of the second end of the first end before the first end of the data is encapsulated in the RDMA write data packet. Memory information in the memory area. The memory information of the memory area may include: address information and/or length information of the memory area.
进一步地,所述第二端的内存区域可包括多个分散内存区域,此时,所述第一端获取所述第二端的内存区域的内存信息可包括:
Further, the memory area of the second end may include a plurality of distributed memory areas. In this case, the acquiring, by the first end, the memory information of the memory area of the second end may include:
所述第一端获取所述第二端的内存区域中的每个分散内存区域的内存信息。The first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
本发明对具体如何获取所述第二端的内存区域的内存信息不作限定,任何通过第一端与第二端的交互获取所述第二端的内存区域的内存信息的方式均可以采用。The present invention is not limited to how to obtain the memory information of the memory area of the second end, and any manner of obtaining the memory information of the memory area of the second end by using the interaction between the first end and the second end may be adopted.
可选地,在所述第二端中存在多个分散内存区域时,所述指定地址可包括分布于所述多个分散内存区域中的多个地址。Optionally, when there are multiple dispersed memory regions in the second end, the specified address may include multiple addresses distributed in the plurality of distributed memory regions.
在本发明实施例中,所述RDMA写入数据报文中还可携带所述指定地址中每个地址对应的分散内存区域的内存信息。In the embodiment of the present invention, the RDMA write data packet may further carry the memory information of the distributed memory area corresponding to each address in the specified address.
同时,在本步骤31之前,第一端还可对自身的内存区域向网卡进行注册。At the same time, before the step 31, the first end can also register the network card with its own memory area.
32、所述第一端向所述第二端发送所述RDMA写入数据报文。32. The first end sends the RDMA write data packet to the second end.
在本步骤所述第一端向所述第二端发送所述RDMA写入数据报文之后,所述第一端可接收所述第二端发送的指示成功写入的确认消息。After the first end sends the RDMA write data message to the second end, the first end may receive an acknowledgement message sent by the second end indicating successful writing.
本发明实施例提供的RDMA下数据写入的方法,第一端在进行RDMA写入操作时,能够将所述第二端的内存区域范围内需要写入数据的多个分散地址和相应的数据发送给第二端,由第二端在每个地址写入该地址相应的数据,实现RDMA下数据写入操作。In the RDMA data writing method provided by the embodiment of the present invention, when the first end performs the RDMA write operation, the first end can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
图4是本发明实施例提供的一种RDMA下数据写入的方法的流程图,参照图4,本发明实施例提供的RDMA下数据写入的方法包括:4 is a flowchart of a method for writing data under RDMA according to an embodiment of the present invention. Referring to FIG. 4, a method for writing data under RDMA according to an embodiment of the present invention includes:
41、第二端接收第一端建发送的RDMA写入数据报文,所述RDMA写入数据报文中携带需要写入数据的指定地址和所述指定地址对应的数据,所述指定地址为所述第二端的内存区域范围内分散的多个地址。The second end receives the RDMA write data message sent by the first end, and the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is A plurality of addresses dispersed within the memory area of the second end.
其中,在步骤41所述第二端接收第一端建发送的RDMA写入数据报文之前,所述第二端可对自身的内存区域进行注册。The second end may register its own memory area before the second end receives the RDMA write data message sent by the first end.
42、所述第二端在所述指定地址写入所述指定地址对应的数据。42. The second end writes data corresponding to the specified address at the specified address.
其中,在所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址时,步骤42中所述第二端在所述指定地址写入对应的数据可包括:所述第二端在所述指定地址中的每一个地址写入该地址对应的数据。其中,各个地址对应的数据可位于不同的分散内存区域中
Wherein the plurality of distributed memory areas exist in the second end, and the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas, and the second end is in the specified address in step 42 Writing the corresponding data may include: the second end writing data corresponding to the address in each of the specified addresses. The data corresponding to each address may be located in different dispersed memory areas.
在本发明实施例中,所述RDMA写入数据报文中还可携带所述指定地址中每个地址对应的分散内存区域的内存信息。其中,所述分散内存区域的内存信息可包括:分散内存区域的地址信息和/或长度信息。In the embodiment of the present invention, the RDMA write data packet may further carry the memory information of the distributed memory area corresponding to each address in the specified address. The memory information of the distributed memory area may include: address information and/or length information of the dispersed memory area.
本发明实施例提供的RDMA下数据写入的方法,第一端在进行RDMA写入操作时,能够将所述第二端的内存区域范围内需要写入数据的多个分散地址和相应的数据发送给第二端,由第二端在每个地址写入该地址相应的数据,实现RDMA下数据写入操作。In the RDMA data writing method provided by the embodiment of the present invention, when the first end performs the RDMA write operation, the first end can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
为了更好地理解本发明的技术方案,下面以客户端向服务端读取或写入分散内存地址为例,通过具体实施例对本发明进行进一步的阐述。In order to better understand the technical solution of the present invention, the present invention is further illustrated by the specific embodiment by taking the client to read or write the distributed memory address to the server.
图5是本发明一实施例提供的RDMA下数据读取的方法的示意图。参照图5,本发明实施例提供的RDMA下数据读取的方法包括:FIG. 5 is a schematic diagram of a method for reading data under RDMA according to an embodiment of the present invention. Referring to FIG. 5, a method for reading data under RDMA according to an embodiment of the present invention includes:
51、客户端通过ibv_reg_mr接口,指定addr参数,注册内存区域到网卡;服务端采用同样方式注册内存区域到网卡。51. The client uses the ibv_reg_mr interface to specify the addr parameter, and registers the memory area to the network card; the server registers the memory area to the network card in the same manner.
52、所述客户端获取所述服务端的内存区域的内存信息。其中,所述服务端的内存区域包括分散内存区域和连续内存区域,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。本发明实施例中,客户端可通过各种不同的方式来获取服务端的内存区域的内存信息,包括但不限于,在所述客户端与服务端建立RDMA通信连接的过程中,所述客户端通过私有通道获取所述服务端中的内存区域的内存信息。52. The client acquires memory information of a memory area of the server. The memory area of the server includes a distributed memory area and a contiguous memory area, and the memory information of the memory area includes: address information and/or length information of the memory area. In the embodiment of the present invention, the client may obtain the memory information of the memory area of the server in various different manners, including but not limited to, in the process of establishing an RDMA communication connection between the client and the server, the client The memory information of the memory area in the server is obtained through a private channel.
53、客户端向服务端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,其中,所述指定地址为所述服务端的内存区域范围内多个分散的地址。可选地,若服务端包括多个分散的内存区域,则所述指定地址可包括分布于所述多个分散内存区域中的多个地址。所述RDMA读取请求消息还可包括所述指定地址对应的分散内存区域的内存信息。The client sends an RDMA read request message to the server, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is a plurality of scattered within the memory area of the server. address. Optionally, if the server includes a plurality of distributed memory regions, the specified address may include multiple addresses distributed in the plurality of distributed memory regions. The RDMA read request message may further include memory information of the dispersed memory area corresponding to the specified address.
54、服务端接收客户端发送的RDMA读取请求消息,读取所述指定地址对应的数据,并向所述客户端发送所述指定地址对应的数据。具体地,所述服务端可读取所述指定地址中的每个地址对应的数据,并向客户端发送所述指定地址中的每个地址对应的数据。The server receives the RDMA read request message sent by the client, reads the data corresponding to the specified address, and sends the data corresponding to the specified address to the client. Specifically, the server may read data corresponding to each address in the specified address, and send data corresponding to each address in the specified address to the client.
其中,在本发明实施例中,若所述服务端中存在多个分散内存区域,所
述指定地址可包括分布于所述多个分散内存区域中的多个地址。In the embodiment of the present invention, if there are multiple distributed memory areas in the server,
The specified address may include a plurality of addresses distributed in the plurality of discrete memory regions.
若所述服务端中存在多个分散内存区域,且所述指定地址包括分布于所述多个分散内存区域中的多个地址,所述服务端读取所述指定地址对应的数据可具体为:所述服务端读取所述指定地址中每个地址对应的数据。同时,所述服务端向所述客户端发送所述指定地址对应的数据可具体为:所述服务端向所述客户端发送所述指定地址中每个地址对应的数据。If the server has a plurality of distributed memory areas, and the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas, the data that the server reads the specified address may be specifically The server reads data corresponding to each address in the specified address. At the same time, the sending, by the server, the data corresponding to the specified address to the client may be: the server sends the data corresponding to each address in the specified address to the client.
以客户端向服务端读取三个分散内存地址为例,RDMA读取请求消息的协议头可如下:Taking the client to read three scattered memory addresses to the server as an example, the protocol header of the RDMA read request message can be as follows:
即,每个SrcTO均对应于一个分散内存地址。三个分散内存地址即存在三个SrcTO。That is, each SrcTO corresponds to a scatter memory address. There are three SrcTOs for the three scattered memory addresses.
其中,各个字段含义如下:Among them, the meaning of each field is as follows:
SinkSTag:数据接收端内存空间标识;SinkSTag: the memory space identifier of the data receiving end;
SinkTO:数据在接收端内存空间存储时起始地址的偏移量;SinkTO: the offset of the starting address when the data is stored in the memory space of the receiving end;
RDMARDSZ:RDMA读消息大小;RDMARDSZ: RDMA read message size;
SrcSTag:数据源端内存空间标识;SrcSTag: data source memory space identifier;
SrcTO:数据在源端内存空间读取时起始地址的偏移量。
SrcTO: The offset of the starting address when the data is read from the source memory space.
由上可知,通过网络连接的任何两台服务器,通过修改RDMA协议中RDMA读取协议头字段并将该能力固化到具有RDMA能力的网卡,客户端可以通过RDMA读取的方式,一次将服务端连分散内存地址的数据读取到本端内存空间。It can be seen from the above that any two servers connected through the network can modify the RDMA read protocol header field in the RDMA protocol and solidify the capability to the RDMA-capable network card. The client can read the server once by RDMA. Data of the decentralized memory address is read to the local memory space.
图6是本发明一实施例提供的RDMA下数据写入的方法的示意图。参照图6,本发明实施例提供的RDMA下数据读取的方法包括:FIG. 6 is a schematic diagram of a method for data writing under RDMA according to an embodiment of the present invention. Referring to FIG. 6, a method for reading data under RDMA according to an embodiment of the present invention includes:
61、客户端通过ibv_reg_mr接口,指定addr参数,注册内存区域到网卡;服务端采用同样方式注册内存区域到网卡。61. The client uses the ibv_reg_mr interface to specify the addr parameter, and registers the memory area to the network card; the server registers the memory area to the network card in the same manner.
62、所述客户端获取所述服务端的内存区域的内存信息。其中,所述服务端的内存区域包括分散内存区域和连续内存区域,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。本发明实施例中,客户端可通过各种不同的方式来获取服务端的内存区域的内存信息,包括但不限于,在所述客户端与服务端建立RDMA通信连接的过程中,所述客户端可通过私有通道获取所述服务端中的分散内存区域的内存信息。62. The client acquires memory information of a memory area of the server. The memory area of the server includes a distributed memory area and a contiguous memory area, and the memory information of the memory area includes: address information and/or length information of the memory area. In the embodiment of the present invention, the client may obtain the memory information of the memory area of the server in various different manners, including but not limited to, in the process of establishing an RDMA communication connection between the client and the server, the client The memory information of the dispersed memory area in the server may be obtained through a private channel.
63、所述客户端将指定地址、所述指定地址对应的数据封装在RDMA写入数据报文中,并向所述服务端发送所述RDMA写入数据报文。其中,所述指定地址为所述服务端的内存区域范围内需要写入数据的多个分散地址。可选地,若服务端包括多个分散的内存区域,则所述指定地址可包括分布于所述多个分散内存区域中的多个地址。所述RDMA写入数据报文还可包括所述指定地址对应的分散内存区域的内存信息。63. The client encapsulates the data corresponding to the specified address and the specified address in an RDMA write data packet, and sends the RDMA write data packet to the server. The designated address is a plurality of distributed addresses in which a data needs to be written in a range of a memory area of the server. Optionally, if the server includes a plurality of distributed memory regions, the specified address may include multiple addresses distributed in the plurality of distributed memory regions. The RDMA write data message may further include memory information of the dispersed memory area corresponding to the specified address.
64、所述服务端接收所述客户端发送的RDMA写入数据报文,并在所述指定地址写入所述指定地址对应的数据。64. The server receives an RDMA write data message sent by the client, and writes data corresponding to the specified address at the specified address.
其中,在本发明实施例中,若所述服务端中存在多个分散内存区域,所述指定地址可包括分布于所述多个分散内存区域中的多个地址。In the embodiment of the present invention, if there are multiple distributed memory areas in the server, the specified address may include multiple addresses distributed in the plurality of distributed memory areas.
所述服务端在所述指定地址写入对应的数据可具体为:所述服务端在所述指定地址中的每一个地址写入该地址对应的数据。The server may write the corresponding data at the specified address, where the server may write data corresponding to the address in each of the specified addresses.
以客户端向服务端写入三个分散内存地址为例,RDMA写入数据报文的协议头可如下:
For example, if the client writes three decentralized memory addresses to the server, the protocol header of the RDMA write data packet can be as follows:
即,每个Data Sink Tagged offset对应于一个分散内存地址。三个分散内存地址即存在三个Data Sink Tagged offset。That is, each Data Sink Tagged offset corresponds to a scatter memory address. There are three Data Sink Tagged offsets for the three scattered memory addresses.
其中,各个字段含义如下:Among them, the meaning of each field is as follows:
Data Sink STag:数据接收端内存空间标识;Data Sink STag: data space identifier of the data receiving end;
Data Sink Tagged offset:数据在接收端内存空间存储时起始地址的偏移量;Data Sink Tagged offset: The offset of the starting address when the data is stored in the memory space of the receiving end;
RDMA write ULP Payload:数据有效载荷。RDMA write ULP Payload: Data payload.
由上可知,通过网络连接的任何两台服务器,通过修改RDMA协议中RDMA写入协议头字段并将该能力固化到具有RDMA能力的网卡,客户端可以通过RDMA写入的方式,一次将客户端分散内存地址的数据写入服务端内存空间。It can be seen from the above that any two servers connected through the network can modify the RDMA protocol protocol header field in the RDMA protocol and solidify the capability to the RDMA-capable network card. The client can write the client once by RDMA write. The data of the decentralized memory address is written to the server memory space.
由以上实施例描述可以看出,本发明提供的RDMA下数据读写的方法具有以下优点:As can be seen from the description of the above embodiments, the method for reading and writing data under RDMA provided by the present invention has the following advantages:
第一,传统的RDMA读取操作不支持将服务端分散内存地址数据读入客户端连续或分散内存地址,仅支持将服务端连续内存地址读入客户端内存地址,本发明支持读取服务端分散内存地址且兼容传统读取连续内存地址的功能。First, the conventional RDMA read operation does not support reading the server-side distributed memory address data into the client's continuous or decentralized memory address, and only supports reading the server-side contiguous memory address into the client memory address. The present invention supports the read server. Distributes memory addresses and is compatible with traditional read contiguous memory addresses.
第二,传统的RDMA写入操作不支持将客户端内存数据写入服务端分
散内存地址,仅支持写入服务端连续内存地址,本发明支持写入服务端分散内存地址且兼容传统写入服务端连续内存地址的功能。Second, the traditional RDMA write operation does not support writing client memory data to the server.
The memory address only supports writing to the server contiguous memory address. The present invention supports the function of writing the server-side distributed memory address and being compatible with the traditional write server contiguous memory address.
而且,需要说明的是,本发明的所叙述的场景支持虚拟机环境、物理机环境、Linux系统、Windows系统等任意平台和操作系统。Moreover, it should be noted that the described scenario of the present invention supports any platform and operating system such as a virtual machine environment, a physical machine environment, a Linux system, and a Windows system.
需要说明的是,本发明实施例中的“服务端”和“客户端”是相对而言的,在文中对它们进行区分仅是为了描述上方便,并不是意为限制,例如,在本发明实施例中,它们可以互换使用,即,服务端可以为客户端,客户端也可以为服务端。It should be noted that the “server” and “client” in the embodiments of the present invention are relatively speaking, and the distinction between them in the text is only for convenience of description, and is not intended to be limiting, for example, in the present invention. In the embodiment, they can be used interchangeably, that is, the server can be a client, and the client can also be a server.
相应的,本发明实施例还提供一种RDMA下数据读取的装置。参照图7a,本发明实施例提供的RDMA下数据读取的装置70包括发送模块71和接收模块72。其中:Correspondingly, an embodiment of the present invention further provides an apparatus for reading data under RDMA. Referring to FIG. 7a, the device 70 for data reading under RDMA provided by the embodiment of the present invention includes a sending module 71 and a receiving module 72. among them:
发送模块71,用于向第二端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The sending module 71 is configured to send an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed within a range of the memory area of the second end Multiple addresses;
接收模块72,用于接收所述第二端发送的、所述指定地址对应的数据。The receiving module 72 is configured to receive data corresponding to the specified address sent by the second end.
可选地,参照图7b,在一个实施例中,所述装置70还包括获取模块73。其中:Optionally, referring to FIG. 7b, in one embodiment, the device 70 further includes an acquisition module 73. among them:
获取模块73,用于在所述发送模块71向第二端发送RDMA读取请求消息之前,获取所述第二端的内存区域的内存信息,其中,所述内存信息包括:地址信息和/或长度信息。The obtaining module 73 is configured to acquire, after the sending module 71 sends an RDMA read request message to the second end, the memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
进一步地,在所述第二端的内存区域包括多个分散内存区域时,所述获取模块73可具体用于:Further, when the memory area of the second end includes a plurality of distributed memory areas, the obtaining module 73 may be specifically configured to:
获取所述第二端的内存区域中的每个分散内存区域的内存信息。Obtaining memory information of each of the scattered memory areas in the memory area of the second end.
更进一步地,在所述指定地址包括分布于所述多个分散内存区域中的多个地址时,所述接收模块72可具体用于:Further, when the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas, the receiving module 72 may be specifically configured to:
接收所述第二端发送的、所述指定地址中的每个地址对应的数据。Receiving data corresponding to each of the specified addresses sent by the second end.
可选地,参照图7c,在一个实施例中,所述装置还包括:注册模块74,用于在所述发送模块向第二端发送RDMA读取请求消息之前,对自身的内存区域向网卡进行注册。Optionally, referring to FIG. 7c, in an embodiment, the apparatus further includes: a registration module 74, configured to: forward the memory area to the network card before the sending module sends the RDMA read request message to the second end Sign up.
本发明实施例提供的RDMA下数据读取的装置,第一端在进行RDMA
读取操作时,能够将所述第二端的内存区域范围内需要读取数据的多个分散地址发送给第二端,由第二端读取每个分散地址的数据,实现RDMA下数据读取操作。The apparatus for reading data under RDMA provided by the embodiment of the present invention, the first end is performing RDMA
During the read operation, a plurality of distributed addresses in the memory area of the second end that need to be read data can be sent to the second end, and the data of each distributed address is read by the second end to implement data reading under RDMA. operating.
图8a是本发明实施例提供的另一种RDMA数据读取的装置的结构框图。参照图8a,本发明实施例提供的RDMA下数据读取的装置80包括接收模块81、处理模块82和发送模块83。其中:FIG. 8 is a structural block diagram of another apparatus for reading RDMA data according to an embodiment of the present invention. Referring to FIG. 8a, the device 80 for data reading under RDMA provided by the embodiment of the present invention includes a receiving module 81, a processing module 82, and a sending module 83. among them:
接收模块81,用于接收第一端发送的RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The receiving module 81 is configured to receive an RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is dispersed within a range of the memory area of the second end Multiple addresses;
处理模块82,用于读取所述指定地址对应的数据;The processing module 82 is configured to read data corresponding to the specified address;
发送模块83,用于向所述第一端发送所述指定地址对应的数据。The sending module 83 is configured to send data corresponding to the specified address to the first end.
可选地,在一个实施例中,所述处理模块82可具体用于:读取所述指定地址中每个地址对应的数据。所述发送模块83可具体用于:向所述第一端发送所述指定地址中每个地址对应的数据。Optionally, in an embodiment, the processing module 82 is specifically configured to: read data corresponding to each address in the specified address. The sending module 83 is specifically configured to: send, to the first end, data corresponding to each address in the specified address.
可选地,在所述第二端中存在多个分散内存区域时,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则Optionally, when there are multiple dispersed memory regions in the second end, the specified address includes multiple addresses distributed in the plurality of distributed memory regions,
所述处理模块82具体用于:读取所述指定地址中每个地址对应的数据;The processing module 82 is specifically configured to: read data corresponding to each address in the specified address;
所述发送模块83具体用于:向所述第一端发送所述指定地址中每个地址对应的数据。The sending module 83 is specifically configured to: send, to the first end, data corresponding to each address in the specified address.
可选地,参照图8b,在一个实施例中,所述装置80还包括:Optionally, referring to FIG. 8b, in an embodiment, the apparatus 80 further includes:
注册模块84,在所述接收模块81接收第一端发送的RDMA读取请求消息之前,用于对自身的内存区域向网卡进行注册。The registration module 84 is configured to register the memory area of the user with the network card before the receiving module 81 receives the RDMA read request message sent by the first end.
本发明实施例提供的RDMA下数据读取的装置,第一端在获取第二端的分散内存区域的内存信息后,进行RDMA读取操作时,能够将所述第二端的分散内存区域范围内需要读取数据的地址发送给第二端,由第二端读取该地址的数据,实现RDMA下数据读取操作。The apparatus for reading data under the RDMA provided by the embodiment of the present invention, after the first end acquires the memory information of the dispersed memory area of the second end, and performs an RDMA read operation, the second end of the dispersed memory area can be required. The address of the read data is sent to the second end, and the data of the address is read by the second end to implement the data reading operation under the RDMA.
图9a是本发明实施例提供的另一种RDMA数据写入的装置的结构框图。参照图9a,本发明实施例提供的RDMA下数据写入的装置90包括处理模块91和发送模块92。其中:
FIG. 9 is a structural block diagram of another apparatus for writing RDMA data according to an embodiment of the present invention. Referring to FIG. 9a, the apparatus 90 for writing data under RDMA according to an embodiment of the present invention includes a processing module 91 and a sending module 92. among them:
处理模块91,用于将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The processing module 91 is configured to encapsulate, in the RDMA write data packet, data corresponding to the specified address that needs to be written into the data, and the specified address is a plurality of distributed in the memory area of the second end address;
发送模块92,用于向所述第二端发送所述RDMA写入数据报文。The sending module 92 is configured to send the RDMA write data message to the second end.
可选地,参照图9b,在一个实施例中,所述装置90还包括:Optionally, referring to FIG. 9b, in an embodiment, the device 90 further includes:
获取模块93,用于在所述处理模块91将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中之前,获取所述第二端的内存区域的内存信息,其中,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。The obtaining module 93 is configured to acquire the memory information of the memory area of the second end before the processing module 91 encapsulates the data corresponding to the specified address and the specified address to be written in the RDMA write data packet. The memory information of the memory area includes: address information and/or length information of the memory area.
进一步地,在所述第二端的内存区域包括多个分散内存区域时,所述获取模块93可具体用于:获取所述第二端的内存区域中的每个分散内存区域的内存信息。Further, when the memory area of the second end includes a plurality of distributed memory areas, the obtaining module 93 may be specifically configured to: acquire memory information of each of the dispersed memory areas in the memory area of the second end.
可选地,本发明实施例中的所述指定地址可包括分布于所述多个分散内存区域中的多个地址。Optionally, the specified address in the embodiment of the present invention may include multiple addresses distributed in the plurality of distributed memory regions.
可选地,参照图9c,在一个实施例中,所述装置90还包括:Optionally, referring to FIG. 9c, in an embodiment, the device 90 further includes:
接收模块94,用于在所述发送模块向所述第二端发送所述RDMA写入数据报文之后,接收所述第二端发送的指示成功写入的确认消息。The receiving module 94 is configured to receive, after the sending module sends the RDMA write data message to the second end, an acknowledgement message sent by the second end indicating successful writing.
本发明实施例提供的RDMA下数据写入的装置,第一端在进行RDMA写入操作时,能够将所述第二端的内存区域范围内需要写入数据的多个分散地址和相应的数据发送给第二端,由第二端在每个地址写入该地址相应的数据,实现RDMA下数据写入操作。The device for writing data under RDMA according to the embodiment of the present invention, when the first end performs an RDMA write operation, can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
图10是本发明实施例提供的另一种RDMA数据写入的装置的结构框图。参照图10,本发明实施例提供的RDMA下数据写入的装置100包括接收模块101和处理模块102。其中:FIG. 10 is a structural block diagram of another apparatus for writing RDMA data according to an embodiment of the present invention. Referring to FIG. 10, an apparatus 100 for data writing under RDMA according to an embodiment of the present invention includes a receiving module 101 and a processing module 102. among them:
接收模块101,用于接收第一端发送的RDMA写入数据报文,所述RDMA写入数据报文中携带需要写入数据的指定地址和所述指定地址对应的数据,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The receiving module 101 is configured to receive an RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is a plurality of addresses dispersed within a memory area of the second end;
处理模块102,用于在所述指定地址写入所述指定地址对应的数据。The processing module 102 is configured to write data corresponding to the specified address at the specified address.
可选地,在所述第二端中存在多个分散内存区域时,所述指定地址可包括分布于所述多个分散内存区域中的多个地址。此时,所述处理模块102具
体用于:在所述指定地址中的每一个地址写入该地址对应的数据。Optionally, when there are multiple dispersed memory regions in the second end, the specified address may include multiple addresses distributed in the plurality of distributed memory regions. At this time, the processing module 102 has
The body is configured to: write data corresponding to the address in each of the specified addresses.
本发明实施例提供的RDMA下数据写入的方法,第一端在进行RDMA写入操作时,能够将所述第二端的内存区域范围内需要写入数据的多个分散地址和相应的数据发送给第二端,由第二端在每个地址写入该地址相应的数据,实现RDMA下数据写入操作。In the RDMA data writing method provided by the embodiment of the present invention, when the first end performs the RDMA write operation, the first end can send a plurality of distributed addresses and corresponding data to be written in the memory area of the second end. To the second end, the second end writes the corresponding data of the address at each address to implement the data writing operation under the RDMA.
此外,本发明实施例还提供一种网络系统,其可包括图7a-图7c中的任一图所示的RDMA读取装置和图8a或图8b所示的RDMA读取装置。Furthermore, embodiments of the present invention also provide a network system that can include the RDMA reading device shown in any of Figures 7a-7c and the RDMA reading device shown in Figure 8a or Figure 8b.
此外,本发明实施例还提供另一种网络系统,其可包括图9a-图9c中的任一图所示的RDMA写入装置和图10所示的RDMA写入装置。In addition, the embodiment of the present invention further provides another network system, which may include the RDMA writing device shown in any of FIGS. 9a to 9c and the RDMA writing device shown in FIG.
需要说明的是:上述实施例提供的RDMA下数据读取的装置和RDMA下数据写入的装置,仅以上述各功能模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能模块完成,即将装置的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。另外,上述实施例提供的RDMA下数据读写的装置与RDMA数据读写的方法实施例属于同一构思,其具体实现过程详见方法实施例,这里不再赘述。It should be noted that the apparatus for reading data under RDMA and the apparatus for writing data under RDMA provided by the above embodiments are merely exemplified by the division of the above functional modules. In actual applications, the functions may be allocated according to needs. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for reading and writing data under the RDMA provided by the above embodiment is the same as the method for reading and writing the RDMA data. For the specific implementation process, refer to the method embodiment, and details are not described herein again.
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成,也可以通过程序来指令相关的硬件完成,所述的程序可以存储于一种计算机可读存储介质中,上述提到的存储介质可以是只读存储器,磁盘或光盘等。A person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium. The storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.
以上所述仅为本发明的较佳实施例,并不用以限制本发明,凡在本发明的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本发明的保护范围之内。
The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalents, improvements, etc., which are within the spirit and scope of the present invention, should be included in the protection of the present invention. Within the scope.
Claims (34)
- 一种远程直接存储器访问RDMA下数据读取的方法,其特征在于,所述方法包括:A remote direct memory access method for reading data under RDMA, characterized in that the method comprises:第一端向第二端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The first end sends an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is a plurality of addresses dispersed in the memory area of the second end ;所述第一端接收所述第二端发送的、所述指定地址对应的数据。The first end receives data corresponding to the specified address sent by the second end.
- 根据权利要求1所述的方法,其特征在于,在所述第一端向第二端发送RDMA读取请求消息之前,所述方法还包括:The method of claim 1, wherein before the sending, by the first end, the RDMA read request message to the second end, the method further comprises:所述第一端获取所述第二端的内存区域的内存信息,其中,所述内存信息包括:地址信息和/或长度信息。The first end acquires memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
- 根据权利要求2所述的方法,其特征在于,所述第二端的内存区域包括多个分散内存区域,所述第一端获取所述第二端的内存区域的内存信息包括:The method according to claim 2, wherein the memory area of the second end comprises a plurality of discrete memory areas, and the first end obtains the memory information of the memory area of the second end, including:所述第一端获取所述第二端的内存区域中的每个分散内存区域的内存信息。The first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
- 根据权利要求3所述的方法,其特征在于,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则The method according to claim 3, wherein said specified address comprises a plurality of addresses distributed in said plurality of discrete memory regions,所述第一端接收所述第二端发送的、所述指定地址对应的数据包括:The receiving, by the first end, the data corresponding to the specified address sent by the second end includes:所述第一端接收所述第二端发送的、所述指定地址中的每个地址对应的数据。The first end receives data corresponding to each address in the specified address sent by the second end.
- 根据权利要求1所述的方法,其特征在于,在所述第一端向第二端发送RDMA读取请求消息之前,所述方法还包括:The method of claim 1, wherein before the sending, by the first end, the RDMA read request message to the second end, the method further comprises:所述第一端对自身的内存区域向网卡进行注册。The first end registers the network card with its own memory area.
- 一种远程直接存储器访问RDMA下数据读取的方法,其特征在于,所述方法包括:A remote direct memory access method for reading data under RDMA, characterized in that the method comprises:第二端接收第一端发送的RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址; The second end receives the RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, and the specified address is a plurality of distributed areas within the memory area of the second end Address所述第二端读取所述指定地址对应的数据;The second end reads data corresponding to the specified address;所述第二端向所述第一端发送所述指定地址对应的数据。The second end sends data corresponding to the specified address to the first end.
- 根据权利要求6所述的方法,其特征在于,所述第二端读取所述指定地址对应的数据包括:所述第二端读取所述指定地址中每个地址对应的数据;The method according to claim 6, wherein the reading, by the second end, the data corresponding to the specified address comprises: the second end reading data corresponding to each address in the specified address;所述第二端向所述第一端发送所述指定地址对应的数据包括:所述第二端向所述第一端发送所述指定地址中每个地址对应的数据。The sending, by the second end, the data corresponding to the specified address to the first end, the second end sends, to the first end, data corresponding to each address in the specified address.
- 根据权利要求6所述的方法,其特征在于,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则The method according to claim 6, wherein a plurality of dispersed memory areas exist in the second end, and the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas,所述第二端读取所述指定地址对应的数据包括:所述第二端读取所述指定地址中每个地址对应的数据;The reading, by the second end, the data corresponding to the specified address, is: the second end reads data corresponding to each address in the specified address;所述第二端向所述第一端发送所述指定地址对应的数据包括:所述第二端向所述第一端发送所述指定地址中每个地址对应的数据。The sending, by the second end, the data corresponding to the specified address to the first end, the second end sends, to the first end, data corresponding to each address in the specified address.
- 根据权利要求6所述的方法,其特征在于,在所述第二端接收第一端发送的RDMA读取请求消息之前,所述方法还包括:The method according to claim 6, wherein before the second end receives the RDMA read request message sent by the first end, the method further includes:所述第二端对自身的内存区域向网卡进行注册。The second end registers the network card with its own memory area.
- 一种远程直接存储器访问RDMA下数据写入的方法,其特征在于,所述方法包括:A remote direct memory access method for data writing under RDMA, characterized in that the method comprises:第一端将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The first end encapsulates the data corresponding to the specified address and the specified address in the RDMA write data packet, where the specified address is a plurality of addresses dispersed within the memory area of the second end;所述第一端向所述第二端发送所述RDMA写入数据报文。The first end sends the RDMA write data message to the second end.
- 根据权利要求10所述的方法,其特征在于,在所述第一端将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中之前,所述方法还包括:The method according to claim 10, wherein before the first end encapsulates the data corresponding to the specified address to be written and the specified address in the RDMA write data message, the method further include:所述第一端获取所述第二端的内存区域的内存信息,其中,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。The first end acquires memory information of the memory area of the second end, where the memory information of the memory area includes: address information and/or length information of the memory area.
- 根据权利要求11所述的方法,其特征在于,所述第二端的内存区域包括多个分散内存区域,所述第一端获取所述第二端的内存区域的内存信息包括: The method according to claim 11, wherein the memory area of the second end comprises a plurality of distributed memory areas, and the first end obtains the memory information of the memory area of the second end, including:所述第一端获取所述第二端的内存区域中的每个分散内存区域的内存信息。The first end acquires memory information of each of the dispersed memory areas in the memory area of the second end.
- 根据权利要求12所述的方法,其特征在于,所述指定地址包括分布于所述多个分散内存区域中的多个地址。The method of claim 12 wherein said specified address comprises a plurality of addresses distributed in said plurality of discrete memory regions.
- 根据权利要求10所述的方法,其特征在于,在所述第一端向所述第二端发送所述RDMA写入数据报文之后,所述方法还包括:The method according to claim 10, wherein after the first end sends the RDMA write data message to the second end, the method further includes:所述第一端接收所述第二端发送的指示成功写入的确认消息。The first end receives an acknowledgement message sent by the second end indicating successful writing.
- 一种远程直接存储器访问RDMA下数据写入的方法,其特征在于,所述方法包括:A remote direct memory access method for data writing under RDMA, characterized in that the method comprises:第二端接收第一端发送的RDMA写入数据报文,所述RDMA写入数据报文中携带需要写入数据的指定地址和所述指定地址对应的数据,所述指定地址为所述第二端的内存区域范围内分散的多个地址;The second end receives the RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is the Multiple addresses dispersed within the memory area of the second end;所述第二端在所述指定地址写入所述指定地址对应的数据。The second end writes data corresponding to the specified address at the specified address.
- 根据权利要求15所述的方法,其特征在于,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,The method according to claim 15, wherein a plurality of distributed memory regions exist in the second end, and the specified address includes a plurality of addresses distributed in the plurality of distributed memory regions.所述第二端在所述指定地址写入对应的数据包括:所述第二端在所述指定地址中的每一个地址写入该地址对应的数据。Writing, by the second end, the corresponding data at the specified address includes: the second end writing data corresponding to the address in each of the specified addresses.
- 一种远程直接存储器访问RDMA下数据读取的装置,其特征在于,所述装置包括:A remote direct memory access device for reading data under RDMA, characterized in that the device comprises:发送模块,用于向第二端发送RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a sending module, configured to send an RDMA read request message to the second end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed in the memory area of the second end Address接收模块,用于接收所述第二端发送的、所述指定地址对应的数据。The receiving module is configured to receive data corresponding to the specified address sent by the second end.
- 根据权利要求17所述的装置,其特征在于,所述装置还包括:The device according to claim 17, wherein the device further comprises:获取模块,用于在所述发送模块向第二端发送RDMA读取请求消息之前,获取所述第二端的内存区域的内存信息,其中,所述内存信息包括:地址信息和/或长度信息。And an obtaining module, configured to acquire, after the sending module sends an RDMA read request message to the second end, the memory information of the memory area of the second end, where the memory information includes: address information and/or length information.
- 根据权利要求18所述的装置,其特征在于,所述第二端的内存区域包括多个分散内存区域,则所述获取模块具体用于: The device according to claim 18, wherein the memory area of the second end comprises a plurality of discrete memory areas, and the acquiring module is specifically configured to:获取所述第二端的内存区域中的每个分散内存区域的内存信息。Obtaining memory information of each of the scattered memory areas in the memory area of the second end.
- 根据权利要求19所述的装置,其特征在于,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则The apparatus according to claim 19, wherein said specified address comprises a plurality of addresses distributed in said plurality of dispersed memory areas,所述接收模块具体用于:The receiving module is specifically configured to:接收所述第二端发送的、所述指定地址中的每个地址对应的数据。Receiving data corresponding to each of the specified addresses sent by the second end.
- 根据权利要求17所述的装置,其特征在于,所述装置还包括:The device according to claim 17, wherein the device further comprises:注册模块,用于在所述发送模块向第二端发送RDMA读取请求消息之前,对自身的内存区域向网卡进行注册。The registration module is configured to register the network area of the memory module with the VR card before sending the RDMA read request message to the second end.
- 一种远程直接存储器访问RDMA下数据读取的装置,其特征在于,所述装置包括:A remote direct memory access device for reading data under RDMA, characterized in that the device comprises:接收模块,用于接收第一端发送的RDMA读取请求消息,所述RDMA读取请求消息中携带需要读取数据的指定地址,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a receiving module, configured to receive an RDMA read request message sent by the first end, where the RDMA read request message carries a specified address that needs to be read data, where the specified address is dispersed within a memory area of the second end Multiple addresses;处理模块,用于读取所述指定地址对应的数据;a processing module, configured to read data corresponding to the specified address;发送模块,用于向所述第一端发送所述指定地址对应的数据。And a sending module, configured to send data corresponding to the specified address to the first end.
- 根据权利要求22所述的装置,其特征在于,所述处理模块具体用于:读取所述指定地址中每个地址对应的数据;The device according to claim 22, wherein the processing module is specifically configured to: read data corresponding to each address in the specified address;所述发送模块具体用于:向所述第一端发送所述指定地址中每个地址对应的数据。The sending module is specifically configured to: send data corresponding to each address in the specified address to the first end.
- 根据权利要求22所述的装置,其特征在于,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,则The apparatus according to claim 22, wherein a plurality of distributed memory areas exist in the second end, and the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas,所述处理模块具体用于:读取所述指定地址中每个地址对应的数据;The processing module is specifically configured to: read data corresponding to each address in the specified address;所述发送模块具体用于:向所述第一端发送所述指定地址中每个地址对应的数据。The sending module is specifically configured to: send data corresponding to each address in the specified address to the first end.
- 根据权利要求22所述的装置,其特征在于,所述装置还包括:The device of claim 22, wherein the device further comprises:注册模块,在所述接收模块接收第一端发送的RDMA读取请求消息之前,用于对自身的内存区域向网卡进行注册。The registration module is configured to register the memory area of the first end with the network card before receiving the RDMA read request message sent by the first end.
- 一种远程直接存储器访问RDMA下数据写入的装置,其特征在于,所述装置包括:A remote direct memory access device for data writing under RDMA, characterized in that the device comprises:处理模块,用于将需要写入数据的指定地址和所述指定地址对应的数据 封装在RDMA写入数据报文中,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a processing module, configured to: use a specified address that needs to write data and data corresponding to the specified address Encapsulating in an RDMA write data message, the specified address is a plurality of addresses dispersed within a range of the memory area of the second end;发送模块,用于向所述第二端发送所述RDMA写入数据报文。And a sending module, configured to send the RDMA write data message to the second end.
- 根据权利要求26所述的装置,其特征在于,所述装置还包括:The device of claim 26, wherein the device further comprises:获取模块,用于在所述处理模块将需要写入数据的指定地址和所述指定地址对应的数据封装在RDMA写入数据报文中之前,获取所述第二端的内存区域的内存信息,其中,所述内存区域的内存信息包括:内存区域的地址信息和/或长度信息。An acquiring module, configured to acquire, after the processing module encapsulates the data corresponding to the data to be written and the data corresponding to the specified address, in the RDMA write data packet, the memory information of the memory area of the second end, where The memory information of the memory area includes: address information and/or length information of the memory area.
- 根据权利要求27所述的装置,其特征在于,所述第二端的内存区域包括多个分散内存区域,所述获取模块具体用于:The device according to claim 27, wherein the memory area of the second end comprises a plurality of discrete memory areas, and the obtaining module is specifically configured to:获取所述第二端的内存区域中的每个分散内存区域的内存信息。Obtaining memory information of each of the scattered memory areas in the memory area of the second end.
- 根据权利要求28所述的装置,其特征在于,所述指定地址包括分布于所述多个分散内存区域中的多个地址。The apparatus of claim 28, wherein the designated address comprises a plurality of addresses distributed in the plurality of discrete memory regions.
- 根据权利要求26所述的装置,其特征在于,所述装置还包括:The device of claim 26, wherein the device further comprises:接收模块,用于在所述发送模块向所述第二端发送所述RDMA写入数据报文之后,接收所述第二端发送的指示成功写入的确认消息。The receiving module is configured to: after the sending module sends the RDMA write data message to the second end, receive an acknowledgement message sent by the second end indicating successful writing.
- 一种远程直接存储器访问RDMA下数据写入的装置,其特征在于,所述装置包括:A remote direct memory access device for data writing under RDMA, characterized in that the device comprises:接收模块,用于接收第一端发送的RDMA写入数据报文,所述RDMA写入数据报文中携带需要写入数据的指定地址和所述指定地址对应的数据,所述指定地址为所述第二端的内存区域范围内分散的多个地址;a receiving module, configured to receive an RDMA write data message sent by the first end, where the RDMA write data message carries a specified address that needs to be written data and data corresponding to the specified address, where the designated address is a plurality of addresses dispersed within a memory area of the second end;处理模块,用于在所述指定地址写入所述指定地址对应的数据。And a processing module, configured to write data corresponding to the specified address at the specified address.
- 根据权利要求31所述的装置,其特征在于,所述第二端中存在多个分散内存区域,所述指定地址包括分布于所述多个分散内存区域中的多个地址,The apparatus according to claim 31, wherein a plurality of distributed memory areas exist in the second end, and the specified address includes a plurality of addresses distributed in the plurality of distributed memory areas,所述处理模块具体用于:在所述指定地址中的每一个地址写入该地址对应的数据。The processing module is specifically configured to: write data corresponding to the address in each of the specified addresses.
- 一种网络系统,其特征在于,包括权利要求17-21中任一项所述的远程直接存储器访问RDMA下数据读取的装置和权利要求22-25中任一项所述的RDMA下数据读取的装置。A network system, comprising the apparatus for remote data access under RDMA according to any one of claims 17-21, and the data reading under RDMA according to any one of claims 22-25 Take the device.
- 一种网络系统,其特征在于,包括权利要求26-30中任一项所述的 远程直接存储器访问RDMA下数据写入的装置和权利要求31-32中任一项所述的RDMA下数据写入的装置。 A network system, comprising the method of any one of claims 26-30 The apparatus for remote data access to data writing under RDMA and the apparatus for data writing under RDMA according to any one of claims 31-32.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410108028.7A CN103929415B (en) | 2014-03-21 | 2014-03-21 | The method, apparatus and network system of reading and writing data under RDMA |
CN201410108028.7 | 2014-03-21 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015139470A1 true WO2015139470A1 (en) | 2015-09-24 |
Family
ID=51147493
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2014/091263 WO2015139470A1 (en) | 2014-03-21 | 2014-11-17 | Data write/read method and apparatus in rdma and network system |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN103929415B (en) |
WO (1) | WO2015139470A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110177118A (en) * | 2019-06-13 | 2019-08-27 | 上海海事大学 | A kind of RPC communication method based on RDMA |
Families Citing this family (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103929415B (en) * | 2014-03-21 | 2018-03-06 | 华为技术有限公司 | The method, apparatus and network system of reading and writing data under RDMA |
CN105472023B (en) * | 2014-12-31 | 2018-11-20 | 华为技术有限公司 | A kind of method and device of direct distance input and output |
CN105487937A (en) * | 2015-11-27 | 2016-04-13 | 华为技术有限公司 | RDMA (Remote Direct Memory Access) implementation method and device |
CN105630426A (en) * | 2016-01-07 | 2016-06-01 | 清华大学 | Method and system for obtaining remote data based on RDMA (Remote Direct Memory Access) characteristics |
CN114579488A (en) * | 2016-04-04 | 2022-06-03 | 马维尔亚洲私人有限公司 | Method and system for accessing host memory through non-volatile memory on a fabric that is bridged with direct target access |
CN105893323A (en) * | 2016-05-23 | 2016-08-24 | 华为技术有限公司 | Data reading method and data reading equipment |
CN106487896B (en) * | 2016-10-14 | 2019-10-08 | 北京百度网讯科技有限公司 | Method and apparatus for handling remote direct memory access request |
CN106873915A (en) * | 2017-02-22 | 2017-06-20 | 郑州云海信息技术有限公司 | A kind of data transmission method and device based on RDMA registers memory blocks |
CN107547623B (en) * | 2017-06-28 | 2021-02-05 | 新华三云计算技术有限公司 | Application data transmission method and device, application server and application proxy server |
CN107493329A (en) * | 2017-08-15 | 2017-12-19 | 诸葛晴凤 | A kind of remote document access method of distributed memory file system |
CN108228476B (en) * | 2017-12-22 | 2021-02-09 | 新华三技术有限公司 | Data acquisition method and device |
CN110535811B (en) * | 2018-05-25 | 2022-03-04 | 中兴通讯股份有限公司 | Remote memory management method and system, server, client and storage medium |
CN111858418B (en) * | 2019-04-30 | 2023-04-07 | 华为技术有限公司 | Memory communication method and device based on remote direct memory access RDMA |
CN110191194B (en) * | 2019-06-13 | 2020-07-03 | 华中科技大学 | RDMA (remote direct memory Access) network-based distributed file system data transmission method and system |
WO2022061763A1 (en) * | 2020-09-25 | 2022-03-31 | 华为技术有限公司 | Data storage method and apparatus |
CN112954068B (en) * | 2021-03-09 | 2022-09-27 | 西安奥卡云数据科技有限公司 | RDMA (remote direct memory Access) -based data transmission method and device |
EP4318251A1 (en) * | 2021-04-14 | 2024-02-07 | Huawei Technologies Co., Ltd. | Data access system and method, and device and network card |
CN113377704B (en) * | 2021-06-11 | 2022-04-12 | 上海壁仞智能科技有限公司 | Artificial intelligence chip and data operation method |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030050990A1 (en) * | 2001-06-21 | 2003-03-13 | International Business Machines Corporation | PCI migration semantic storage I/O |
CN1818890A (en) * | 2005-01-21 | 2006-08-16 | 国际商业机器公司 | Rnic-based offload of iscsi data movement function by initiator |
CN103440202A (en) * | 2013-08-07 | 2013-12-11 | 华为技术有限公司 | RDMA-based (Remote Direct Memory Access-based) communication method, RDMA-based communication system and communication device |
CN103929415A (en) * | 2014-03-21 | 2014-07-16 | 华为技术有限公司 | Method and device for reading and writing data under RDMA and network system |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050182884A1 (en) * | 2004-01-22 | 2005-08-18 | Hofmann Richard G. | Multiple address two channel bus structure |
CN101459676B (en) * | 2008-12-31 | 2012-01-04 | 中国科学院计算技术研究所 | Message transmission frame and method based on high-speed network oriented to file system |
CN101582058A (en) * | 2009-06-10 | 2009-11-18 | 华为技术有限公司 | Method and device for accessing registers |
CN102571925B (en) * | 2011-12-19 | 2014-08-27 | 华中科技大学 | RDMA (Remote Direct Memory Access)-based data access method in fiber channel environment |
-
2014
- 2014-03-21 CN CN201410108028.7A patent/CN103929415B/en active Active
- 2014-11-17 WO PCT/CN2014/091263 patent/WO2015139470A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030050990A1 (en) * | 2001-06-21 | 2003-03-13 | International Business Machines Corporation | PCI migration semantic storage I/O |
CN1818890A (en) * | 2005-01-21 | 2006-08-16 | 国际商业机器公司 | Rnic-based offload of iscsi data movement function by initiator |
CN103440202A (en) * | 2013-08-07 | 2013-12-11 | 华为技术有限公司 | RDMA-based (Remote Direct Memory Access-based) communication method, RDMA-based communication system and communication device |
CN103929415A (en) * | 2014-03-21 | 2014-07-16 | 华为技术有限公司 | Method and device for reading and writing data under RDMA and network system |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110177118A (en) * | 2019-06-13 | 2019-08-27 | 上海海事大学 | A kind of RPC communication method based on RDMA |
Also Published As
Publication number | Publication date |
---|---|
CN103929415B (en) | 2018-03-06 |
CN103929415A (en) | 2014-07-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2015139470A1 (en) | Data write/read method and apparatus in rdma and network system | |
US11599490B1 (en) | Packet queueing for network device | |
US10282192B1 (en) | Updating device code through a bus | |
US9304896B2 (en) | Remote memory ring buffers in a cluster of data processing nodes | |
EP3125126B1 (en) | Data processing system and data processing method | |
US9742877B2 (en) | Clustering support across geographical boundaries | |
WO2017049945A1 (en) | Accelerator virtualization method and apparatus, and centralized resource manager | |
WO2016101288A1 (en) | Remote direct memory accessmethod, device and system | |
US10817448B1 (en) | Reducing read transactions to peripheral devices | |
CN105335309B (en) | A kind of data transmission method and computer | |
US12026103B1 (en) | Dynamic resource management of network device | |
CN105389120A (en) | Supporting RMA API over active message | |
JP6788691B2 (en) | Improved throughput in OpenFabrics | |
US10621124B2 (en) | Method, device and computer program product for enabling SR-IOV functions in endpoint device | |
US11467998B1 (en) | Low-latency packet processing for network device | |
WO2022001417A1 (en) | Data transmission method, processor system, and memory access system | |
US9098354B2 (en) | Management of application to I/O device communication requests between data processing systems | |
CN109656844A (en) | A kind of AT24xx EEPROM driving method and device | |
CN113422669B (en) | Data transmission method, device and system, electronic equipment and storage medium | |
US20130013755A1 (en) | Diskless pc network communication agent system | |
US8521895B2 (en) | Management of application to application communication requests between data processing systems | |
CN109189705A (en) | A kind of usb expansion method, apparatus, equipment, storage medium and system | |
Nguyen et al. | Reducing data copies between gpus and nics | |
CN113422792B (en) | Data transmission method, device, electronic equipment and computer storage medium | |
US11847316B2 (en) | System and method for managing data storage in network interface controllers |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14886596 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 14886596 Country of ref document: EP Kind code of ref document: A1 |