CN117917884A - Data acquisition method, system and related equipment - Google Patents
Data acquisition method, system and related equipment Download PDFInfo
- Publication number
- CN117917884A CN117917884A CN202211296811.1A CN202211296811A CN117917884A CN 117917884 A CN117917884 A CN 117917884A CN 202211296811 A CN202211296811 A CN 202211296811A CN 117917884 A CN117917884 A CN 117917884A
- Authority
- CN
- China
- Prior art keywords
- instance
- data
- information
- network device
- read request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000004044 response Effects 0.000 claims description 40
- 238000004590 computer program Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 20
- 238000012545 processing Methods 0.000 description 17
- 238000004891 communication Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 11
- 238000013507 mapping Methods 0.000 description 10
- 238000013523 data management Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000002146 bilateral effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The application discloses a data acquisition method, a system and related equipment, wherein the data acquisition method comprises the following steps: the network equipment acquires a first read request of first data sent by a first instance to a second instance, wherein the first read request comprises a first data identifier, and the second instance comprises information of a third instance with read-write permission on the first data; and when the information of the third instance is included in the network equipment, generating a second read request according to the information of the third instance and the first read request, and sending the second read request to the third instance, wherein the second read request comprises the identification of the first data and the information of the third instance and is used for requesting the third instance to read the first data. According to the method, the network equipment can acquire the data owner information of the first data without communicating with the data owner of the first data, and the second reading request is sent to the data owner of the first data, so that the load of the data owner in the data acquisition process can be reduced.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a data acquisition method, system, and related device.
Background
Currently, a data management system is used for managing a certain amount of data, and the system includes a data requester, a data owner, and a data master, where the data master includes metadata of the data, such as data owner information of the data, and the data owner has write permission of the data, and can modify the data. When the data requester acquires the target data, a read request of the target data is required to be sent to a data master, the data master determines data owner information of the data according to metadata of the stored data, the read request is forwarded to the data owner of the target data, and the data owner acquires the target data and sends the target data to the data requester.
In the above manner of acquiring data by the data requester, the data owner determines the data owner information of the data according to the data identifier, so that the load is large, and therefore, how to reduce the load on the data owner side in the process of acquiring the data is a problem to be solved.
Disclosure of Invention
The application provides a data acquisition method, a system and related equipment, which can reduce participation of a data master in the process of data acquisition and reduce load of the data master through direct communication between a data requester and a data owner.
In a first aspect, an embodiment of the present invention provides a method for acquiring data, where the method includes: the network equipment acquires a first read request of first data sent by a first instance to a second instance, wherein the first read request comprises a first data identifier, and the second instance comprises information of a third instance with read-write permission on the first data; the network device determines whether the network device comprises information of a third instance, generates a second read request according to the information of the third instance and the first read request when the network device comprises the information of the third instance, and sends the second read request to the third instance, wherein the second read request comprises identification of the first data and the information of the third instance and is used for requesting the third instance to read the first data.
In the data acquisition method provided by the application, in the case that the network equipment comprises the information of the third instance, the network equipment generates the second reading request according to the first reading request by replacing the content in the first reading request sent by the first instance to the second instance with the information of the third instance. The network device sends the second read request to the data owner of the first data, i.e. to the third instance, thereby obtaining the first data. The process can determine the information of the data owner of the first data without communicating with the data owner of the first data, and send the read request for reading the first data to the data owner of the first data, so that the CPU resource occupation of the server where the data owner is located can be reduced.
In one possible implementation, the network device further receives the first data read by the third instance and sends the first data to the first instance.
In one possible implementation, the second read request is further used to instruct the third instance to return the first data to the first instance after reading the first data.
In one possible implementation, the network device further sends the first read request to the second instance when the information of the third instance is not included in the network device. In the case that the first read request cannot acquire the first data from the second instance, the first instance resends the request for reading the first data to the second instance, so as to ensure the normal acquisition of the data.
In one possible implementation, before the network device sends the first read request to the second instance, when the first read request is not a first packet sent to the second instance, the network device modifies the packet sequence number PSN of the first read request to be contiguous with the PSN of the previous packet sent to the second instance.
The above process can ensure that the PSN of the data packet received by the second instance is continuous, and the first instance can normally acquire data from the second instance, so that the situation that the PSN of the data packet acquired by the second instance is discontinuous and cannot acquire data due to the fact that the network equipment forwards a read request is avoided.
In one possible implementation manner, before the network device obtains the first read request, the network device further receives a claim request sent by the third embodiment to the second instance, where the claim request is used to request to have the read-write permission of the first data; checking whether the network equipment stores information of other examples with read-write authority of the first data, if not, sending a statement request to the second example, and if so, deleting the information of the other examples and then sending the statement request to the second example; and receiving response information of the statement request sent by the second instance, wherein the response information comprises the first data and information of the third instance, the third instance has the read-write authority of the first data, and after the information of the first data and the information of the third instance are recorded, the response information is sent to the third instance. The above-described process is a process in which the third instance declares to be the data owner of the first data, and the network device stores information of the third instance in the case of receiving a response message of the declaration request transmitted by the second instance.
Therefore, the instance can acquire the information of the third instance from the second instance, can acquire the information of the third instance from the network equipment, does not need to communicate with the second instance when acquiring the information of the third instance from the network equipment, and can reduce the CPU resource occupation of the server where the second instance is located.
In a second aspect, an embodiment of the present invention provides a data acquisition system, where the data acquisition system includes a first instance, a second instance, a network device, and a third instance. The first instance is used for sending a first read request of first data to the second instance, the first read request comprises a first data identifier, and the second instance comprises information of a third instance with read-write authority on the first data; the network equipment is used for acquiring a first reading request; and when the network equipment comprises the information of the third instance, generating a second read request according to the information of the third instance and the first read request, and sending the second read request to the third instance, wherein the second read request comprises the identification of the first data and the information of the third instance and is used for requesting the third instance to read the first data.
In one possible implementation, the network device is further configured to receive the first data read by the third instance and send the first data to the first instance. The second read request is also used to instruct the third instance to return the first data to the first instance after the first data is read.
In one possible implementation, the network device is further configured to send the first read request to the second instance when the information of the third instance is not included in the network device.
In one possible implementation, before sending the first read request to the second instance, the network device is further configured to modify the packet sequence number PSN of the first read request to be consecutive to the PSN of the previous packet sent to the second instance when the first read request is not the first packet sent to the second instance.
In a possible implementation manner, before the network device is used to obtain the first read request, the network device is further configured to receive a declaration request sent by the third embodiment to the second instance, where the declaration request is used to request to have the read-write permission of the first data; checking whether the network equipment stores information of other examples with read-write authority of the first data, if not, sending a statement request to the second example, and if so, deleting the information of the other examples and then sending the statement request to the second example; and receiving response information of the statement request sent by the second instance, wherein the response information comprises the first data and information of the third instance, the third instance has the read-write authority of the first data, and after the information of the first data and the information of the third instance are recorded, the response information is sent to the third instance.
In summary, in the data acquisition system provided by the present application, the network device generates the second read request by changing the content of the first read request of the first data sent by the first instance, and sends the second read request to the third instance of the data owner of the first data, so that the first data can be acquired without communicating with the data owner, and the CPU resource occupation in the server where the data owner is located can be reduced.
In a third aspect, an embodiment of the present invention provides a data acquisition apparatus, which may be applied to a network device in a data acquisition system provided in the first aspect, where the data acquisition apparatus includes: the receiving module is used for acquiring a first reading request of first data sent by a first instance to a second instance, wherein the first reading request comprises a first data identifier, and the second instance comprises information of a third instance with read-write permission on the first data; the processing module is used for determining whether the storage module comprises information of a third instance or not, and generating a second read request according to the information of the third instance and the first read request when the storage module comprises the information of the third instance, wherein the second read request comprises an identification of the first data and the information of the third instance and is used for requesting the third instance to read the first data; and a sending module, configured to send the second read request to the third instance.
In one possible implementation manner, the storage module includes information of the third instance, and the receiving module is further configured to receive the first data read by the third instance in a case where the sending module sends the second read request to the third instance; the sending module is also used for sending the first data to the first instance.
In one possible implementation, the second read request is further used to instruct the third instance to return the first data to the first instance after reading the first data.
In a possible implementation, the sending module is further configured to send the first read request to the second instance when the information of the third instance is not included in the storage module.
In one possible implementation, before the sending module sends the first read request to the second instance, when the first read request is not the first packet sent to the second instance, the processing module is configured to modify the packet sequence number PSN of the first read request to be consecutive to the PSN of the previous packet sent to the second instance.
In one possible implementation manner, before the receiving module is configured to obtain a first read request of first data sent by a first instance to a second instance, the receiving module is further configured to receive a claim request sent by a third embodiment to the second instance, where the claim request is used to request to have a read-write permission of the first data; the processing module is also used for checking whether the storage module stores information of other examples with the read-write permission of the first data, if not, the sending module is used for sending the statement request to the second example, and if so, the sending module is used for sending the statement request to the second example after the processing module deletes the information of the other examples; the receiving module is further configured to receive response information of the claim request sent by the second instance, where the response information includes information of the first data and the third instance, which indicates that the third instance has a read-write permission of the first data; the sending module is further configured to send response information to the third instance after the storage module is configured to record the first data and the information of the third instance.
In a fourth aspect, an embodiment of the present invention provides a network device, which is applied to the data acquisition method provided in the first aspect, where the network device includes a processor, a memory, a communication interface, and a bus. The processor, the memory and the communication interface can be connected with each other through an internal bus, and can also realize communication through other means such as wireless transmission. The memory may store computer instructions for executing any one of the possible implementations of the first aspect or the third aspect to implement the functions of the respective modules. And the communication interface is used for receiving the first reading request sent by the first instance, sending the second reading request to the third instance and the like.
In a fifth aspect, an embodiment of the present invention provides a computer readable storage medium, where instructions are stored, where the instructions are executed on the network device, to cause the network device to perform the method according to the first aspect.
Further combinations of the present application may be made to provide further implementations based on the implementations provided in the above aspects.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described.
FIG. 1 is a schematic diagram of a data management system according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for declaring an instance as a data owner provided by an embodiment of the present invention;
FIG. 3 is a flowchart of a data acquisition method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a PSN table according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of PSN table change in a data acquisition process according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of PSN table change in another data acquisition process according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data acquisition device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a network device according to an embodiment of the present invention.
Detailed Description
As shown in fig. 1, the present application provides a schematic architecture of a data management system, where the data management system is used for managing a certain amount of data, and a mode of managing data may be the same as a mode of managing data in a database system, which is not particularly limited. The data management system includes a network device 110, a storage system 120, and a plurality of servers 130. The storage system 120 may be a distributed storage system or a centralized storage system for storing data. Multiple instances are run on multiple servers 130, one or more instances running on each server. Among the multiple instances are a data requestor 140, a data owner 150, a data owner 160, where the data owner is the instance where the metadata of the data is located, which may include the exclusive lock state of the data, the data owner information of the data, and so on. The data owner has exclusive lock of the data, that is, has write permission of the data, and can modify the data. After an instance acquires an exclusive lock of data, a claim request needs to be sent to the data master, the claim request being used to instruct the data master to record the exclusive lock of data, making the instance the data owner of the data. The instance becomes the data owner of the data after receiving the statement success message sent by the data owner. A data requestor is an instance of a request for data that needs to be obtained from a data owner.
The server to which the data requester belongs, the server to which the data owner belongs, and the server to which the data owner belongs are connected through a network device, and it should be understood that one or more data requesters, one or more data owners, and one or more data owners may be included in the database system, which is not particularly limited to this application. In fig. 1, only a server to which a data requester belongs, a server to which a data owner belongs, and a plurality of servers to which other examples respectively belong are exemplified as being connected through a network device.
In the database system, when the data requester needs to read a first data, the data requester forwards the first data read request to the data owner through the network device, the data owner forwards the first data read request to the data owner through the network device after determining the data owner of the first data, and the data owner sends the first data to the data requester after receiving the first data read request. Or after receiving the read request of the first data sent by the data requester through the network device, the data owner sends the data owner identification and the target address of the first data to the data requester through the network device, and the data requester acquires the first data from the data owner according to the data owner identification and the target address of the first data. In the process of acquiring the data by the data requester, the CPU of the server where the data owner is located is required to participate so as to determine the information of the data owner of the data, and the problem that the CPU resource occupation of the server where the data owner is located is too high exists.
The application provides a data acquisition method, which is characterized in that data owner information of data is stored in a network device, a read request of target data is sent to a data master under the condition that a data requester needs to read the target data, the network device sends the read request of the target data to the data owner of the target data according to the data owner information of the target data under the condition that the network device receives the read request of the target data, and the data owner sends the target data to the data requester according to the read request of the target data. The method provided by the application can determine the data owner information of the data under the condition that the data owner does not participate, and can reduce the CPU load of the server where the data owner is located.
Before introducing the data acquisition method provided by the application, the embodiment of the application also provides a method for declaring an instance as a data owner. Taking the example that the third example declares to be the data owner of the first data as an example, the method that the example declares to be the data owner provided by the embodiment of the present application is described. As shown in FIG. 2, FIG. 2 is a flow chart of a method for declaring an instance as a data owner, comprising the steps of.
S210: the third instance sends a claim request to the second instance, applying for the claim to be the data owner of the first data.
The claim request is used for requesting the third instance to have the read-write permission of the first data, and the claim request comprises the first data identifier and the first exclusive lock, wherein the second instance is a data master of the first data and comprises metadata of the first data, such as data owner information of the first data, and the like.
S220: the network device obtains the claim request and sends the claim request to the second instance.
After the network device obtains the statement request, the network device inquires whether information of other examples with the read-write permission of the first data is stored in the network device according to the first data identifier.
In one possible implementation, in case the information of the original data owner of the first data is stored in the network device, the network device deletes the information of the original data owner of the first data and sends the claim request to the second instance.
In another possible embodiment, the network device sends the claim request to the second instance after receiving the claim request in case no information of the original data owner of the first data is present in the network device.
S230: the second instance receives the claim request, generates response information based on the claim request, and sends the response information to the network device.
The second instance receives the claim request and generates response information based on the claim request. In the case where there is an original data owner of the first data in the system, the second instance replaces the exclusive lock corresponding to the original data owner of the first data with the first exclusive lock in the claim request to cause the third instance to switch to the data owner of the first data. The second instance sends an invalidation message including the identity of the first data to the original data owner of the first data to inform the original data owner that is no longer the data owner of the first data. The second instance generates response information after receiving an invalid response message sent by the original data owner. Or in the case that the information of the original data owner of the first data does not exist in the network device, the second instance records the first exclusive lock in the declaration request, and takes the third instance as the data owner of the first data to generate response information. The response information includes the first data and information of the third instance, and is used for indicating that the third instance has the read-write authority of the first data. The information of the third instance may include a second queue pair number (queue pair number, QPN), where the second QPN is used to indicate that the corresponding second Queue Pair (QP) belongs to the third instance. Or the information of the third instance may further comprise second location information, which is allocated by the second instance based on the first data, for indicating the location of the first data in the third instance. The response information may further include more information, which is not particularly limited to the present application.
The second instance sends the response information to the network device after generating the response information.
S240: the network device receives and stores the response information sent by the second instance, and sends the response information to the third instance.
The network device receives the response information sent by the second instance and stores the response information, and the network device may store the correspondence between the first data identifier included in the response information and the information of the third instance, and then send the response message to the third instance.
S250: the third instance receives the response message as a data owner of the first data.
In summary, in the process that the third instance declares to be the data owner of the first data, the network device stores the information of the third instance when receiving the response message of the declaring request sent by the second instance. Therefore, in the case where the instance needs to acquire information of the data owner of the first data, not only information of the third instance but also information of the third instance may be acquired from the second instance. The instance does not need to send a request to the second instance, and the second instance does not need to process the request for acquiring the information of the data owner, so that the CPU resource occupation of the server where the second instance is located can be reduced. In case the network device receives the response information sent by the second instance, the network device stores the response information, in fact a mapping relation of the first data identification and the information of the third instance. In the case that the information of the third instance includes the second QPN, the network device stores a mapping relationship between the first data identifier and the second QPN; in the case where the information of the third instance includes the second QPN and the second location information, the network device stores a mapping relationship of the first data identifier and the second QPN, and a mapping relationship of the first data identifier and the second location information.
It should be understood that the network device stores information of data owners corresponding to each of the plurality of data, that is, stores a mapping relationship between a plurality of sets of data identifiers and the data owner information of the data. The network device may also store a mapping relationship between the first data identifier and the further information of the third instance, which is not particularly limited to the present application.
In the following, a data acquisition method provided by the present application is described, as shown in fig. 3, fig. 3 is a schematic flow chart of the data acquisition method provided by the present application, and the method can be applied to the data management system shown in fig. 1, and the method includes the following steps.
S310: the first instance sends a first read request for first data to the second instance via the network device.
In the case that a first instance in the data management system needs to read first data, the first instance is used as a data requester, and a first reading request of the first data is sent to a second instance, wherein the second instance is a data master of the first data, metadata of the first data such as data owner information of the first data and the like are included, the first reading request includes a first data identifier and a preset value, and the preset value is a constraint value between the first instance and the network device and is used for indicating the network device to inquire whether the information of the data owner of the first data is stored in the network device.
In one possible implementation, where a server of multiple instances in a data management system supports a remote direct data access (remote direct memory access, RDMA) protocol, the multiple instances may communicate over RDMA technology. Where the basic unit of communication is the queue pair QP. Each local QP in one instance may be associated with one QP in another instance, with multiple instances communicating through interrelated QP. Each QP in the instances corresponds to a QPN for indicating the QP, and in the case of determining a QPN, the network device may determine the corresponding QP, thereby determining the instance to which the QP belongs. For example, the network device receives target data sent by the first instance, where the target data includes a second QPN corresponding to a second QP, and after determining, according to the second QPN, that the second QP belongs to the third instance, the network device sends the target data to the third instance.
Thus, in the case where the server of the first instance and the server of the second instance both support the RDMA protocol, the first read request also includes a first QPN, which belongs to the second instance. It should be appreciated that more or less information may be included in the first read request, as embodiments of the application are not specifically limited in this regard.
S320: the network device obtains a first read request, querying whether the network device includes information for a third instance, wherein the third instance is a data owner of the first data. In the case where the network device includes information of the third instance, the network device performs step S330, and in the case where the network device does not include information of the third instance, the network device performs step S340.
The network device obtains a first read request, and queries whether the network device includes information of a data owner of the first data if it is determined that a preset value in the first read request is an appointment value between the first instance and the network device. Since the third instance is a data owner of the first data, it is queried whether information of the third instance is included in the network device.
In the embodiment of the application, the information of the data owners corresponding to the data respectively is stored in the network equipment. There are multiple sets of mapping relations between data identifiers and QPNs in the network device. Under the condition that the network equipment acquires the data identifier in the read request, the network equipment can determine the data owner corresponding to the acquired data identifier according to the acquired data identifier and the mapping relation between the stored data identifier and the QPN. The network device may further store a mapping relationship between a plurality of sets of data identifiers and location information, and the network device may determine, according to the acquired data identifiers and the corresponding relationship between the stored data identifiers and location information, a location of data corresponding to the acquired data identifiers in the data owner of the network device.
The network device searches whether the information of the third instance exists or not in the information of the data owners corresponding to the data stored in the network device according to the first data identification in the first reading request.
In one possible implementation manner, the network device determines a second QPN corresponding to the first data identifier according to the first data identifier and the mapping relationship between the stored data identifier and the QPN, where the second QPN belongs to the third instance. Or the network equipment also determines second position information according to the first data identification and the corresponding relation between the stored data identification and the position information, and determines the position of the first data in the third instance. The network device determines that information including a third instance of the second QPN, or further including second location information, exists in the network device.
In another possible implementation, the network device queries information of a data owner that is not aware of the first data according to the first data identification, and the network device determines that information of the third instance is not present in the network device.
S330: the network device generates a second read request according to the information of the third instance and the first read request, and sends the second read request to the third instance.
In the case that the network device includes information of the third instance, the network device generates a second read request according to the information of the third instance and the first read request, and sends the second read request to the third instance. The second read request includes an identifier of the first data and information of the third instance, and is used for requesting the third instance to read the first data, and also is used for indicating the third instance to return the first data to the first instance after the third instance reads the first data.
The network device generates a second read request according to the first read request and the information of the third instance, wherein the network device generates the second read request includes the following two ways:
Under the condition that both the server and the network device of the first example support a remote direct data access (remote direct memory access, RDMA) unilateral function, after determining the second QPN and the second position information corresponding to the first data identifier, the network device replaces the first QPN in the first reading request with the second QPN, and replaces the preset value with the second position information to obtain the second reading request. The network device sends the second read request to the third instance according to the second QPN in the second read request. The second read request includes the first data identifier, the second QPN, and the second location information, and the information included in the second read request is not specifically limited in the present application.
Under the condition that the server or the network equipment of the first example does not support the RDMA unilateral function, after determining the second QPN corresponding to the first data identifier, the network equipment replaces the first QPN in the first reading request with the second QPN to obtain a second reading request. The network device sends the second read request to the third instance according to the second QPN in the second read request. The second read request includes the first data identifier and the second QPN, and the information included in the second read request is not particularly limited in the present application.
In the above process, the network device replaces the first QPN in the first read request with the second QPN to obtain the second read request, and may send the second read request to the third instance according to the second QPN, so that communication with the second instance (data master) is not required in the process of acquiring data, and occupation of CPU resources of the second instance is reduced. Under the condition of utilizing the RDMA unilateral function, the first instance can directly read the first data according to the second read request, CPU participation of a server where the third instance is located is not needed, network communication can be reduced, and data delay is reduced.
In one possible implementation, the first read request may further include a first field for recording a sequence number of the read request sent by the first instance to the second instance. Illustratively, the value of the first field may be expressed as [ (A, B); m ], wherein a represents a transmitting instance as a first instance, B represents a receiving instance as a second instance, and m represents a packet sequence number (packet sequence number, PSN). Where PSN is an integer greater than or equal to 0, with an initial value of 0, can indicate the number of requests one instance sends to another instance. For example, the first read request includes a first field having a value of [ (A, B); 4, indicating that the sequence number of the first read request sent by the first instance to the second instance is 4, the first read request is the fifth read request sent by the first instance to the second instance, and after the first instance sends the first read request with PSN of 4, PSN is increased by 1, so that the PSN of the next read request sent by the first instance to the second instance is 5.
In the embodiment of the present application, the network device may further include a PSN table, as shown in fig. 4, and fig. 4 is a schematic diagram of the PSN table provided in the embodiment of the present application. The PSN table is used for recording PSN of the data packet sent by the network equipment, and a sending instance and a receiving instance corresponding to the data packet. When the network device sends the data packet, the PSN corresponding to the data packet in the PSN table is automatically increased by 1 according to the sending instance and the receiving instance corresponding to the data packet, and recording is carried out.
Illustratively, in fig. 4, the first, second, and third instances in the first column represent transmit instances, and the first, second, and third instances in the first row represent receive instances. A "2" in the PSN table indicates that the first instance has sent two packets to the second instance. A "3" in the PSN table indicates that the third instance has sent three packets to the first instance.
In a specific embodiment, in a case that the network device replaces the first QPN with the second QPN, since the second QPN belongs to the third instance, the network device determines that the data owner of the first data is the third instance, and determines to send the second read request to the third instance. The network equipment combines the PSN table to modify the first field in the second read request into [ (A, C); s ], wherein a represents that the sending instance is a first instance, C represents that the receiving instance is a third instance, s represents that the sending instance is the first instance in the PSN table, and the receiving instance is the PSN corresponding to the third instance. After the network device sends the second read request to the third instance, s in the PSN table is incremented by 1 and recorded.
After the network device has performed step S330, the third example performs step S350.
S340: the network device sends the first read request to the second instance.
In the case where the network device does not include information of the third instance, the network device transmits the first read request to the second instance, and the first data cannot be acquired from the second instance. And then, the network equipment sends a response message which is sent by the second instance and fails to acquire the first data to the first instance, so that the first instance resends the request for acquiring the first data to the second instance. The second read request may forward the request for acquiring the first data sent by the first instance to the third instance, so that the third instance sends the first data to the first instance, thereby ensuring normal acquisition of the data. After the network device has performed step S340, no further steps are performed.
In one possible implementation, before the network device sends the first read request to the second instance, when the first read request is not a first packet sent to the second instance, the network device modifies the packet sequence number PSN of the first read request to be contiguous with the PSN of the previous packet sent to the second instance. The above process can ensure that PSNs of data packets sent by the first instance and received by the second instance are continuous, and the first instance can acquire data from the second instance.
In a specific embodiment, as shown in fig. 5, fig. 5 is a schematic diagram of PSN table change in a data acquisition process according to an embodiment of the present application. The PSN of the first read request sent by the first instance to the second instance via the network device is 3, and the network device obtains the first read request, determines that the first read request is not the first data packet sent by the first instance to the second instance.
Before the network device sends the first read request to the second instance, the network device modifies the PSN of the first read request to be the first instance according to the first read request and the PSN table, and the receiving instance is the PSN corresponding to the second instance, where the PSN is 1. The PSN of the first read request sent by the network device and received by the second instance is 1, and since the second instance has previously received a packet sent by the first instance, the PSN of the packet is 0, and the PSN of the first read request received by the second instance is continuous with the PSN of the packet sent by the first instance. The second instance may retrieve data in accordance with the first read request.
After the network device sends the first read request to the second instance, the network device takes the sending instance in the PSN table as the first instance, the receiving instance is that the PSN corresponding to the second instance is increased by 1, generates a new PSN value of 2, and records the current PSN value in the PSN table.
S350: the third instance receives the second read request, obtains the first data according to the second read request, and returns the first data to the network device.
The third instance is a data owner of the first data, and after receiving the second read request, the third instance determines the first data stored by the third instance according to the second read request, and returns the first data to the network device.
In the case that the second location information is included in the second read request, the third instance determines the first data based on the second location information and sends the first data to the network device.
In the case that the third instance does not support the RDMA single-sided function or the second location information is not included in the second read request, the third instance finds the first data according to the first data identifier included in the second read request, and sends the first data to the network device.
In one possible implementation, before the third instance obtains the first data according to the second read request, the third instance needs to determine whether the PSN of the received data packet sent by the first instance is continuous. The third instance may include a third field, where the third field is used in the third instance to record a sequence number of the read request sent by the first instance to be received by the third instance, and to determine whether the sequence number of the read request sent by the first instance is consecutive. Illustratively, the value of the third field may be expressed as [ (A, C); i ], wherein a represents that the transmitting instance is the first instance, C represents that the receiving instance is the third instance, i represents that the PSN is obtained by self-increasing 1 in the number of read requests sent by the first instance that the third instance has received. For example, the third field recorded in the third example is [ (a, C); 5, it may be determined from the third field that the third instance has received five read requests sent by the first instance, the sequence number of the fifth read request is 4, the sequence number of the read request sent by the first instance that the third instance next receives should be 5, and if the sequence number of the read request sent by the first instance that the third instance next receives is 5, the PSN is continuous.
In the case where the PSN of the second read request is the same as the PSN in the third field included in the third instance, the PSN is continuous, and the third instance may acquire the first data. In the case where the PSN of the second read request is not the same as the PSN in the third field included in the third instance, the PSN is discontinuous, the third instance cannot acquire the first data, and the first instance resends the read request to acquire the first data.
S360: and the network equipment sends the first data returned by the third instance to the first instance.
The network equipment receives the first data sent by the third instance and sends the first data to the first instance.
In a specific embodiment, as shown in fig. 6, fig. 6 is a schematic diagram of PSN table change in another data acquisition process according to an embodiment of the present application. The first instance sends a first reading request of first data to the second instance through the network equipment, wherein the first reading request comprises a preset value, a first QPN and a first field [ (1, 2); 3]. Wherein the first field indicates that the sending instance of the first read request is the first instance, the receiving instance is the second instance, and the PSN of the first read request is 3, which indicates that the first read request is the fourth packet sent from the first instance to the second instance. After the first instance sends the first read request, the first instance determines that the sequence number of the next packet sent to the second instance is 4.
The network device receives the first read request, determines that a preset value in the first read request is an appointment value between the first instance and the network device, and queries the network device whether information of a third instance is included in the network device, wherein the third instance is a data owner of the first data. And under the condition that the network equipment finds a corresponding second QPN according to the first data identifier, the network equipment replaces the first QPN with the second QPN to generate a second reading request. The network device determines that the sending instance of the second read request is the first instance and the receiving instance is the third instance according to the second QPN. The network equipment combines a sending instance in a PSN table as a first instance, a receiving instance is a PSN corresponding to a third instance, and a first field in a second read request is modified to [ (1, 3); 1]. After the network device sends the second read request to the third instance, the sending instance in the PSN table is taken as the first instance, the receiving instance is that the PSN corresponding to the third instance is increased by 1, a new PSN value is obtained as 2, and recording is performed in the PSN table.
In the third instance, when the second read request is received and the PSN of the second read request is determined to be 1, the PSN in the second read request is determined to be the same as the PSN in the third field, the PSN is determined to be continuous, and the third instance can normally acquire the first data and send the first data to the network device. The data packet where the first data is located further includes a third instance as a sending instance, and the receiving instance is a PSN corresponding to the first instance. The network equipment determines that a data packet sending instance where the first data is located is a third instance, a receiving instance is the first instance, and the network equipment sends the data packet where the first data is located to the first instance. After the network device sends the first data to the first instance, the network device takes the sending instance in the PSN table as a third instance, the receiving instance is PSN self-increment 1 corresponding to the first instance, a new PSN value is 2, and records in the PSN table.
In summary, according to the data acquisition method provided by the application, in the process of acquiring the first data by the first instance, the network device can directly send the read request for reading the first data sent by the first instance to the data owner of the first data by changing the content included in the first read request, so that communication with the data owner of the first data is not needed, network communication in the process of acquiring the data by the data requester can be reduced, data delay is reduced, and CPU resource occupation of the data owner is reduced. The RDAM technology is used for reading data among the first instance, the network device and the third instance, and can reduce data delay.
As shown in fig. 7, fig. 7 is a schematic structural diagram of a data acquisition device according to the present application, where the device is applied to a network apparatus in a data management system, and the data acquisition device 700 includes: a receiving module 710, configured to obtain a first read request of first data sent by a first instance to a second instance, where the first read request includes a first data identifier, and the second instance includes information of a third instance having a read-write right to the first data; a processing module 720, configured to determine whether the storage module 730 includes information of the third instance, and generate a second read request according to the information of the third instance and the first read request when the storage module 730 includes information of the third instance, where the second read request includes an identifier of the first data and the information of the third instance, and is configured to request the third instance to read the first data; a sending module 740, configured to send the second read request to the third instance.
The second read request is also used to instruct the third instance to return the first data to the first instance after the first data is read.
When the storage module does not include the information of the third instance, the sending module is further configured to send the first read request to the second instance, where the data obtaining device cannot obtain the first data from the second instance, and the first instance is required to send a bilateral request for obtaining the first data to the second instance.
In one possible implementation manner, before the receiving module is configured to obtain a first read request of first data sent by a first instance to a second instance, the receiving module is further configured to receive a claim request sent by a third embodiment to the second instance, where the claim request is used to request to have a read-write permission of the first data; the processing module is also used for checking whether the storage module stores information of other examples with the read-write permission of the first data, if not, the sending module is used for sending the statement request to the second example, and if so, the sending module is used for sending the statement request to the second example after the processing module deletes the information of the other examples; the receiving module is further configured to receive response information of the claim request sent by the second instance, where the response information includes information of the first data and the third instance, which indicates that the third instance has a read-write permission of the first data; the sending module is further configured to send response information to the third instance after the storage module is configured to record the first data and the information of the third instance.
In one possible implementation, the storage module includes a PSN table as shown in fig. 4. And the processing module is further used for modifying the data packet sequence number PSN of the third read request in combination with the PSN table so that the PSN of the third read request received by the second instance is continuous with the PSN of the read request sent by the first instance received on the second instance when the fact that the third read request which is sent by the first instance to the second instance is sent to the second instance is determined.
The receiving module 710 is configured to perform steps S220 and S240 in fig. 2, the processing module 720 and the storage module 730 are configured to perform step S240 in fig. 2, the transmitting module 740 is configured to perform steps S220 and S240 in fig. 2, the receiving module 710 and the processing module 720 are further configured to perform steps S320 and S330 in fig. 3, the transmitting module 740 is further configured to perform steps S330 and S340 in fig. 3, wherein the receiving module may be further configured to perform steps performed by the processing module, or the processing module may be further configured to perform steps performed by the receiving module, etc., and the steps specifically performed by the module in the data acquisition device are not particularly limited.
In summary, the data acquisition device, through the cooperation of the receiving module, the processing module, the storage module and the sending module, can enable the data acquisition device to send the first read request corresponding to the first data to the data owner of the first data under the condition that the first instance acquires the first data, without communicating with the data owner of the first data, so that occupation of CPU resources in the server where the data owner is located can be reduced.
As shown in fig. 8, fig. 8 is a schematic structural diagram of a network device according to an embodiment of the present invention, where the network device 800 includes: processor 810, memory 820, communication interface 830, and bus 840. The processor, the memory and the communication interface can be connected with each other through an internal bus, and can also realize communication through other means such as wireless transmission.
The processor 810 may be comprised of at least one general purpose processor, such as a central processing unit (central processing unit, CPU), or a combination of CPU and hardware chips. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (programmable logic device, PLD), or a combination thereof. The PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (FPGA) GATE ARRAY, general-purpose array logic (GENERIC ARRAY logic, GAL), or any combination thereof. The processor 810 is configured to execute various types of digitally stored instructions. The processor can execute the steps executed by the acquisition module, the processing module and the sending module to realize corresponding functions.
The memory 820 may be volatile memory (volatile memory), such as random access memory (random access memory, RAM), dynamic random access memory (DYNAMIC RAM, DRAM), static random access memory (STATIC RAM, SRAM), synchronous dynamic random access memory (synchronous DYNAMIC RAM, SDRAM), double-rate synchronous dynamic random access memory (double DATA RATE RAM, DDR), cache memory (cache), and the like, and may also include combinations of the above. Memory 820 includes program code that, when executed, causes processor 810 to perform the steps performed by the network devices of fig. 2 and 3, implementing the functions of fig. 2 and 3. The memory 820 further includes data, where the data may include a plurality of sets of correspondence between data identifiers and QPNs, a plurality of sets of correspondence between data identifiers and location information, a correspondence between a received read request and a transmitted read request, and the like, and the data of the memory may further include more content, which is not specifically limited to this embodiment of the present application.
The communication interface 830 may be configured to receive a first read request sent by the first instance, or send a second read request generated by a processor or processing module to the third instance.
It should be noted that fig. 8 is only one possible implementation of the embodiment of the present application, and in practical applications, the network device may further include more or fewer components, which is not limited herein.
Embodiments of the present application also provide a computer-readable storage medium having instructions stored therein that, when executed on a processor, implement the method flows shown in fig. 2 and 3.
The embodiment of the application also provides a computer program product, which when run on a processor, implements the method flows shown in fig. 2 and 3.
When the computer program instructions are loaded or executed on a computer, the processes or functions in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus.
The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). Computer readable storage media can be any available media that can be accessed by a computer or data storage nodes, such as servers, data centers, etc. that contain at least one set of available media. Usable media may be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., compact digital video disc (digital video disc, DVD), or semiconductor media).
The present invention is not limited to the above embodiments, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the present invention, and these modifications and substitutions are intended to be included in the scope of the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (14)
1. A method of data acquisition, comprising:
The method comprises the steps that network equipment obtains a first read request of first data sent by a first instance to a second instance, wherein the first read request comprises a first data identifier, and the second instance comprises information of a third instance with read-write permission on the first data;
The network device determines whether the network device includes information of the third instance, generates a second read request according to the information of the third instance and the first read request when the network device includes the information of the third instance, and sends the second read request to the third instance, wherein the second read request includes an identifier of the first data and the information of the third instance and is used for requesting the third instance to read the first data.
2. The method according to claim 1, wherein the method further comprises:
And receiving the first data read by the third instance and sending the first data to the first instance.
3. The method of claim 1, wherein the second read request is further for instructing the third instance to return the first data to the first instance after reading the first data.
4. A method according to any one of claims 1 to 3, characterized in that the method further comprises:
the first read request is sent to the second instance when the information of the third instance is not included in the network device.
5. A method according to any of claims 1-3, characterized in that before sending the first read request to the second instance, the method comprises:
When the first read request is not a first packet sent to the second instance, the network device modifies a packet sequence number, PSN, of the first read request to be contiguous with a PSN of a previous packet sent to the second instance.
6. The method of claim 1, wherein prior to the network device obtaining the first read request for the first data sent by the first instance to the second instance, the method further comprises:
The network device receives a claim request sent to the second instance by the third embodiment, wherein the claim request is used for requesting to have the read-write permission of the first data;
The network equipment checks whether the network equipment stores information of other examples with the read-write authority of the first data, if not, the declaration request is sent to the second example, and if so, the declaration request is sent to the second example after deleting the information of the other examples;
The network device receives response information of the claim request sent by the second instance, wherein the response information comprises information of the first data and the third instance, the information indicates that the third instance has the read-write authority of the first data, and the response information is sent to the third instance after the information of the first data and the third instance is recorded.
7. A data acquisition system, the system comprising a first instance, a second instance, a network device, and a third instance:
The first instance is configured to send a first read request of first data to the second instance, where the first read request includes a first data identifier, and the second instance includes information of a third instance having a read-write right to the first data;
The network device is configured to obtain the first read request;
The network device is configured to determine whether the network device includes information of the third instance, and when the network device includes information of the third instance, generate a second read request according to the information of the third instance and the first read request, and send the second read request to the third instance, where the second read request includes an identifier of the first data and the information of the third instance, and is configured to request the third instance to read the first data.
8. The system of claim 7, wherein the network device is further configured to receive the first data read by the third instance and send the first data to the first instance.
9. The system of claim 7, wherein the second read request is further to instruct the third instance to return the first data to the first instance after reading the first data.
10. The system of any of claims 7-9, wherein the network device is further configured to send the first read request to the second instance when information of the third instance is not included in the network device.
11. The system of any of claims 7-9, wherein the network device, prior to sending the first read request to the second instance, is further configured to modify, by the network device, a packet sequence number, PSN, of the first read request to be contiguous with a PSN of a previous packet sent to the second instance when the first read request is not a first packet sent to the second instance.
12. The system of claim 7, wherein prior to the network device being used to obtain the first read request, the network device is further configured to:
Receiving a claim request sent to the second instance by the third embodiment, wherein the claim request is used for requesting to have the read-write permission of the first data;
Checking whether the network equipment stores information of other examples with the read-write permission of the first data, if not, sending the statement request to the second example, and if so, deleting the information of the other examples and then sending the statement request to the second example;
Receiving response information of the claim request sent by the second instance, wherein the response information comprises information of the first data and the third instance, the information indicates that the third instance has the read-write authority of the first data, and the response information is sent to the third instance after the information of the first data and the third instance is recorded.
13. A network device comprising a processor and a memory, the memory storing a computer program, the processor executing the computer program to cause the network device to perform the method of any of claims 1 to 6.
14. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a program which, when run on a network device, causes the network device to perform the method according to any of claims 1 to 6.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211296811.1A CN117917884A (en) | 2022-10-21 | 2022-10-21 | Data acquisition method, system and related equipment |
| PCT/CN2023/114830 WO2024082815A1 (en) | 2022-10-21 | 2023-08-25 | Data acquisition method and system, and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211296811.1A CN117917884A (en) | 2022-10-21 | 2022-10-21 | Data acquisition method, system and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117917884A true CN117917884A (en) | 2024-04-23 |
Family
ID=90729655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211296811.1A Pending CN117917884A (en) | 2022-10-21 | 2022-10-21 | Data acquisition method, system and related equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN117917884A (en) |
| WO (1) | WO2024082815A1 (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741731A (en) * | 2009-12-03 | 2010-06-16 | 中兴通讯股份有限公司 | Content metadata storing, inquiring method and managing system in content delivery network (CDN) |
| CN102761572B (en) * | 2011-04-28 | 2016-07-27 | 腾讯科技（深圳）有限公司 | The method and system of dynamic adjusting data flow |
| CN103166978A (en) * | 2011-12-08 | 2013-06-19 | 中兴通讯股份有限公司 | Method and device of data obtaining |
| CN107306279B (en) * | 2016-04-18 | 2020-08-04 | 阿里巴巴集团控股有限公司 | Data processing method, content distribution network node and resident gateway |
| CN107872486B (en) * | 2016-09-28 | 2020-06-02 | 华为技术有限公司 | Communication method and device |
| CN111835396B (en) * | 2019-04-18 | 2022-03-29 | 华为技术有限公司 | Method and device for processing data packet |
| CN110430286A (en) * | 2019-08-30 | 2019-11-08 | 杭州领智云画科技有限公司 | Method, apparatus, equipment and the storage medium of resource are obtained in CDN system |
| CN117177266A (en) * | 2020-04-28 | 2023-12-05 | 华为技术有限公司 | Address acquisition method and device |
-
2022
- 2022-10-21 CN CN202211296811.1A patent/CN117917884A/en active Pending
-
2023
- 2023-08-25 WO PCT/CN2023/114830 patent/WO2024082815A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024082815A1 (en) | 2024-04-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831612B2 (en) | Primary node-standby node data transmission method, control node, and database system | |
| WO2022016861A1 (en) | Hotspot data caching method and system, and related device | |
| CN105426321A (en) | RDMA friendly caching method using remote position information | |
| CN109697127B (en) | Method and device for locking access operation of shared resource | |
| US12001450B2 (en) | Distributed table storage processing method, device and system | |
| US20060112083A1 (en) | Object relation information management program, method, and apparatus | |
| EP2629206B1 (en) | Directory replacement method and device | |
| US8510513B2 (en) | Network load reducing method and node structure for multiprocessor system with distributed memory | |
| CN111190861B (en) | Hot spot file management method, server and computer readable storage medium | |
| CN115114042A (en) | Storage data access method and device, electronic equipment and storage medium | |
| CN117917884A (en) | Data acquisition method, system and related equipment | |
| CN113301173A (en) | Domain name updating system and method, message forwarding method and server | |
| CN111400327B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN118260304B (en) | Request processing method, device, equipment and medium based on cache consistency directory | |
| CN115426375A (en) | Data processing method and data processing system | |
| CN113590507A (en) | Distributed storage system, cache layer thereof, data access method and data access equipment | |
| CN114238518B (en) | Data processing method, device, equipment and storage medium | |
| WO2021063242A1 (en) | Metadata transmission method of storage system, and storage system | |
| CN118519998B (en) | Distributed database local temporary table management method, device, equipment and medium | |
| CN116991333B (en) | Distributed data storage method, device, electronic equipment and storage medium | |
| CN115328947A (en) | Data request method and related equipment | |
| CN112416871B (en) | Data access method, device and system | |
| US10579526B2 (en) | Responding to snoop requests | |
| CN116521354A (en) | Caching method, caching server, storage medium and electronic equipment | |
| CN114238518A (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |