CN114844910B - Data transmission method, system, equipment and medium of distributed storage system - Google Patents
Data transmission method, system, equipment and medium of distributed storage system Download PDFInfo
- Publication number
- CN114844910B CN114844910B CN202210411972.4A CN202210411972A CN114844910B CN 114844910 B CN114844910 B CN 114844910B CN 202210411972 A CN202210411972 A CN 202210411972A CN 114844910 B CN114844910 B CN 114844910B
- Authority
- CN
- China
- Prior art keywords
- data
- transmission
- connection
- message
- data transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 266
- 238000000034 method Methods 0.000 title claims abstract description 99
- 238000003860 storage Methods 0.000 title claims abstract description 54
- 230000004044 response Effects 0.000 claims abstract description 114
- 238000012545 processing Methods 0.000 claims abstract description 11
- 230000006870 function Effects 0.000 claims description 78
- 230000011218 segmentation Effects 0.000 claims description 39
- 238000012544 monitoring process Methods 0.000 claims description 25
- 238000001514 detection method Methods 0.000 claims description 22
- 238000013507 mapping Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 10
- 230000002776 aggregation Effects 0.000 claims description 6
- 238000004220 aggregation Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 6
- 238000009826 distribution Methods 0.000 abstract description 5
- 238000005204 segregation Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 19
- 239000000523 sample Substances 0.000 description 16
- 238000012546 transfer Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000000354 decomposition reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/12—Network monitoring probes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
The data transmission method, the system, the equipment and the medium of the distributed storage system provided by the invention are characterized in that the data to be transmitted is divided into a plurality of divided data, each divided data is respectively transmitted to the target object corresponding to each divided data through the transmission queue, the response message of each target object is obtained, the transmission result of the data to be transmitted is generated, the data to be transmitted can be divided, the segregation of the messages is realized, the request distribution and the result merging of a plurality of nodes of the cluster are completed in the local thread of the client, and the data transmission processing efficiency in the distributed storage system is greatly improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data transmission method, system, device, and medium for a distributed storage system.
Background
The distributed storage system comprises a storage cluster formed by a plurality of nodes, and a large amount of data transmission exists in the storage cluster in the operation process, so that the function and the performance of the data transmission are important to the distributed storage system.
The same task in the data transmission mode in the related art needs more processes to be realized, and the task needs to be circularly requested or responded when the task is completed, or needs to be realized by relying on the technologies such as multithreading, which is time-consuming and increases the consumption of system resources.
Disclosure of Invention
In view of the above-mentioned drawbacks of the prior art, the present invention provides a data transmission method, system device and medium for a distributed storage system, so as to solve the above-mentioned technical problems.
The invention provides a data transmission method of a distributed storage system, which comprises the following steps:
dividing data to be transmitted into a plurality of divided data, and respectively transmitting each divided data to a target object corresponding to each divided data through a transmission queue;
and obtaining response information of each target object, and generating a transmission result of the data to be transmitted.
Optionally, the sending the split data to the target object corresponding to each split data through the sending queue includes:
generating at least one piece of redundant data of the data to be transmitted, and newly adding the redundant data into the segmentation data;
obtaining a target object identifier corresponding to the segmentation data, and generating a message to be transmitted according to the segmentation data, the target object identifier and the transmission identifier to obtain a message to be transmitted corresponding to each segmentation data;
And respectively sending each message to be sent to a target object corresponding to each piece of divided data through a sending queue.
Optionally, the response message includes a response identifier, the response identifier corresponds to the sending identifier one by one, and the obtaining the response message of each target object and generating the transmission result of the data to be transmitted include:
acquiring a response identifier in each response message;
determining a transmission identifier corresponding to the response identifier according to a preset response identifier-transmission identifier mapping relation, and determining the transmission state of each piece of divided data;
and determining the transmission result according to the transmission state of each piece of the divided data.
Optionally, determining the transmission result according to the transmission state of each of the split data includes:
if the response identification corresponding to the split data does not exist, the transmission state of the split data is transmission failure;
and obtaining the failure number of the target segmentation data, and determining the transmission result, wherein the target segmentation data comprises segmentation data with transmission state of transmission failure.
Optionally, generating the transmission result of the data to be transmitted includes at least one of:
monitoring response messages of all the target objects, acquiring monitoring time length, stopping monitoring if the monitoring time length exceeds a preset time length threshold value, and generating a transmission result of the data to be transmitted according to the currently acquired response messages of all the target objects;
Monitoring response messages of the target objects, acquiring monitoring time length, and discarding the response messages acquired after the monitoring time length exceeds a preset time length threshold.
Optionally, the message to be sent further includes a calling function identifier, and the method further includes:
after the target object obtains the message to be sent, the message to be sent is processed through a calling function corresponding to the calling function identifier, and a feedback message is generated;
and acquiring the feedback information fed back by the target object.
Optionally, before each piece of the split data is sent to the target object corresponding to each piece of the split data through the sending queue, the method further includes:
and setting a custom data segment for the target object, wherein the custom data segment comprises a plurality of calling functions and calling function identifiers of the calling functions, and the calling functions are used for processing the segmentation data.
Optionally, the method further comprises:
acquiring an update message, wherein the update message comprises at least one of a new function and a delete function;
and updating the custom data segment according to the update message.
Optionally, before the sending end sends each piece of the split data to the target object corresponding to each piece of the split data through the sending queue, the method further includes:
Determining the sending end and each target object as data transmission nodes, and sequentially establishing connection between the data transmission nodes;
assigning a network handle to each connection, identifying the connection by the network handle;
acquiring address information of each data transmission node, and establishing a mapping relation between each address information and a network handle;
each data transmission node stores address information, network handles and mapping relations of the connection nodes, wherein the connection nodes are other data transmission nodes connected with the data transmission nodes.
Optionally, the sending end sends each piece of the split data to the target object corresponding to each piece of the split data through a sending queue, including:
acquiring a transmission identifier corresponding to the segmentation data, wherein the transmission identifier comprises the address information;
determining a network handle corresponding to the split data according to the address information and the mapping relation to obtain the network handle corresponding to each split data;
the transmitting terminal transmits the divided data to the target object corresponding to the divided data through a transmitting queue based on the network handles.
Optionally, the method further comprises:
Respectively acquiring data flow of each data transmission node under each connection;
if zero-flow connection exists, respectively controlling one data transmission node of each zero-flow connection to send a detection message to the other data transmission node of the zero-flow connection at a first preset time interval, wherein the detection message is used for triggering the other data transmission node to generate a response detection message and feeding back the response detection message through the zero-flow connection, and the data flow of at least one data transmission node in the zero-flow connection is zero;
and if the data flow of the other data transmission node of the zero-flow connection at intervals of a second preset time is zero, determining the zero-flow connection as invalid connection, wherein the second preset time is less than the first preset time.
The invention also provides a data transmission system of the distributed storage system, which comprises:
the segmentation module is used for segmenting the data to be transmitted into a plurality of segmentation data and respectively transmitting the segmentation data to target objects corresponding to the segmentation data through a transmission queue;
and the aggregation module is used for acquiring the response message of each target object and generating a transmission result of the data to be transmitted.
The invention also provides an electronic device, which comprises a processor, a memory and a communication bus;
the communication bus is used for connecting the processor and the memory;
the processor is configured to execute a computer program stored in the memory to implement the method according to any one of the embodiments described above.
The present invention also provides a computer-readable storage medium, having stored thereon a computer program,
the computer program is configured to cause a computer to perform the method according to any one of the embodiments described above.
The invention has the beneficial effects that: the data transmission method, the system, the equipment and the medium of the distributed storage system provided by the invention are characterized in that the data to be transmitted is divided into a plurality of divided data, each divided data is respectively transmitted to the target object corresponding to each divided data through the transmission queue, the response message of each target object is obtained, the transmission result of the data to be transmitted is generated, the data to be transmitted can be divided, the segregation of the messages is realized, the request distribution and the result merging of a plurality of nodes of the cluster are completed in the local thread of the client, and the data transmission processing efficiency in the distributed storage system is greatly improved.
Drawings
FIG. 1 is a flow chart of a method for data transmission in a distributed storage system according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data transmission method of a distributed storage system according to an embodiment of the present invention;
FIG. 3 is a flow chart of a data transmission method of a distributed storage system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a connection of call relationships between data transfer nodes according to an embodiment of the present invention;
FIG. 5 is a flow chart of a data transmission method of a distributed storage system according to an embodiment of the present invention;
FIG. 6 is a flow chart of a data transmission method of a distributed storage system according to an embodiment of the present invention;
FIG. 7 is a flow chart of a data transmission method of a distributed storage system according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a data transmission system of a distributed storage system according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of a specific example architecture of a data transfer system application of a distributed storage system provided in an embodiment of the present invention;
Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.
It should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present invention by way of illustration, and only the components related to the present invention are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
In the following description, numerous details are set forth in order to provide a more thorough explanation of embodiments of the present invention, it will be apparent, however, to one skilled in the art that embodiments of the present invention may be practiced without these specific details, in other embodiments, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the embodiments of the present invention.
As shown in fig. 1, the present embodiment provides a data transmission method of a distributed storage system, where the method includes:
step S101: dividing the data to be transmitted into a plurality of divided data, and respectively transmitting each divided data to a target object corresponding to each divided data through a transmission queue.
In the related art, the distributed storage system generally needs to loop requests or respond to other processes when completing the task of distributing the messages, such as multithreading, which is time-consuming and occupies large system resources. The data transmission method of the distributed storage system provided by the embodiment divides the data to be transmitted and then the client to obtain a plurality of divided data, and transmits the divided data through the transmission queue of the local thread of the client, so that the data transmission efficiency is improved.
The method of dividing the data to be transmitted into a plurality of divided data may be implemented in a manner known to those skilled in the art, and the data alignment may be implemented by performing the data alignment on each divided data before each divided data is respectively transmitted to the target object corresponding to each divided data through the transmission queue, which is not limited herein.
In one embodiment, since in a distributed system, multiple copies or erasure codes can be used to ensure the reliability of data, a client needs to write data to multiple nodes (target objects) at this time. At this time, the step of transmitting each piece of divided data to the target object corresponding to each piece of divided data through the transmission queue includes:
generating at least one piece of redundant data of the data to be transmitted, and newly adding the redundant data into the segmentation data;
acquiring a target sending object identifier corresponding to the segmentation data, and generating a message to be sent according to the segmentation data, the target object identifier and the sending identifier to obtain a message to be sent corresponding to each segmentation data;
and respectively sending each message to be sent to the target object corresponding to each piece of divided data through the sending queue.
The generation of the redundant data may be implemented in a manner known to those skilled in the art, and will not be described herein.
Because the redundancy data is added on the basis of the original data to be transmitted to ensure the reliability of the data, the redundancy data also needs to be sent to the corresponding target object at the moment, so that the obtained redundancy data can be determined to be new divided data, added into a divided data set and sent to the target object. Therefore, even if the segmented data (original segmented data) obtained by segmentation of a certain data to be transmitted is lost in the transmission process, the data to be transmitted can be restored by the redundant data and other original segmented data, so that the reliability of data transmission is improved.
The target object may be a server, and the data to be transmitted is stored in the client in an initial state, the data to be segmented is segmented by the client, redundant data is generated, and the segmented data determined by the redundant data are transmitted to the server. The target object is an object to which the split data is to be transmitted. In a distributed storage system, in general, objects (target objects) to be transmitted of respective divided data corresponding to the same data to be transmitted are different.
In order to distinguish the service ends (target objects), a globally unique target object identifier can be preconfigured for each target object, a globally unique sending identifier is preconfigured for each piece of divided data (the divided data obtained by dividing the data to be transmitted and the divided data obtained by determining the generated redundant data), and then a message to be sent corresponding to the divided data is generated according to the sending identifier, the target object identifier and the divided data, and the message to be sent carries the divided data to reach the target object.
In one embodiment, the transmit queue is typically multiple, the number of transmit queues being the same as the number of CPU cores of the CPU that transmitted the message to be transmitted.
Optionally, if the target object identifier includes an IP address, when each message to be sent is sent to the target object corresponding to each split data through the sending queue, each message to be sent may be first hashed according to the IP address, and then uniformly distributed in multiple sending columns in the sending queue.
Step S102: and obtaining response information of each target object, and generating a transmission result of the data to be transmitted.
Because the data to be transmitted is split into a plurality of split data for transmission, the response messages obtained later are also a plurality of, and at this time, the plurality of response messages can be aggregated to obtain the final transmission result of the data to be transmitted.
In one embodiment, the response message includes response identifiers, the response identifiers are in one-to-one correspondence with the sending identifiers, the obtaining the response message of each target object, and the generating the transmission result of the data to be transmitted includes:
acquiring a response identifier in each response message;
determining a transmission identifier corresponding to the response identifier according to a preset mapping relationship between the response identifier and the transmission identifier, and determining the transmission state of each piece of divided data;
and determining a transmission result according to the transmission state of each piece of divided data.
Alternatively, the response identifier may be the same data as the content of the transmission identifier, with the only difference that the response identifier is carried by the message returned by the target object, and the transmission identifier is configured locally. Thus, the response identifier is received locally, and it is possible to know which transmission identifier corresponds to the response identifier, and further to know the transmission state of the divided data corresponding to the transmission identifier, and if the response identifier M is identical to the transmission identifier N in content and the transmission identifier corresponds to the divided data D, it is possible to know that the transmission of the divided data D is successful. If the response identification which is the same as the transmission identification N is not received, the transmission failure of the segmentation data D is indicated.
Of course, the response identifier and the transmission identifier may be different in content, but the response identifier corresponds to the target object one by one, so that after the response identifier of the response message is received, the corresponding relationship between the received response identifier and the transmission identifier and the received response identifier and the received transmission identifier and the received response identifier are known according to the corresponding relationship between the response identifier and the target object, the corresponding relationship between the response identifier and the transmission identifier and the received response identifier are further known, and if the response identifier corresponding to a certain piece of the received response identifier is received, the transmission of the received response identifier and the received response identifier are further known. If the response identification corresponding to a certain piece of divided data is not received or is received overtime, the transmission failure of the piece of divided data is indicated.
In one embodiment, determining the transmission result according to the transmission state of each of the divided data includes:
if the response identification corresponding to the split data does not exist, the transmission state of the split data is transmission failure;
and obtaining the failure number of the target segmentation data, and determining a transmission result, wherein the target segmentation data comprises segmentation data with transmission state of transmission failure.
If the response identification corresponding to the split data exists, the transmission state of the split data is that the transmission is successful.
Alternatively, a type of failure data may be obtained, which includes redundancy (the failed split data is determined by the redundancy data) or original splitting (the failed split data is directly obtained by splitting the data to be transmitted). The user may then be informed of which data failed to transmit for retransmission or further processing by the user.
The manner of determining the transmission result based on the number of failures and the type of failure data may be implemented using decoding rules known to those skilled in the art. For example, when the data to be transmitted is divided into m pieces of divided data, the data to be transmitted is encoded by an erasure coding algorithm to generate n pieces of redundant data, and when the number of failures (whether the original divided data or the redundant data fails) is less than or equal to n, the original m pieces of data can be recovered by a corresponding reconstruction algorithm. At this time, it may be determined that the transmission is successful. Otherwise, when the number of failures is greater than the number of redundant data n, the transmission fails.
Optionally, if the number of failures is greater than the number of redundant data, the transmission fails, otherwise, the transmission is successful.
In one embodiment, generating the transmission result of the data to be transmitted includes at least one of:
Monitoring response messages of all target objects, acquiring monitoring time length, and stopping monitoring to generate a transmission result of data to be transmitted according to the currently acquired response messages of all target objects if the monitoring time length exceeds a preset time length threshold;
monitoring response messages of all target objects, acquiring monitoring time length, and discarding the response messages acquired after the monitoring time length exceeds a preset time length threshold.
The preset time period threshold may be set by those skilled in the art according to needs, and is not limited herein. The monitoring duration can be counted from the completion of sending the message to be sent, and can also be determined by a person skilled in the art according to the need.
By setting the preset time threshold, the occupation of system resources can be saved, and the phenomenon that the whole data transmission process is not dragged in time due to the return of individual response messages is avoided.
By the data transmission method, the data to be transmitted can be segmented, message segregation is realized, request distribution and result merging of a plurality of nodes of the cluster are completed in the local thread of the client, and the data transmission processing efficiency is greatly improved. The method solves the problems that in the related technology, as the decomposition and aggregation of the message request and the response cannot be supported, the task is completed by circulating the request or the response or other processes (such as multithreading and other technologies), which is time-consuming and also increases the consumption of system resources.
The above data transmission method will be described by way of example with reference to a specific embodiment. In a distributed system, the reliability of data is generally ensured by adopting modes such as multiple copies or erasure codes and the like. In this case, the client needs to write one data to multiple nodes. Assuming that the client uses the erasure coding strategy to write data a to the servers S1, S2, S3, S4, S5, S6 (aligned, the data alignment may be implemented in a manner known to those skilled in the art, and is not limited herein), referring to fig. 2, the schematic steps of the data transmission method are as follows:
step S201: the client divides the data to be transmitted into a plurality of divided data to generate redundant data.
For example, the client divides data a (data to be transmitted) into A1, A2, A3, A4 (divided data) according to the size of the slice, calculates erasure codes, and the redundant data thereof is A5, A6. The redundant data is determined as newly added split data.
Step S202: and the client generates a message to be sent according to the segmentation data.
For example, A1, A2, A3, A4, A5, A6 in step S201 are respectively configured as message bodies (messages to be sent) mA1, mA2, mA3, mA4, mA5, mA6, and are characterized as follows:
Each message body has a number (transmission identifier) of 1,2,3,4,5,6 in turn.
Each message body corresponds to each server (target object identifier) to be transmitted, and is sequentially S1, S2, S3, S4, S5 and S6.
Step S203: and sending the message to be sent through the sending queue and monitoring the response message.
mA1, mA2, mA3, mA4, mA5, mA6 are added to a transmit queue, which is typically multiple and the same as the CPU cores. mA1, mA2, mA3, mA4, mA5, mA6 can be basically balanced in a plurality of sending columns after being hashed according to the IP address.
The client waits for response results of mA1, mA2, mA3, mA4, mA5, mA 6.
Step S204: and obtaining response messages of the messages to be sent, and generating a transmission result.
After the service ends S1, S2, S3, S4, S5, S6 process the request (message to be sent), the results are returned, the response bodies (response messages) are rA1, rA2, rA3, rA4, rA5, rA6, and each result return has a response number (response identifier) which corresponds to the numbers of mA1, mA2, mA3, mA4, mA5, mA6 one by one and is 1,2,3,4,5,6 respectively.
The client receives rA1, rA2, rA3, rA4, rA5 and rA6, and the serial numbers (response identification) 1,2,3,4,5 and 6 are aggregated into a response (transmission result) for writing the data A.
Step S205: the timeout response message is discarded.
The client processes the data after receiving the response A, the waiting time of the client can set a timeout (a preset duration threshold), and if the request is not returned within the specified time, the request fails. If the response is returned later, but the client has timed out, the response result (response message) is discarded.
Through the segregation of the messages, the request distribution and result merging of a plurality of nodes of the cluster are completed in the local thread of the client, and the efficiency of data transmission processing is greatly improved.
The embodiment of the invention also provides a data transmission method, referring to fig. 3, the data transmission method comprises the following steps:
step S301: and determining the sending end and each target object as data transmission nodes, and sequentially establishing connection among the data transmission nodes.
In general, a client sends a request to a server and processes a response. In a distributed system, however, there is a relationship between each node, each node is not an isolated presence, and there is a call to each other on the interface. Therefore, it is not possible to just connect a client and a server to a connection, and a connection is called, and both ends should be equal positions, and there is no directivity. Such as nodes S1, S2, S3, S4, S5, S6, the call graph of which can be seen in fig. 4. In fig. 4, S1, S2, S3, S4, S5, S6 respectively represent nodes, and a connection line between the nodes represents a network connection. Therefore, the call among S1, S2, S3, S4, S5 and S6 has no directivity, and can be mutually called on one connection, and each node has call, so the connection call relationship of the cluster forms an undirected complete call graph.
The connection between the data transmission nodes is established, namely, the connection between any two data transmission nodes is established. In other words each data transfer node establishes a connection with each data transfer node other than the data transfer node.
Step S302: each connection is assigned a network handle by which the connection is identified.
The network handle may be system generated during the connection. There is one network handle for each connection.
Step S303: and obtaining the address information of each data transmission node, and establishing a mapping relation between each address information and the network handle.
That is, for a distributed storage system, there will be a connection between every two data transmission nodes, corresponding to a network handle, and the two data transmission nodes have two IP addresses (address information), so that, for convenience in subsequent searching, a mapping relationship between each address information and the network handle may be established first.
Step S304: each data transmission node stores address information, network handle and mapping relation of the connection node, and the connection node is other data transmission nodes connected with the data transmission node.
Optionally, when the distributed storage system is started and initiates connection, the two ends of the connection store the network handle and the opposite end IP (address information of the other end of the connection) generated in the connection process in the local memory. For example, with continued reference to fig. 4, the data transmission node S1 will store 5 address information of the data transmission nodes S2, S3, S4, S5, S6, 5 network handles for 5 connections between S1 and S2, S3, S4, S5, S6, respectively, and correspondence between each network handle and address information. Thus, when the subsequent S1 wants to transmit data to S3, the network handle connected between S3 and S1 can be found through the IP address (address information) of S3, so that the data or the request to be transmitted can be sent.
By the data transmission mode, the request transmission and the response message transmission between any two data transmission nodes can be realized.
When the new data transmission node is needed, the adding of the new node can be realized only by executing according to the steps S301-S304, the new data transmission node is connected with the existing data transmission node in a new way, the network handle of each new connection is generated, each existing data transmission node stores the address information and the network handle of the new data transmission node, and the new data transmission node stores the network handle and the address information of each existing data transmission node.
In one embodiment, in the method shown in fig. 1, the sending end sends each piece of split data to the target object corresponding to each piece of split data through the sending queue, and the data transmission method shown in fig. 3 may also be executed before the sending end sends each piece of split data to the target object corresponding to each piece of split data through the sending queue in the data transmission method shown in fig. 1, and connection establishment and data transmission preparation are performed on each data transmission node of the distributed storage system through the method shown in fig. 3. At this time, the transmitting end transmits each piece of divided data to the target object corresponding to each piece of divided data through the transmitting queue, respectively, including:
acquiring a transmission identifier corresponding to the segmentation data, wherein the transmission identifier comprises address information;
determining a network handle corresponding to the segmentation data according to the address information and the mapping relation to obtain the network handle corresponding to each segmentation data;
the transmitting end transmits each piece of divided data to the corresponding target object of each piece of divided data through the transmitting queue based on each network handle.
The distributed storage system has a large number of connections inside the cluster and needs to be connected with the outside. Therefore, management of the connection is very important, and a large number of connections inevitably incur a large resource management consumption. In the related art, whether the transmission connection is normal or not needs to be continuously polled to send the connection keep-alive message, and when a large number of connections are connected, the performance is greatly affected. To this end, an embodiment of the present invention further provides a data transmission method, referring to fig. 5, where the method includes:
Step S501: and respectively acquiring the data flow of each data transmission node under each connection.
The data transmission nodes are all nodes in the distributed storage system, each connection has two data transmission nodes, and the two data transmission nodes under the connection can communicate through the connection.
Alternatively, when connection is established, a traffic status bit may be set in each of the two connected data transmission nodes, and the data flow status of each data transmission node is monitored, where the initial value is zero, i.e. no traffic passes.
Step S502: and if the zero-flow connection exists, respectively controlling one data transmission node of each zero-flow connection to send a detection message to the other data transmission node of the zero-flow connection at intervals of a first preset time. The detection message is used for triggering another data transmission node to generate a response detection message and feeding back the response detection message through zero-flow connection, and the data flow of at least one data transmission node in the zero-flow connection is zero.
That is, for any connection, if there is zero data traffic of at least one data transmission node, and this state continues for a certain period of time (i.e., the data traffic under a certain preset period of time can be obtained in step S501, if it is zero, it is indicated that the connection is a zero-flow connection), at this time, one data transmission node in the connection period can be controlled to actively send a probe message to another data transmission node, and in the case that the connection state is normal, after receiving the probe message, the other data transmission node will generate a response probe message, and return the response probe message to the one data transmission node through the connection. However, if the connection fails, the other data transmission node will not receive the probe message, and it is impossible to generate a response probe message, and the data traffic of the other data transmission node remains zero.
Step S503: and if the data flow of the other data transmission node of the zero flow connection at intervals of the second preset time is zero, determining the zero flow connection as the invalid connection. Wherein the second preset time is less than the first preset time.
The first preset time and the second preset time may be set by those skilled in the art according to need, and are not limited herein. It should be noted that the first preset time is less than the first preset time. That is, a probe message is sent through the connection F at a certain interval a, and if a certain data transmission node has not received the probe message of the connection F after a certain time B, it is indicated that the connection F fails, that is, the connection F is determined as a failed connection. The connection may be closed for subsequent automatic reconnection.
In one embodiment, since many connections may have natural data transmission, to save resources, the data traffic of each connection may be monitored, and connections with zero data traffic may be probed, while connections with natural data transmission may be protected from probing messages. At this time, the determination modes of the zero-flow connection include:
respectively acquiring data flow of each data transmission node in each connection;
If the data traffic of a certain data transmission node is zero, determining the connection of the data transmission node as zero-flow connection.
In one embodiment, before the data transmission nodes respectively controlling each zero-flow connection transmit the probe message to another data transmission node of the zero-flow connection at intervals of a first preset time, the method further includes:
if at least two zero-stream connections exist, the occurrence frequency of each data transmission node in each zero-stream connection is obtained, and the data transmission node with the larger occurrence frequency in each zero-stream node is determined to be one data transmission node. In this way, as few data transfer nodes as possible can be controlled to implement connection probing.
That is, when the data traffic of a certain connection is zero, any one of the two data transmission nodes H at the two ends of the connection is controlled to send a probe message to the connection node J to be probed at a first preset time interval. In this way, the data traffic of the connection node J is monitored again to know whether the connection state of the connection to be detected is valid. Alternatively, the data transmission node H does not need to wait for a response message of the connection node J.
In one embodiment, the data transmission method shown in fig. 5 may also be applied to the data transmission methods provided in fig. 3 and fig. 1, and the connection state is detected after the connection is established.
By the data transmission method provided by fig. 5, the resource consumption of connection state management can be effectively reduced, and the performance is improved.
The data transmission method provided in fig. 5 is exemplified by one specific embodiment. Referring to fig. 6, the specific method includes:
step S601: a connection is established.
Optionally, the connection is established, the connection information is stored at both ends of the connection, and the connection information is characterized by comprising a traffic status bit, and the initial value is zero when no traffic passes.
Step S602: the data traffic of each connection is monitored and a zero flow connection is determined.
For example, the connection detection module checks the status bit every 1 second, and if there is traffic passing, no other operation is required. If no traffic passes, the connection is regarded as a connection to be detected, and step S603 is performed.
Step S603: and sending a detection message to determine the connection state of the zero-flow connection.
For example, one data transfer node in a zero-flow connection sends a probe message once to the peer data transfer node, but does not have to wait for the peer to respond.
One end (opposite end data transmission node) receiving the probe message, i.e. immediately responding to the probe message, generates a response probe message.
The probe message sender (a data transfer node) changes the status bit to have traffic passing (not zero) because there is traffic passing at this time.
If none of the status bits of the other data transfer node changes from no traffic to traffic, 3 times (3 seconds) consecutively, the connection failure is closed awaiting automatic reconnection.
It should be noted that, in a general state, the network is normal, and the distributed storage system has huge traffic, and traffic passes all the time, that is, only the status bit needs to be checked to determine whether the connection is normal, that is, only the data traffic needs to be monitored, and the data traffic is less frequently zero.
After connection of each data transmission node in the distributed storage system is established, the two ends of the connection can call the opposite end function through the network, in the related technology, the call function is distinguished by manually registering the message type or the call function is distinguished by manually registering the function name, and in the distributed development, the two schemes need to pay attention to whether message conflict or omission exists or not. To this end, the embodiment of the present application further provides a data transmission method, which is also applicable to the data transmission method in any one of the foregoing embodiments, referring to fig. 7, where the method includes:
step S701: and setting a custom data segment for the target object, wherein the custom data segment comprises a plurality of calling functions and calling function identifiers of the calling functions, and the calling functions are used for processing the segmentation data.
And the calling function identification comprising the function to be called is directly sent to realize the calling of the client-side calling function.
The above data transmission method may also be applied to the data transmission method provided in fig. 1, where the message to be sent further includes a calling function identifier, and the method further includes:
after the target object obtains the message to be sent, the message to be sent is processed by calling a function corresponding to the function identifier, and a feedback message is generated;
and acquiring a feedback message fed back by the target object.
Because the target object stores the reflection table related to the mapping relation between the calling function and the calling function identifier, the target object can obtain the calling function from the reflection table according to the acquired calling function identifier, and then the calling function is used for processing the message to be sent (particularly the segmentation data included in the message) so as to generate the feedback message.
In one embodiment, the method further comprises:
acquiring an update message, wherein the update message comprises at least one of a new function and a deletion function;
and updating the custom data segment according to the update message.
If the update message includes the newly added function, the function is newly added in the custom data segment. And if the update message comprises a deleting function, deleting the function in the custom data segment.
If the new function exists, adding the new function and the new function name into the reflection table.
Since the custom data segment of each target object is preset, at this time, if the function of the target object needs to be modified or newly added, the modification can be realized by directly modifying the custom data segment, that is, no matter how to add or delete the calling process later, no configuration is required to be modified, or manual registration is required, only the self business is required to be concerned, and the full-automatic direct remote procedure call is realized.
By the method, remote procedure call can be directly realized, a large number of message types do not need to be manually registered for distinguishing, and development and management are facilitated. The message reflection table (the corresponding relation table of the request name and the processing function) is not required to be concerned when the calling function is updated. The calling end directly corresponds to the end function name.
The method described above with respect to fig. 7 is illustrated by a specific example.
Calling end:
when the remote function is requested, the remote function name is directly called. Such as the remote function name Func,
its call form is: and X (control, # Func, ret, param, … …), the X function packages and sends the Func and the param, … … to the opposite terminal according to the control parameters, the opposite terminal returns a response result to the calling terminal after receiving the request, and ret is the received response result.
The called terminal:
when the user defines the data section XPROC, the process function and the process function name are stored between the data section XPROC_start and the XPROC_end start during development, all the compiled process functions are in the data section XPROC, and can be seen in the elf file:
XPROC_start
FuncName1
Func1
FuncName2
Func2
……
FuncNameN
FuncN
XPROC_end
when a new call is added, the expression form is as follows:
XPROC Char*FuncName＝#Func
XPROC Func(……)；
and then adding the process function Func and the process function name FuncName into a reflection table, and scanning all contents between XPROC_start and XPROC_end start of a data segment XPROC in a starting code before function call to obtain the reflection table.
When a request is received, the process function name corresponding to the request function is searched through the reflection table, and the message parameter is transmitted to the process function name, so that the call can be completed.
After the call is completed, the response is returned to the calling end.
Through the specific embodiment, functions of all nodes in the distributed storage system can be added or deleted conveniently and quickly, and meanwhile, a foundation can be laid for data transmission in a mode of directly calling the functions.
Next, a data transmission method in the above embodiment will be further exemplarily described by a specific embodiment. The connection relationship of each node in the distributed storage system is exemplified by the call graph of fig. 4.
First, when the system is started, the network handle and the opposite end IP (address information) generated by the system in the connection process are stored in the local memory by the two ends of the connection when the connection is initiated.
Secondly, after a connection is established, calling functions and connection exception handling methods can be added into the system at both ends of the connection, and when the call is made, a request can be sent only by finding a corresponding network handle through an IP address (determining the network handle according to the address information in the mapping relation between preset address information and the network handle).
Thirdly, at this time, the call graph is built, the node is newly added subsequently, and the call graph can be continuously expanded according to the first execution and the second execution of the steps.
Fourth, a request (message to be sent) and a response message can be mutually sent between any two nodes on the call graph. For example, S1 initiates a connection and establishes a connection with S4, S1 adds FUNC1, S4 adds FUNC2, S1 queries the IP address of S4 to obtain a network handle, and can call the FUNC2 of S4 through the network, S4 queries the IP address of S1 to obtain a network handle, and can call the FUNC1 of S1 through the network.
In addition, when the connection is disconnected, the transmission module calls an exception handling method of the exceptions S1 and S4, the connection exception handling is triggered by the exceptions S1 and S4, and the corresponding IP address or network handle is handled without checking the connection exception by external polling, so that the undirected complete call of the cluster can be supported, a large number of connections or polls are required to be established, and the complexity and the consumption of system resources are increased.
The data transmission method of the distributed storage system provided by the embodiment supports the decomposition and aggregation of the message request (data to be transmitted) and the response (response message), and the client can respond to the request message of the server to connect with the micro probe and support the direct remote procedure call. The data transmission method for the undirected complete direct remote procedure call is high in cohesion, good in expansibility, high in performance and high in availability, network transmission performance of the distributed storage system is greatly improved, consumption of system resources is reduced, development cost is reduced, and operation and maintenance are convenient.
Referring to fig. 8, the present embodiment provides a data transmission system 800 of a distributed storage system, which includes:
the dividing module 801 divides data to be transmitted into a plurality of divided data, and sends each divided data to a target object corresponding to each divided data through a sending queue;
the aggregation module 802 is configured to obtain a response message of each target object, and generate a transmission result of the data to be transmitted.
Alternatively, the data transmission system 800 of the distributed storage system may be applied to a client (sender) of data transmission to implement decomposition of a message and aggregation of response messages locally.
In one embodiment, the system further comprises at least one of:
the custom data segment module is used for setting custom data segments for target objects, the custom data segments comprise a plurality of calling functions and calling function identifiers of the calling functions, and the calling functions are used for processing the segmentation data.
The function can be added or deleted by editing the custom data segment module.
In one embodiment, the sending end sends each piece of the split data to a target object corresponding to each piece of the split data through a sending queue, and the system further includes:
the connection establishment module is used for determining the sending end and each target object as data transmission nodes and sequentially establishing connection among the data transmission nodes;
the distribution module is used for distributing network handles for each connection and identifying the connection through the network handles;
the mapping establishing module is used for acquiring the address information of each data transmission node and establishing a mapping relation between each address information and the network handle;
and the storage module is used for storing the address information, the network handle and the mapping relation of the connecting nodes respectively by the data transmission nodes, wherein the connecting nodes are other data transmission nodes connected with the data transmission nodes.
In one embodiment, the system further comprises:
the flow monitoring module is used for respectively acquiring the data flow of each data transmission node under each connection;
the control module is used for respectively controlling one data transmission node of each zero-flow connection to send a detection message to the other data transmission node of the zero-flow connection at a first preset time interval if the zero-flow connection exists, and the detection message is used for triggering the other data transmission node to generate a response detection message and feeding back the response detection message through the zero-flow connection, wherein the data flow of at least one data transmission node exists in the zero-flow connection is zero;
and the determining module is used for determining the zero-stream connection as invalid connection if the data flow of the other data transmission node of the zero-stream connection at intervals of a second preset time is zero, wherein the second preset time is less than the first preset time.
In this embodiment, the system is essentially provided with a plurality of modules for executing the method in the above embodiment, and specific functions and technical effects are only required by referring to the above method embodiment, which is not described herein again.
In the following, an exemplary description is made of the data transmission system provided in fig. 8 by using a specific embodiment, referring to fig. 9, the distributed storage data system includes a client C and a server S, where the client C performs data transmission through the data transmission system, the server S invokes a data transmission system interface to start monitoring, the client C invokes a remote function Func of the transmission system, the inside of the data transmission system is automatically connected to the server S, and then a request (such as a message to be sent including split data) is sent to the server S, the server S responds to the request, invokes the Func, and sends a response message to the client C, where the client C processes the response result (obtains the transmission result). In the system, the related process function is directly updated by updating the network business flow, other modifications are not needed, and the system is more convenient and faster.
Referring to fig. 10, an embodiment of the present invention also provides an electronic device 1100 comprising a processor 1101, a memory 1102, and a communication bus 1103;
a communication bus 1103 is used to connect the processor 1101 and the memory connection 1102;
the processor 1101 is configured to execute a computer program stored in the memory 1102 to implement the method as described in one or more of the above embodiments.
The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored,
the computer program is for causing a computer to perform the method according to any one of the above embodiments.
The embodiment of the present application further provides a non-volatile readable storage medium, where one or more modules (programs) are stored, where the one or more modules are applied to a device, and the device may be caused to execute instructions (instructions) of a step included in the embodiment one of the embodiment of the present application.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The above embodiments are merely illustrative of the principles of the present invention and its effectiveness, and are not intended to limit the invention. Modifications and variations may be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, it is intended that all equivalent modifications and variations of the invention be covered by the claims, which are within the ordinary skill of the art, be within the spirit and scope of the present disclosure.
Claims (12)
1. A method for data transmission in a distributed storage system, the method comprising:
dividing data to be transmitted into a plurality of divided data;
determining a sending end and each target object as data transmission nodes, and sequentially establishing connection between the data transmission nodes;
assigning a network handle to each connection, identifying the connection by the network handle;
acquiring address information of each data transmission node, and establishing a mapping relation between each address information and a network handle;
each data transmission node stores address information, a network handle and a mapping relation of a connection node, wherein the connection node is other data transmission nodes connected with the data transmission node;
Respectively acquiring data flow of each data transmission node under each connection;
if zero-flow connection exists, respectively controlling one data transmission node of each zero-flow connection to send a detection message to the other data transmission node of the zero-flow connection at a first preset time interval, wherein the detection message is used for triggering the other data transmission node to generate a response detection message and feeding back the response detection message through the zero-flow connection, and the data flow of at least one data transmission node in the zero-flow connection is zero;
if the data flow of the other data transmission node of the zero-flow connection at intervals of a second preset time is zero, determining the zero-flow connection as invalid connection, wherein the second preset time is less than the first preset time;
the sending end sends the divided data to the target object corresponding to the divided data through a sending queue;
and obtaining response information of each target object, and generating a transmission result of the data to be transmitted.
2. The method of claim 1, wherein transmitting each of the split data to a target object corresponding to each of the split data via a transmit queue comprises:
generating at least one piece of redundant data of the data to be transmitted, and newly adding the redundant data into the segmentation data;
Obtaining a target object identifier corresponding to the segmentation data, and generating a message to be transmitted according to the segmentation data, the target object identifier and the transmission identifier to obtain a message to be transmitted corresponding to each segmentation data;
and respectively sending each message to be sent to a target object corresponding to each piece of divided data through a sending queue.
3. The method of claim 2, wherein the response message includes response identifiers, the response identifiers correspond to the transmission identifiers one-to-one, the obtaining the response message of each target object, and the generating the transmission result of the data to be transmitted includes:
acquiring a response identifier in each response message;
determining a transmission identifier corresponding to the response identifier according to a preset response identifier-transmission identifier mapping relation, and determining the transmission state of each piece of divided data;
and determining the transmission result according to the transmission state of each piece of the divided data.
4. The method of claim 3, wherein determining the transmission result based on the transmission status of each of the divided data comprises:
if the response identification corresponding to the split data does not exist, the transmission state of the split data is transmission failure;
And obtaining the failure number of the target segmentation data, and determining the transmission result, wherein the target segmentation data comprises segmentation data with transmission state of transmission failure.
5. The method according to any of claims 1-4, wherein generating a transmission result of the data to be transmitted comprises at least one of:
monitoring response messages of all the target objects, acquiring monitoring time length, stopping monitoring if the monitoring time length exceeds a preset time length threshold value, and generating a transmission result of the data to be transmitted according to the currently acquired response messages of all the target objects;
monitoring response messages of the target objects, acquiring monitoring time length, and discarding the response messages acquired after the monitoring time length exceeds a preset time length threshold.
6. The method of any of claims 2-4, wherein the message to be sent further comprises a call function identification, the method further comprising:
after the target object obtains the message to be sent, the message to be sent is processed through a calling function corresponding to the calling function identifier, and a feedback message is generated;
and acquiring the feedback information fed back by the target object.
7. The method of claim 6, wherein before each of the split data is sent to the target object corresponding to each of the split data through a send queue, the method further comprises:
And setting a custom data segment for the target object, wherein the custom data segment comprises a plurality of calling functions and calling function identifiers of the calling functions, and the calling functions are used for processing the segmentation data.
8. The method of claim 7, wherein the method further comprises:
acquiring an update message, wherein the update message comprises at least one of a new function and a delete function;
and updating the custom data segment according to the update message.
9. The method of claim 1, wherein the transmitting the split data to the target object corresponding to the split data through the transmit queue by the transmitting end includes:
acquiring a transmission identifier corresponding to the segmentation data, wherein the transmission identifier comprises the address information;
determining a network handle corresponding to the split data according to the address information and the mapping relation to obtain the network handle corresponding to each split data;
the transmitting terminal transmits the divided data to the target object corresponding to the divided data through a transmitting queue based on the network handles.
10. A data transmission system for a distributed storage system, the system comprising:
The segmentation module segments the data to be transmitted into a plurality of segmented data, determines a transmitting end and each target object to be data transmission nodes, and establishes connection among the data transmission nodes in sequence; assigning a network handle to each connection, identifying the connection by the network handle; acquiring address information of each data transmission node, and establishing a mapping relation between each address information and a network handle; each data transmission node stores address information, a network handle and a mapping relation of a connection node, wherein the connection node is other data transmission nodes connected with the data transmission node; respectively acquiring data flow of each data transmission node under each connection; if zero-flow connection exists, respectively controlling one data transmission node of each zero-flow connection to send a detection message to the other data transmission node of the zero-flow connection at a first preset time interval, wherein the detection message is used for triggering the other data transmission node to generate a response detection message and feeding back the response detection message through the zero-flow connection, and the data flow of at least one data transmission node in the zero-flow connection is zero; if the data flow of the other data transmission node of the zero flow connection at intervals of a second preset time is zero, determining the zero flow connection as invalid connection, wherein the second preset time is less than the first preset time, and respectively transmitting each piece of divided data to a target object corresponding to each piece of divided data through a transmission queue by a transmitting end;
And the aggregation module is used for acquiring the response message of each target object and generating a transmission result of the data to be transmitted.
11. An electronic device comprising a processor, a memory, and a communication bus;
the communication bus is used for connecting the processor and the memory;
the processor is configured to execute a computer program stored in the memory to implement the method of any one of claims 1-9.
12. A computer-readable storage medium, having stored thereon a computer program for causing a computer to perform the method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210411972.4A CN114844910B (en) | 2022-04-19 | 2022-04-19 | Data transmission method, system, equipment and medium of distributed storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210411972.4A CN114844910B (en) | 2022-04-19 | 2022-04-19 | Data transmission method, system, equipment and medium of distributed storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114844910A CN114844910A (en) | 2022-08-02 |
| CN114844910B true CN114844910B (en) | 2023-07-25 |
Family
ID=82565994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210411972.4A Active CN114844910B (en) | 2022-04-19 | 2022-04-19 | Data transmission method, system, equipment and medium of distributed storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114844910B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113949554A (en) * | 2021-10-13 | 2022-01-18 | 东南大学 | High-speed transmission method for distributed network global situation awareness data |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104283906B (en) * | 2013-07-02 | 2018-06-19 | 华为技术有限公司 | Distributed memory system, clustered node and its section management method |
| US9823969B2 (en) * | 2014-09-02 | 2017-11-21 | Netapp, Inc. | Hierarchical wide spreading of distributed storage |
| CN104486442B (en) * | 2014-12-29 | 2018-08-24 | 成都极驰科技有限公司 | Data transmission method, the device of distributed memory system |
| CN105095013B (en) * | 2015-06-04 | 2017-11-21 | 华为技术有限公司 | Date storage method, restoration methods, relevant apparatus and system |
| CN106101213A (en) * | 2016-06-08 | 2016-11-09 | 四川新环佳科技发展有限公司 | Information-distribution type storage method |
| CN107343037B (en) * | 2017-06-30 | 2020-07-07 | 郑州云海信息技术有限公司 | Data transmission method and device of distributed storage system |
| CN112969198A (en) * | 2021-02-24 | 2021-06-15 | 天冕信息技术(深圳)有限公司 | Data transmission method, terminal and storage medium |
| CN113515531B (en) * | 2021-05-08 | 2022-12-02 | 重庆紫光华山智安科技有限公司 | Data access method, device, client and storage medium |
| CN113434273B (en) * | 2021-06-29 | 2022-12-23 | 平安科技（深圳）有限公司 | Data processing method, device, system and storage medium |
-
2022
- 2022-04-19 CN CN202210411972.4A patent/CN114844910B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113949554A (en) * | 2021-10-13 | 2022-01-18 | 东南大学 | High-speed transmission method for distributed network global situation awareness data |
Non-Patent Citations (3)
| Title |
|---|
| Distributed storage scenario in a wide area WDM mesh architecture under heterogeneous traffic;Taisir E.H. El-Gorashi et al.;《2009 International Conference on Optical Network Design and Modeling》;1-6 * |
| 基于Hadoop分布式文件系统的地震勘探大数据样本采集及存储优化;杨河山等;《油气地质与采收率》(第1期);121-127 * |
| 面向实时数据流的高效分布式存储与查询算法研究;林峰级;《中国优秀硕士学位论文全文数据库 (信息科技辑)》(第2期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114844910A (en) | 2022-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109120678B (en) | Method and apparatus for service hosting of distributed storage system | |
| CN107295080B (en) | Data storage method applied to distributed server cluster and server | |
| US9705752B2 (en) | Reliably updating a messaging system | |
| CN115004673B (en) | Message pushing method, device, electronic equipment and computer readable medium | |
| CN106888233B (en) | Data updating system and method | |
| CN103237060B (en) | A kind of data object acquisition, Apparatus and system | |
| CN112118315A (en) | Data processing system, method, device, electronic equipment and storage medium | |
| CN112445626B (en) | Data processing method and device based on message middleware | |
| CN109783151B (en) | Method and device for rule change | |
| CN113794652B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111506388B (en) | Container performance detection method, container management platform and computer storage medium | |
| CN114844910B (en) | Data transmission method, system, equipment and medium of distributed storage system | |
| CN101189581A (en) | Techniques for handling lock-related inconsistencies | |
| US10951732B2 (en) | Service processing method and device | |
| CN116319758A (en) | Data migration method, device, electronic equipment and readable storage medium | |
| CN112860505A (en) | Method and device for regulating and controlling distributed clusters | |
| CN114070889B (en) | Configuration method, traffic forwarding device, storage medium, and program product | |
| CN114816866A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN117354956A (en) | 5G-based session establishment method and device, electronic equipment and storage medium | |
| CN109309583B (en) | Information acquisition method and device based on distributed system, electronic equipment and medium | |
| CN114884805A (en) | Data transmission method, device, terminal and storage medium | |
| CN114338461A (en) | Network connection monitoring method and related equipment | |
| CN113254097A (en) | Configuration information issuing method and device, electronic equipment and storage medium | |
| CN113365231A (en) | Method and device for sending short message | |
| CN113495794B (en) | Module bridging method and device for Android system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240315 Address after: No. 182 Liangjiang Avenue, Yuzui Town, Liangjiang New Area, Jiangbei District, Chongqing, 400000 (self committed) Patentee after: Chongqing Ziguang Huazhi Electronic Technology Co.,Ltd. Country or region after: China Address before: No.117-386, Yunhan Avenue, Beibei District, Chongqing Patentee before: CHONGQING UNISINSIGHT TECHNOLOGY Co.,Ltd. Country or region before: China |