CN111064587B - Node of distributed data system and broadcast transmission data management method - Google Patents
Node of distributed data system and broadcast transmission data management method Download PDFInfo
- Publication number
- CN111064587B CN111064587B CN201911116881.2A CN201911116881A CN111064587B CN 111064587 B CN111064587 B CN 111064587B CN 201911116881 A CN201911116881 A CN 201911116881A CN 111064587 B CN111064587 B CN 111064587B
- Authority
- CN
- China
- Prior art keywords
- data
- data packet
- node
- received
- sending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/02—Details
- H04L12/16—Arrangements for providing special services to substations
- H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
- H04L12/1854—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast with non-centralised forwarding system, e.g. chaincast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
- H04L43/0847—Transmission error
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/32—Flow control; Congestion control by discarding or delaying data units, e.g. packets or frames
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention relates to a node of a distributed data system and a broadcast transmission data management method, which manage data in the distributed data system and a broadcast transmission mode, realize a set of redundant data elimination mechanism of broadcast transmission and avoid repeated storage and transmission; the invention can realize the management of the node data in the distributed data system, ensure the high performance of the distributed system, improve the communication efficiency and the equipment utilization rate of the broadcast transmission network, reduce the network delay as much as possible and reduce the network congestion influence. The robustness of the whole system is improved through broadcast transmission, and the influence of the disconnection and restart processes of local nodes on the data transmission of the system is reduced.
Description
Technical Field
The present invention relates to the field of communications, and in particular, to a node of a distributed data system and a broadcast transmission data management method.
Background
Conventional financial data typically employs a central host in a master-slave architecture to record transaction data. In the current financial market, more and more networks of distributed systems are used to process financial data, i.e., distributed processing tasks are performed on multiprocessor architectures interconnected by communication networks. Each data of the distributed cluster system is stored in a plurality of nodes of the distributed cluster, and in such an environment, data redundancy is brought to a certain extent. The redundant data is necessary for improving the reliability of the system, but if the computing node resources in the distributed system are not managed, unnecessary easy data is cleared in time, and network congestion and node resource waste are easily caused.
In the process of broadcast transmission of the distributed data system, the system broadcasts the node computers of the same channel, and the node computers need to be forwarded through the directly connected node computers under the condition that the node computers are not directly connected. Each node computer sends data to the directly connected node computer, and after receiving the data packet, each node computer forwards the data packet to the directly connected node computer until all node computers receive the data packet. Under the mechanism, the data packets are repeatedly sent in the process of being forwarded through different paths, which results in huge data volume stored in the distributed system and very large number of copies of each data object, thereby undoubtedly increasing the redundancy of network transmission and reducing the network communication efficiency. It is therefore desirable to provide a node and a broadcast transmission data management method for a distributed data system.
Disclosure of Invention
The invention aims to overcome the defects and provides the nodes of the distributed data system and the broadcast transmission data management method, and the invention can realize the management of the node data in the distributed data system, ensure the high performance of the work of the distributed system, improve the communication efficiency and the equipment utilization rate of the broadcast transmission network, reduce the network delay as much as possible and reduce the network congestion influence. The robustness of the whole system is improved through broadcast transmission, and the influence of the disconnection and restart processes of local nodes on the data transmission of the system is reduced.
Each node receives a data packet sent by a direct connection node, further processes the data packet and sends the data packet to other direct connection nodes, or directly forwards the data packet received by the node to other direct connection nodes. The node comprises a node data receiving unit, a node data sending unit, a receiving checking unit, a data processing queue, a message sending queue and a data processing unit.
The node data receiving unit establishes network connection with the directly connected nodes and receives data packets sent by the directly connected nodes through the IO ports.
And the node data sending unit is used for establishing network connection with the directly connected nodes and sending the data packets in the message sending queue to other directly connected nodes one by one through the IO port.
And the receiving and checking unit is used for judging and verifying the received data packet, and then storing the judged and verified data packet into a message sending queue for sending or entering a data processing queue for further processing.
And the message sending queue is used for sending the message queues to other nodes and sending/forwarding the data packets in the queues through the node data sending unit according to the principle of first-in first-out.
And the data processing queue is used for storing the data packet queue which needs to be subjected to data processing in the verified data packets, so that the subsequent data processing unit can process the data packets in the queue on the principle of first-in first-out.
The data processing unit can respond to the data updating operation information and perform updating operation on the data packets in the data processing queue; and sending the updated data through the message sending queue.
Preferably, the system also comprises a data caching unit and an encryption unit; and the data cache unit is used for storing the data packets processed by the data processing unit, and after the data packets are encrypted by the encryption unit and transmitted, the data packets in the data cache can be discarded.
And the encryption unit is used for encrypting the data packet subjected to the data updating operation according to the sending priority, and the encrypted data packet is stored in the message sending queue to wait for sending the data packet.
Preferably, the determining and verifying of the received data packet in the receiving and verifying unit includes: judging whether the data packet is a data packet which is continuously sent after the sending node is restarted; verifying whether the received data packet has been received from other forwarding paths.
Preferably, whether the data packet is a data packet which is continuously sent after the restarting of the sending node can be judged through the timestamp of the sending node starting packet; the sequence number of the data packet can be obtained through internal program accumulation, so that the data packet is verified to be received from other forwarding paths.
Preferably, the received data packet is verified, the receiving verification unit needs to perform additional verification on the length of the data packet so as to verify whether transmission of the received data packet is in error, and after the verification is completed, packet loss processing is performed on the received data packet.
The invention also provides a broadcast transmission data management method of the distributed data system, which comprises the following operation steps:
a) receiving a data packet;
b) recording the sequence number and the length of the data packet;
c) judging whether the data packet sending node is restarted or not;
d) continuously verifying whether the data packet is received for the first time or not under the condition of no restarting;
e) for a data packet received for the first time, the data packet needs to be received, the accumulated length of the data packet of the same sending node is recorded, and the data packet is forwarded or otherwise processed; for the data packet which is not received for the first time, after recording the accumulation length, carrying out length check on the accumulation length, and then discarding the data packet;
f) returning to the step a), receiving the next new data packet.
Preferably, in step c), whether the data packet is a data packet that is continuously sent after the sending node is restarted can be judged by the timestamp of the sending node start packet; in step d), the sequence number of the data packet can be obtained through internal program accumulation, so as to verify whether the data packet is received for the first time.
Preferably, in step c), when there is a restart of the transmitting node, if there is a transmission delay in the data packet after the restart, it is necessary to wait for all the delayed data packets to be transmitted, and packet loss processing cannot be performed; if the sending node does not have the condition of sending delay after restarting, the step d) is continued to verify whether the data packet is received for the first time.
Preferably, in step e), for the data packets that are not received for the first time, the length check method is to encode the header of the same data packet and number the data packets in sequence, and compare the accumulated value of the length of the data packet that is received for the first time, sent from the same sending node and forwarded through the same path with the accumulated value of the length of the data packet that is received for the first time, sent from the same sending node and forwarded through other paths, so as to check whether the data packet is in error or has packet loss during the forwarding process of different paths.
The invention has the beneficial effects that: by managing data in the distributed data system and a broadcast transmission mode, a set of broadcast transmission redundant data elimination mechanism is realized, repeated storage and transmission are avoided, the network propagation efficiency and the equipment utilization rate of the system are improved, the network delay is reduced, and the network congestion influence is reduced. The node restart management and the length check can effectively reduce the influence of local node disconnection and restart on the whole data transmission, improve the system reliability and increase the system robustness under the condition of reducing the additional operation overhead of the system.
Drawings
FIG. 1 is a block diagram of a distributed data system node according to a preferred embodiment of the present invention;
fig. 2 is a flowchart of a method for managing broadcast transmission data according to a preferred embodiment of the present invention.
Detailed Description
The invention will be further described with reference to specific examples, but the scope of the invention is not limited thereto:
example (b): in the embodiments of the present invention, it should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance. In the specification and claims, the terms "include" and "comprise" are used in an open-ended fashion and thus should be interpreted to mean "include, but not limited to … …".
Referring to fig. 1, fig. 1 is a schematic structural diagram of a node, and a node of a distributed data system includes a node data receiving unit, a node data transmitting unit, a receiving checking unit, a data processing queue, a message transmitting queue, a data processing unit, a data cache, an encryption unit, and a node control unit.
In a distributed data system, data needs to be transmitted broadcast by nodes on a communication network. The node may be a general purpose computer, a special purpose computer, a server, an embedded processor, other programmable data processing apparatus, or one or more computer programs stored on a computer readable storage medium that can execute instructions to perform specified functions. Each node receives the data packet sent by the direct connection node, further processes the data packet and sends the processed data packet to other direct connection nodes, or directly forwards the data packet received by the node to other direct connection nodes.
And the node data receiving unit runs on the node computer, establishes network connection with the directly connected nodes, receives data packets sent by the directly connected nodes through the IO ports and carries out the next processing.
And the node data sending unit runs on the node computer, establishes network connection with the directly connected nodes, and sends the data packets in the message sending queue to other directly connected nodes one by one through the IO port.
And the receiving and checking unit is used for judging and verifying the received data packet, and then storing the judged and verified data packet into a message sending queue for sending or entering a data processing queue for further processing.
In one embodiment, the determination of the received data packet includes determining whether the data packet is a data packet that is continuously transmitted after the restart of the transmitting node. Specifically, in a distributed data system, a node computer may broadcast a boot package including the node computer name and a boot timestamp after booting. After other nodes receive the broadcast data packet information, whether the data packet is the data packet sent after the node is restarted or not can be judged through the timestamp of the node computer startup packet received first. Because the data packet number can be restored to the initial value after the node computer is restarted, the step of removing the duplicate packet and losing the packet can not be carried out when the data packet number is judged to be the data packet sent after the node computer is restarted.
In a specific embodiment, it is necessary to verify whether the received packet was first received. Specifically, the sending node information may be obtained from the header data encoding of each received data packet, and the sequence number of the data packet may be obtained by internal program accumulation, thereby verifying whether the data packet is received for the first time. If the data packet is received for the first time, the data packet is received, the accumulated length of the data packet sent from the same node is recorded through data coding of a data packet header, and then data forwarding or data processing is carried out in a message queue according to needs.
In a specific embodiment, the data packet is not first received, the receiving verification unit needs to perform additional verification on the length of the data packet to verify whether transmission of the received data packet is in error, and after the verification is completed, packet loss processing is performed on the data packet which is repeatedly received. Specifically, the accumulated values of the lengths of the data packets received from the same transmitting node and the same forwarding path node for the first time are recorded, and compared with the accumulated values of the lengths of the data packets received from the same transmitting node and different forwarding path nodes for non-first time, so as to check whether transmission errors occur. And if the transmission is correct, the next data packet processing is carried out. And if the transmission is wrong, the abnormity of the verification error needs to be thrown out, log recording is carried out, and the user is prompted according to the requirement. After verification, packet loss processing is carried out on the data packets which are repeatedly received, and equipment resources are released in time.
The node sends/forwards the data packet in the queue through the node data sending unit according to the principle of first-in first-out.
And the data processing queue is used for storing the data packet queue which needs to be subjected to data processing in the verified data packets, so that the subsequent data processing unit can process the data packets in the queue on the principle of first-in first-out.
And the data processing unit can respond to the data updating operation information and perform updating operation on the data packets in the data processing queue. In one embodiment, the data update operation comprises: packet addition, deletion, modification, or data rollback. After each successful update/rollback operation, the packet number is recorded, the version number of the data packet is changed, and a confirmation message of successful or failed update/rollback is sent to other unit modules in the node through the node control unit.
And the data cache unit is used for storing the data packets processed by the data processing unit, and after the data packets are encrypted by the encryption unit and transmitted, the data packets in the data cache can be discarded.
And the encryption unit is used for encrypting the data packet subjected to the data updating operation according to the sending priority, and the encrypted data packet is stored in the message sending queue to wait for sending the data packet.
In one embodiment, the system divides the data packets into high priority data packets and low priority data packets according to the content of the data packets, and the encryption unit encrypts the data packets respectively by using different encryption programs according to different priorities. The high-priority data packet has a priority sending right in the sending process, and can preempt a sending channel.
In one embodiment, high priority data packets include transaction signals, initiation packets, market data, return for trade, etc., and low priority packets include control data that is transaction independent, monitoring data, bin data that does not have a high real-time requirement, etc. Where low priority packets comprise the vast majority of all data packets.
In a specific embodiment, in order to complete the sending of the unsent low-priority data packet in the case of the node restart, the unsent low-priority data packet needs to be converted into a high-priority data packet. Therefore, the data buffer unit stores the unencrypted data packet which is sent recently so as to encrypt the data by the high-priority encryption program.
In one embodiment, for the data packet in the data cache unit, when the data packet is broadcast, the data packet in the data cache unit may be discarded after all node computers receive the data packet.
And the node control unit is used for managing the running states of all the node units in the nodes, collecting the running information of the nodes, and managing the unique address and the monitoring port of each node. And receiving the data packet check information to confirm whether the data packet reception is in error. Each node can receive and process the data packet request of the directly connected node, and the reliability and the efficiency of data information transmission of the distributed system are improved.
By the node structure, the management of node data in a broadcast transmission system can be realized, the synchronization problem of data transmission in a distributed network in the whole network is realized, the robustness of the system is improved, and the influence of the disconnection and restart processes of local nodes on the data transmission of the whole system is reduced.
Based on the same inventive concept, the invention also provides a broadcast transmission data management method of the distributed data system, which comprises the following steps:
a) the data packet is received and the data packet is transmitted,
b) the packet sequence number and the packet length are recorded,
c) judging whether the data packet sending node is restarted or not;
d) continuously verifying whether the data packet is received for the first time or not under the condition of no restarting;
e) for a data packet received for the first time, the data packet needs to be received, the accumulated length of the data packet of the same sending node is recorded, and the data packet is forwarded or otherwise processed; for the data packet which is not received for the first time, after recording the accumulation length, carrying out length check on the accumulation length, and then discarding the data packet;
f) returning to the step a), receiving the next new data packet.
As shown in fig. 2, fig. 2 is a flowchart illustrating a broadcast transmission data management method according to an embodiment of the present invention, and when the present invention is applied, the steps of the broadcast transmission data management method of the distributed data system should be implemented according to the following flows.
S101, the node computer receives a new data packet.
In one embodiment, in broadcast transmission, the node computers need to forward through the directly connected node computers without being directly connected. Each node computer sends data to the directly connected node computer, and after receiving the data packet, each node computer forwards the data packet to the directly connected node computer until all node computers receive the data packet.
And S102, recording the sequence number and the length of the data packet. The packet sequence number can be acquired by the node internal program count, and then the process proceeds to step S103.
In a specific embodiment, according to the characteristics of broadcast network transmission, each node computer sends a data packet to all directly connected node computers, and the node computer receiving the data packet forwards each received data packet until all node computers receive the data packet. Each data packet header has a code, the header code records the information of the sending node, and the data packet sequence number can be obtained through internal program counting. Under the condition that the node is not restarted, the data packet sending/forwarding of the same node computer is carried out according to a certain time sequence, so that the data packets sent/forwarded from the same node computer are received according to the data packet sequence numbering sequence for the receiving node computer, but the data packets forwarded from different paths are not received according to the data packet sequence numbering before being received by the receiving node computer.
S103, obtaining the computer information of the sending node according to the data packet number, and judging whether the node restart exists according to the starting packet of the sending node. If there is no node restart, step S104 is performed. If the storage node is restarted, step S111 is performed.
In one embodiment, each time the node computer boots up, it broadcasts a boot packet with a high priority in transmission. The start packet header contains the name of the start node and a timestamp recording the start time of the node. After each data node receives the data packet, whether the node restart exists in the sending node can be judged by inquiring the starting packet timestamp of the data packet sending node.
S104, verifying whether the data packet is received for the first time. The sending node of the data packet can be obtained through the header coding of the data packet, and the sequence number of the data packet can be obtained through the internal program counting, so that whether the data packet is received or not is judged.
In one embodiment, the same packet may be forwarded from the node computer via different paths. Received data packets may also be forwarded from different nodes, and the receiving time may also be different due to different forwarding paths. If the data packet is received for the first time, step S105 is performed, and if the data packet has been forwarded from another path node, step S107 is performed.
And S105, receiving the data packet which is received for the first time after verification, and recording the accumulated length of the data packet of the same sending node. The process advances to step S106.
In one embodiment, packets sent to the same sending node computer may be forwarded by different nodes. After the data packets pass through different paths, the time for receiving the data packets by the same receiving node is different. For the data packets sent by the same sending node and forwarded through the same path, the data packets are sequentially received at the receiving node. When the data packet is received for the first time, the receiving node needs to receive the data packet, and according to the data packet header code, the lengths of the data packets which are sent by the same sending node and forwarded through the same path are accumulated, and the accumulated length value is recorded, because of extra verification and comparison in the later period.
And S106, forwarding the data packet or performing other processing. Then proceeds to step S101 to prepare for receiving a new packet.
In one embodiment, for a first received packet, the receiving node computer may need to receive the packet, forward it, or otherwise process it after recording the accumulated length.
S107 records the accumulated length of the data packet. If the received data packet with the same number sent by the same sending node has already been received from another path, it is only necessary to record the accumulated length of the path for receiving the data packet sent by the same node, and then the process goes to step S108.
In one embodiment, after a receiving node receives a data packet, if it is verified that the data packet is not received for the first time, i.e. has been received from another path, only the lengths of the data packets transmitted from the same transmitting node and forwarded through the same path need to be accumulated.
And S108, checking the accumulation length. If the length check is passed, the flow proceeds to step S109 to discard and deduplicate the packet. If the length check is not passed, the broadcast forwarding of the data packet is in error, and the process proceeds to step S110.
In one embodiment, the length check method includes coding the same data packet header and numbering the data packets in sequence, and checking and comparing the accumulated value of the lengths of the data packets which are received for the first time, sent from the same sending node and forwarded through the same path with the accumulated value of the lengths of the data packets which are not received for the first time, sent from the same sending node and forwarded through other paths, so as to check whether the data packets are in error or have packet loss in the forwarding process of different paths.
And S109, discarding the data packet. Then proceeds to step S101 to prepare for receiving a new packet.
And S110, when the length check fails, the data packet is abnormal in the forwarding process, the system throws the abnormality, and warning reminding and log recording are required according to the abnormal grade. The flow proceeds to step S109 to discard the packet.
In one embodiment, after the length check passes, it is indicated that the data packet has no packet loss or other abnormality in the broadcast transmission, and the data packet is directly discarded without duplication. And when the length check fails, the data packet is abnormal in the forwarding process, warning reminding and log recording are required to be carried out according to the abnormal level, and then the data packet is discarded.
And S111, if the transmitting node is restarted, judging whether the data packet is transmitted by the total size of the received data without delay. If the vehicle has not caught up, the process proceeds to step S112, and if the vehicle has caught up, the process proceeds to step S104.
In one embodiment, the existence of the restart condition of the data packet sending node computer can be judged through the timestamp of the startup packet. Due to timeliness of data packet sending, the sending of a node data packet needs to be synchronized with surrounding direct-connection node computers. After the node is restarted, the data packet transmission is delayed, and the delayed data packet needs to be transmitted to perform transmission pursuit. And judging the degree of data transmission delay by the total length of the transmitted data packet after the node is restarted. If the data packet has caught up, the flow proceeds to step S104, where the normal data packet transmission flow is performed. If the packet does not catch up, the flow proceeds to step S112, where the packet is directly received and forwarded or otherwise processed.
S112, directly receiving the data packet, and forwarding or performing other processing. After the forwarding and processing are completed, the process proceeds to step S101 to start receiving a new packet.
In one embodiment, the numbering of the packets may begin to be renumbered due to the restart. Therefore, the packet loss deduplication step cannot be performed until the packet has not caught up as judged by the total length of the received packets. The data packet needs to be directly received, forwarded or further processed, and steps such as judgment, verification, packet loss and duplicate removal are not performed. After catching up, step S104 may be performed to perform the accumulated length verification and the packet loss deduplication according to the normal data packet processing flow.
While the invention has been described in connection with specific embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (8)
1. A node of a distributed data system is used for broadcast data transmission, each node receives a data packet sent by a direct-connected node, further processes the data packet and sends the data packet to other direct-connected nodes, or directly forwards the data packet received by the node to other direct-connected nodes, and the distributed data system is characterized in that the node comprises a node data receiving unit, a node data sending unit, a receiving and checking unit, a data processing queue, a message sending queue and a data processing unit;
the node data receiving unit establishes network connection with the direct connection node and receives a data packet sent by the direct connection node through an IO port;
the node data sending unit is used for establishing network connection with the directly connected nodes and sending the data packets in the message sending queue to other directly connected nodes one by one through the IO port;
the receiving and checking unit is used for judging and verifying the received data packet, and then storing the judged and verified data packet into a message sending queue for sending or entering a data processing queue for further processing;
judging and verifying the received data packet, specifically comprising: judging whether the data packet is a data packet which is continuously sent after the sending node is restarted; verifying whether the received data packet has been received from other forwarding paths;
the data packet is verified to be not received for the first time, the receiving verification unit needs to perform extra verification on the length of the data packet so as to verify whether transmission of the received data packet is error or not, and after the verification is completed, packet loss processing is performed on the data packet which is received repeatedly;
the message sending queue is used for sending/forwarding the data packets in the message queue sent by other nodes through the node data sending unit according to the principle of first-in first-out;
the data processing queue is used for storing the data packet queue which needs to be subjected to data processing in the verified data packets, so that the subsequent data processing unit can process the data packets in the queue on the principle of first-in first-out;
the data processing unit can respond to the data updating operation information and perform updating operation on the data packets in the data processing queue; and sending the updated data through the message sending queue.
2. The node of a distributed data system according to claim 1, further comprising a data caching unit and an encryption unit; the data cache unit is used for storing the data packets processed by the data processing unit, and after the data packets are encrypted and transmitted by the encryption unit, the data packets in the data cache can be discarded;
and the encryption unit is used for encrypting the data packet subjected to the data updating operation according to the sending priority, and the encrypted data packet is stored in the message sending queue to wait for sending the data packet.
3. The node of claim 2, wherein the timestamp of the start packet of the sending node is used to determine whether the data packet is a data packet that is continuously sent after the sending node is restarted; the sequence number of the data packet can be obtained through internal program accumulation, so that the data packet is verified to be received from other forwarding paths.
4. A node of a distributed data system according to claim 3, wherein for a received data packet, the receiving check unit needs to perform an additional check on the length of the data packet to verify whether the received data packet is transmitted with errors, and the length of the data packet needs to be accumulated for data packets transmitted from the same transmitting node and forwarded through the same path; the length checking method comprises the steps of coding the same data packet header and numbering the data packets in sequence, and checking and comparing the accumulated value of the lengths of the data packets which are received for the first time, sent from the same sending node and forwarded through the same path with the accumulated value of the lengths of the data packets which are not received for the first time, sent from the same sending node and forwarded through other paths; and after the verification is finished, performing packet loss processing on the received data packet.
5. A method for managing broadcast transmission data of a distributed data system, comprising the steps of:
a) receiving a data packet;
b) recording the sequence number and the length of the data packet;
c) judging whether the data packet sending node is restarted or not;
d) continuously verifying whether the data packet is received for the first time or not under the condition of no restarting;
e) for a data packet received for the first time, the data packet needs to be received, the accumulated length of the data packet of the same sending node is recorded, and the data packet is forwarded or otherwise processed; for the data packet which is not received for the first time, after recording the accumulation length, carrying out length check on the accumulation length, and then discarding the data packet;
the accumulation length is the length accumulation of data packets which are sent from the same sending node and forwarded through the same path;
the length checking method comprises the steps of coding the same data packet header and numbering the data packets in sequence, and checking and comparing the accumulated value of the lengths of the data packets which are received for the first time, sent from the same sending node and forwarded through the same path with the accumulated value of the lengths of the data packets which are not received for the first time, sent from the same sending node and forwarded through other paths;
f) returning to the step a), receiving the next new data packet.
6. The method for managing broadcast transmission data according to claim 5, wherein in step c), it can be determined whether the data packet is a data packet that is continuously transmitted after the restart of the transmitting node by the timestamp of the transmitting node start packet; in step d), the sequence number of the data packet can be obtained through internal program accumulation, so as to verify whether the data packet is received for the first time.
7. The method according to claim 6, wherein in step c), for the case that the sending node is restarted, if there is a transmission delay in the data packet after restarting, it is necessary to wait for all the delayed data packets to be sent out, and packet loss processing cannot be performed; if the sending node does not have the condition of sending delay after restarting, the step d) is continued to verify whether the data packet is received for the first time.
8. The method for managing data transmission in broadcast according to claim 7, wherein in step e), for the non-first received data packet, the length check method is as follows: the method comprises the steps of coding the same data packet header and numbering data packets in sequence, and comparing the accumulated value of the length of the data packet which is received for the first time, sent from the same sending node and forwarded through the same path with the accumulated value of the length of the data packet which is not received for the first time, sent from the same sending node and forwarded through other paths, so as to check whether the data packet has errors or not in the forwarding process of different paths and whether the data packet has packet loss or not.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911116881.2A CN111064587B (en) | 2019-11-15 | 2019-11-15 | Node of distributed data system and broadcast transmission data management method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911116881.2A CN111064587B (en) | 2019-11-15 | 2019-11-15 | Node of distributed data system and broadcast transmission data management method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111064587A CN111064587A (en) | 2020-04-24 |
| CN111064587B true CN111064587B (en) | 2021-09-07 |
Family
ID=70298535
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911116881.2A Active CN111064587B (en) | 2019-11-15 | 2019-11-15 | Node of distributed data system and broadcast transmission data management method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111064587B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113163346B (en) * | 2021-04-21 | 2022-04-05 | 成都锐成芯微科技股份有限公司 | Broadcast packet filtering method and wireless communication system |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101742554A (en) * | 2010-01-15 | 2010-06-16 | 南京邮电大学 | Congestion control method of multi-path transmission mechanism of wireless multimedia sensor network |
| CN102201903A (en) * | 2011-06-07 | 2011-09-28 | 合肥华云通信技术有限公司 | Simple and reliable remote file transmission method |
| CN202841192U (en) * | 2012-09-01 | 2013-03-27 | 林宏刚 | High-speed reliable one-way transmission system supporting wide area network |
| CN104753790B (en) * | 2013-12-26 | 2018-05-04 | 华为技术有限公司 | A kind of message transmitting method and equipment based on TRILL network |
| BR112018013438A2 (en) * | 2015-12-29 | 2019-04-24 | Amazon Technologies, Inc. | network technologies |
| CN108881008B (en) * | 2017-05-12 | 2021-06-08 | 华为技术有限公司 | Data transmission method, device and system |
| CN110351024B (en) * | 2018-04-04 | 2021-06-15 | 华为技术有限公司 | Data transmission method and device |
| CN109889312A (en) * | 2019-01-28 | 2019-06-14 | 深圳市比速智网技术有限公司 | Multilink data transmission method, device and computer readable storage medium |
-
2019
- 2019-11-15 CN CN201911116881.2A patent/CN111064587B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111064587A (en) | 2020-04-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10877935B2 (en) | Stream computing system and method | |
| US7337214B2 (en) | Caching, clustering and aggregating server | |
| US7801997B2 (en) | Asynchronous interconnect protocol for a clustered DBMS | |
| US20070076625A1 (en) | Data communication apparatus | |
| US10862786B2 (en) | Method and device for fingerprint based status detection in a distributed processing system | |
| CN104584524B (en) | It polymerize the data in intermediary system | |
| US20100229182A1 (en) | Log information issuing device, log information issuing method, and program | |
| WO2007076613A1 (en) | Systems and methods for improved network based content inspection | |
| CN109388481B (en) | Transaction information transmission method, system, device, computing equipment and medium | |
| CN114338651A (en) | File transmission method and device, electronic equipment and readable storage medium | |
| CN106921665B (en) | Message processing method and network equipment | |
| CN113259143A (en) | Information processing method, device, system and storage medium | |
| CN112381649A (en) | Transaction consensus method, device and equipment based on block chain | |
| CN111064587B (en) | Node of distributed data system and broadcast transmission data management method | |
| US9652307B1 (en) | Event system for a distributed fabric | |
| CN114584560A (en) | Fragmented frame recombination method and device | |
| CN107819697B (en) | Data transmission method, switch and data center | |
| CN114090555A (en) | AIS data processing method and system | |
| CN115473858B (en) | Data transmission method, stream data transmission system, computer device and storage medium | |
| CN109284257B (en) | Log writing method and device, electronic equipment and storage medium | |
| CN114866395A (en) | Multi-component distributed consistency protocol cluster heartbeat communication method and device | |
| US20190073281A1 (en) | Information processing system | |
| CN112313638A (en) | Efficient time-based association of data streams | |
| CN110071868B (en) | Link aggregation method, device and network equipment | |
| CN116723154A (en) | Route distribution method and system based on load balancing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |