CN115623000B - Method and device for efficiently distributing data on digital networking - Google Patents

Method and device for efficiently distributing data on digital networking Download PDF

Info

Publication number
CN115623000B
CN115623000B CN202211194855.3A CN202211194855A CN115623000B CN 115623000 B CN115623000 B CN 115623000B CN 202211194855 A CN202211194855 A CN 202211194855A CN 115623000 B CN115623000 B CN 115623000B
Authority
CN
China
Prior art keywords
node
target
tree
nodes
failure
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211194855.3A
Other languages
Chinese (zh)
Other versions
CN115623000A (en
Inventor
黄罡
蔡华谦
李影
郭京申
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peking University
Original Assignee
Peking University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peking University filed Critical Peking University
Publication of CN115623000A publication Critical patent/CN115623000A/en
Application granted granted Critical
Publication of CN115623000B publication Critical patent/CN115623000B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00Data switching networks
    • H04L12/28Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
    • H04L12/44Star or tree networks

Abstract

The embodiment of the application provides a method and a device for efficiently distributing data on a digital network, which comprise the following steps: a planning algorithm is executed through a local client, and at least part of target nodes are selected from all other nodes of the digital network to construct a node topology set of a tree structure; generating a tree network planning file according to a transmission tree structure at a local client, packaging and uploading the tree network planning file and a target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a digital network by means of a transmission relation of the tree network; the tree-shaped transmission tree is established to distribute data to all the nodes of the digital network, so that the information transmission efficiency among the nodes of the digital network can be greatly improved.

Description

Method and device for efficiently distributing data on digital networking
Technical Field
The application relates to the field of digital networking, in particular to a method and a device for efficiently distributing data on the digital networking.
Background
The digital networking is a "virtual/data" network formed over the "physical/machine" internet by connecting together a large variety of heterogeneous data platforms and systems via a data-centric open software architecture and standardized interoperability protocols based on software definitions. Today, data interaction transmission between large-scale cluster servers mainly relies on server-Client (C/S) architecture networking.
The client is connected with the central server through the local area network, receives the user request, and makes a request to the server through the network to operate the database; the C/S architecture can fully exert the processing capability of the client computer, so that a plurality of works can be submitted to the server after the processing of the local equipment of the client is completed, and the running data load on the server is reduced.
However, in the prior art, as the operation requirement of the cluster service increases, the number of local devices for data processing increases; all rely on the data transmission mode of the C/S architecture networking, for the large-scale cross-cloud cross-region digital networking system, the file data transmission and running task distribution rate among all nodes is limited by the upper limit of the uploading bandwidth of the central server, and the data distribution efficiency is continuously reduced along with the continuous increase of the scale of the digital networking system.
Disclosure of Invention
The embodiment of the application provides a method, a device, electronic equipment and a computer readable storage medium for efficiently distributing data on a digital networking, which are used for solving the problems that in the related art, in the process of file data transmission and running task distribution between large-scale cross-cloud and cross-region digital networking system nodes, the distribution rate is limited by the upper limit of the uploading bandwidth of a central server, and the data transmission efficiency is low.
In a first aspect, an embodiment of the present application provides a data distribution method, applied to a client in a digital network, where the method includes:
acquiring node information of each node in the Internet of things, wherein the node information is used for representing attribute characteristics of the nodes;
according to the node information, selecting at least part of target nodes from all the nodes according to a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure;
generating a tree network planning file according to the node topology set, wherein the tree network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure;
and distributing the target data packet and the tree network planning file to the target node according to the identification of the target node.
In an alternative implementation, the method further includes:
performing failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node, and acquiring a connection response result;
judging the target node as a failure node under the condition that the connection response time of the target node is larger than a first threshold value;
Acquiring the identification of the failure node and the position information of the failure node in a tree structure according to the tree network planning file, and transmitting the identification and the position information of the failure node to a root node in the target node;
and executing node failure dynamic adjustment operation for the failure node.
In an optional implementation manner, the obtaining, according to the tree network planning file, the identifier of the failed node and the location information of the failed node in the tree structure, and sending the identifier of the failed node and the location information to a root node in the target node, includes:
according to the tree network planning file, node information of the failure node is sent to a father node of the failure node; the father node continues to transmit the node information of the failure node to the root node;
and after the root node receives the node information of the failure node, notifying the root node to mark the failure node as abnormal.
In an alternative implementation manner, the performing a node failure dynamic adjustment operation for the failed node includes:
re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a father node of the failure node so as to update the position of the descendant node in a tree structure;
And adjusting the transmission task of the descendant node to be executed in the father node.
In an alternative implementation, distributing the target data packet and the tree network planning file to the tree network transmission tree includes:
according to the identification of the root node in the tree structure, the tree network planning file and the target data packet are sent to the root node, so that the root node can construct the target nodes in the digital network into the tree structure through the tree network planning file, and the target data packet is distributed to each target node in the tree structure.
In an alternative implementation, the node information includes:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the digital network node; converting the node information into a standard data structure format through analysis processing operation; wherein the parsing operation includes: data structure storage of format check and key information of user input content; the composition of the standard data structure format includes: the node's internet protocol address, secure shell protocol user name, login password, and node description.
In an optional implementation manner, according to node information of the nodes, selecting at least part of target nodes from all the nodes according to a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure, including:
according to the standard data structure obtained by analyzing the node information, the geographic distance between the nodes is obtained by calculating the geographic position of the nodes;
selecting one target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
selecting a target node with the shortest transmission time from the digital network to other nodes in the current moment as a descendant node of the root node according to the standard data structure obtained by analyzing the node information, and adding the target node into a node topology set of the tree structure;
repeating the steps until the number of target nodes added to the node topology set of the tree structure reaches a target threshold value.
In an alternative implementation manner, before the node information according to the nodes selects at least part of target nodes from all the nodes according to a preset planning algorithm and constructs all the target nodes into a node topology set of a tree structure, the method further includes:
And according to the obtained node information, calling an analysis function to analyze and process the node information, and converting the node information into a standard data structure format.
In a second aspect, an embodiment of the present application provides a data distribution apparatus, which is applied to a client in a digital network, and includes:
the information acquisition module is used for acquiring node information of each node in the digital network, wherein the node information is used for representing attribute characteristics of the nodes;
the topology set construction module is used for selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure;
the planning file generation module is used for generating a tree network planning file according to the node topology set, and the tree network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure;
and the data distribution module is used for distributing the target data packet and the tree network planning file to the target node according to the identification of the target node.
In an alternative implementation, the apparatus further includes:
the node detection module is used for performing failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node, and acquiring a connection response result;
the result judging module is used for judging that the target node is a failure node under the condition that the connection response time of the target node is larger than a first threshold value;
the failure node information transmission module is used for acquiring the identification of the failure node and the position information of the failure node in the tree structure according to the tree network planning file, and sending the identification and the position information of the failure node to a root node in the target node;
and the dynamic adjustment module is used for executing node failure dynamic adjustment operation on the failed node.
In an alternative implementation manner, the failure node information transmission module includes:
the failure node information transmission sub-module is used for transmitting the node information of the failure node to the father node of the failure node according to the tree network planning file; the father node continues to transmit the node information of the failure node to the root node;
And the abnormal node marking sub-module is used for notifying the root node to mark the invalid node as abnormal after the root node receives the node information of the invalid node.
In an alternative implementation, the dynamic adjustment module includes:
the topology relation updating sub-module is used for re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topology relation of the descendant node to a father node of the failure node so as to update the position of the descendant node in a tree structure;
and the task adjustment sub-module is used for adjusting the transmission task of the descendant node to the father node for execution.
In an alternative implementation, the data distribution module includes:
and the data distribution sub-module is used for sending the tree network planning file and the target data packet to the root node according to the identification of the root node in the tree structure so that the root node can construct the target node in the digital network into the tree structure through the tree network planning file, and distributing the target data packet to each target node in the tree structure.
In an alternative implementation, the node information includes:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the digital network node; the parsing operation includes: data structure storage of format check and key information of user input content; the composition of the standard data structure format includes: the node's internet protocol address, secure shell protocol user name, login password, and node description.
In an alternative implementation, the topology set construction module includes:
the relative distance calculation sub-module is used for calculating the geographic distance between the nodes according to the standard data structure obtained by analyzing the node information and the geographic position of the nodes;
the optimal distance selecting sub-module is used for selecting one target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
the target node selection sub-module is used for selecting a target node with the shortest transmission time from the digital network to other nodes in the digital network at the current moment as a descendant node of the root node and adding the node into a node topology set of the tree structure according to the standard data structure obtained by analyzing the node information;
And the set construction circulation submodule is used for repeating the steps until the number of target nodes added to the node topology set of the tree structure reaches a target threshold value.
In an alternative implementation, before the topology assembly construction module starts to perform a function, the apparatus further includes:
and the node information analysis module is used for calling an analysis function to analyze and process the node information according to the acquired node information and converting the node information into a standard data structure format.
In the embodiment of the application, a planning algorithm is executed through a local client, partial nodes in the digital network are planned to generate a tree topology set based on node information of each node in the digital network system, and the set can realize the establishment of mesh connection among each node in the digital network to generate a tree network transmission tree; generating a tree network planning file according to a transmission tree structure through a local client, packaging and uploading the tree network planning file and a target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a digital networking system by means of a transmission relation of a tree network; the tree-shaped transmission tree is established to distribute data to all nodes in the digital networking system, so that the information transmission efficiency among the nodes is greatly improved.
The foregoing description is only an overview of the present application, and is intended to be implemented in accordance with the teachings of the present application in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present application more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a schematic step implementation flow chart of a data distribution method according to an embodiment of the present application;
fig. 2 is a diagram showing a comparison of a data distribution network structure of a digital network according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating specific steps of a data distribution method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a node failure dynamic processing step according to an embodiment of the present application;
fig. 5 is a relational block diagram of a data distribution device according to an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.
Fig. 1 is a schematic step flow diagram of a data distribution method according to an embodiment of the present application, which is applied to a client in the internet of things; as shown in fig. 1, the method may include:
step 101, obtaining node information of each node in the digital network, wherein the node information is used for representing attribute characteristics of the nodes.
In the step, firstly, a client acquires node information of each node to be processed in the digital network, wherein the node information is specifically used for representing attribute characteristics of the nodes; the attribute characteristics of each node can obtain the information such as the relative physical distance and the information transmission delay among the nodes, and the like, and the method has important significance in the subsequent generation process of the cluster tree transmission tree.
Specifically, an operation user or operation and maintenance personnel is connected to a digital networking system information base through a local client, and node information of all nodes to be operated in the whole digital networking is obtained according to a preset target; it is worth noting that in this process, all target nodes in the data network are peer-to-peer device nodes, and no physical links or transmission hierarchy exists.
Referring to fig. 2, fig. 2 is a diagram illustrating a comparison of network transmission relationships according to an embodiment of the present application; as shown in fig. 2, in the existing Server-Client (C/S) architecture, all Client host nodes 202 are linked to a central Server 201, and the data distribution speed is limited by the network upload bandwidth of the central Server 201; the application establishes a service cluster by means of a Peer-to-Peer computer network (P2P), the link transmission relation among the nodes is tree-like rhizome type mesh transmission, the Peer-to-Peer relation among the nodes is not a central server in the traditional sense; referring to fig. 2, all nodes in the P2P architecture can be divided into: a root node 203, an intermediate node 204, a leaf node 205; the root node 203 is used as a data input node of the whole transmission network, and is only responsible for file transmission to other nodes after receiving the data file, the leaf node 205 is only responsible for receiving the data file because of the absence of a lower link node, and the intermediate node 204 in the middle part is responsible for downloading and uploading the file.
Step 102, selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure.
After node information of all target nodes in the network to be operated is acquired, the client starts to execute a preset planning algorithm according to the acquired node information to generate a tree-shaped transmission network frame; according to the node attribute characteristics represented by the node information, the client side at least selects part of target nodes in the digital network to construct a node topology set of a tree structure; at least partial nodes in the digital networking are used for constructing a node topology set of a tree structure to generate a tree network transmission tree, and the basic idea of a planning algorithm is to perform the increment of the accepted instruction nodes at the fastest speed so as to enable the final total transmission time of the whole node cluster.
Specifically, the true geographic position of each node is obtained through the obtained node information, and the relative distance between each node is obtained through calculation; selecting one node with the shortest average distance from other nodes in the cluster as a root node through a clustering algorithm; the root node is used as an initial node of the whole tree topology set and is used as an access point of external data in a tree-like transmission network; when the root node is selected, continuing to obtain the transmission rate and network transmission delay between the nodes from the information such as the cloud service provider of the city service where the node is in, and the like in the obtained node information; according to the transmission parameter characteristics among the nodes, firstly selecting a node with the shortest current transmission ending time as a father node, and then selecting a node with the shortest distance from the rest nodes to be selected as a child node of the rest nodes to be added into a topology set; repeating the steps until the number of nodes selected in the topology set reaches a preset threshold; the large-scale cluster deployment and operation and maintenance system formed by the tree-like transmission network can be called as a Treeops system.
In the step, based on node information of nodes, gradually screening an optimal target node by using a topology algorithm, and adding the optimal target node into a topology set to generate a tree network transmission tree; constructing a topology set of an optimal structure through Torons cyclic screening based on the fast increment idea; the generation principle of the spanning tree is to achieve the technical effect that the transmission time of the whole data transmission working cluster is shortest, so that in the process, the information such as the geographic position, the transmission rate and the like contained in the node information is taken as an important basis for algorithm execution; the network transmission tree constructed by the method can achieve optimal transmission rate aiming at data distribution and transmission of multiple nodes in actual use, and improves data transmission efficiency in a large-scale analog-to-digital service network.
Step 103, generating a tree network planning file according to the node topology set, wherein the tree network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure.
After the client executes a planning algorithm to generate a tree-shaped transmission network, generating a tree-shaped network planning file according to the connection relation among the nodes and the position information of the single node in the whole transmission network; the tree network planning file includes the following information: the identity of each target node in the transmission tree, the node type of the target node in the tree structure, the interconnection relationship between each target node and the like.
The network planning file is used as a tree network transmission tree blueprint formed by the whole digital networking nodes, and the identity information corresponding to each node is recorded in detail, so that the identity of the node is accurately identified in the network on one hand; on the other hand, in the later data transmission process of the file, the position of the file in the current tree structure can be clearly identified among all nodes, so that data distribution and transfer can be accurately carried out; the upper and lower layer relation of the links between the nodes can be correctly identified, and the upper and lower layer relation is an important guarantee for ensuring correct transmission of data information.
And 104, distributing a target data packet and the tree network planning file to the target node according to the identification of the target node.
After the client generates the tree network planning file, the system can be started to realize the data distribution of the large-scale digital networking; the types of data that may be distributed for transmission include: an application source data packet, a functional instruction script file, an audio-video media file, an encrypted document and the like.
Specifically, the client firstly packages target data required to be transmitted by a user and a tree-shaped network planning file, then uploads the packaged target data and the tree-shaped network planning file to a root node of the tree-shaped transmission tree, the root node starts to send the target data and the tree-shaped network planning file to a first layer target node which directly establishes a connection relation with the root node, and then the target node subjected to the data confirms a lower layer target node object required to be transmitted by the client through analyzing the tree-shaped network planning file.
The tree-like network planning file contains detailed information of all aspects of the tree-like transmission structure of the whole cluster, which is equivalent to a blueprint of the transmission network structure, is an important basis of the whole cluster system in the data transmission process, and is different from the traditional S/C network transmission architecture in which all service nodes acquire data from a central server, other nodes except leaf nodes at the tail end of the transmission tree of the tree-like transmission network built in the embodiment of the application bear two responsibilities of data receiving and distributing simultaneously; for distributing this action, the service cluster as a whole will have more data upload capacity and is no longer limited by the limited upload bandwidth of the single-center server.
In summary, according to the data transmission method provided by the embodiment of the present application, a planning algorithm is executed by a local client, and a tree topology set is generated by planning a part of nodes in the digital network based on node information of each node in the digital network, where the set can realize that mesh connection is established between each node in the digital network, so as to generate a tree network transmission tree; generating a tree network planning file according to a transmission tree structure through a local client, packaging and uploading the tree network planning file and a target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a digital network by means of a transmission relation of the tree network; the tree-shaped transmission tree is established to distribute data to all nodes in the digital networking system, so that the information transmission efficiency among the nodes is greatly improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating specific steps of a data transmission method according to an embodiment of the present application; as shown in fig. 3, the steps specifically include:
step 301, obtaining node information of each node in the digital network, wherein the node information is used for representing attribute characteristics of the nodes.
This step may refer to step 101, and will not be described herein.
Optionally, step 301 may further include:
and sub-step 3011, calling an analysis function to analyze the node information according to the acquired node information, and converting the node information into a standard data structure format.
Wherein the node information includes: the method comprises the steps of connecting a host name, a login password, a geographic position of a city and a cloud provider of each host of a digital networking node; the parsing operation includes: data structure storage of format check and key information of user input content; the composition of the standard data structure format includes: the node's internet protocol address, secure shell protocol user name, login password, and node description.
Specifically, before the cluster task starts, firstly, the tree network planning module shown in fig. 3 reads the node information of the digital network input by the user, and the analysis function is called to analyze the input file containing the node information, so that the user input is converted into a standard data format (SSH Machine, secure Shell Machine) under the secure shell protocol, and the corresponding information is converted into a standard data structure table; the data types contained therein include: SSH user name, internet protocol address (IP, internet Protocol Address), SSH port, SSH password, city of node, node cloud service provider, node description, node intranet IP.
Notably, SSHMachine can provide SSH connectivity and instruction execution to class execution target nodes responsible for SSH connectivity; sshmachines are stored in list form in Virtual machines (VMInfo) and exist all the time when the program runs.
The node information input by the user is analyzed and standardized to generate a data structure table, so that the node information corresponding to the huge number of nodes in the digital network can be more uniformly and conveniently managed, and meanwhile, the node information is used as an important basis for standard input and algorithm execution in the generation of a later tree-shaped transmission tree.
Step 302, selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure.
This step may refer to step 102, and will not be described herein.
Optionally, step 302 may further include:
and step 3021, calculating a geographical distance between each two nodes according to the standard data structure obtained by analyzing the node information and the geographical position of the node.
In the standard SSHMachine structure data obtained by calling the analysis operation, the relative geographic position between the target nodes can be obtained by calling the information of cloud service providers of the target nodes, cities where the nodes are located and the like, so that the geographic distance between the nodes can be obtained by calculation.
The relative distance between the target nodes in the real environment can also have a certain influence on data transmission, and the data information transmitted by a long distance can be transferred by data of a plurality of switches, so that higher transmission delay is generated. In the application, the optimal transmission network generation strategy is selected by calculating the relative distance, so that unnecessary transmission delay between nodes can be removed as much as possible according to the basic attribute characteristics of the nodes.
A sub-step 3022 of selecting, by a clustering algorithm, a target node in the digital network as a root node; and the target node is the node with the shortest average distance to other nodes in the digital network.
The root node is used as an access point between the whole digital network and the outside world, and the average distance from the root node to each node in the cluster needs to be ensured to be the shortest to the greatest extent.
And step 3023, selecting a target node with the shortest transmission time from the digital network to other nodes in the digital network at the current moment as a descendant node of the root node according to the standard data structure obtained by analyzing the node information, and adding the target node into the node topology set of the tree structure.
The algorithm design idea of the transmission tree is that proliferation of the received instruction nodes is carried out at the fastest speed, so that the final total transmission time of the node cluster is as short as possible; after the root node is selected, the receiving sub-step 3022 selects the target node from the other nodes in the digital network by comparing the transmission time lengths in the tree in the current structure, adds the target node to the transmission tree, and circularly selects the transmission tree generated by the node with the shortest transmission time in multiple rounds, so that the transmission time required by the cluster is also shortest.
Specifically, there is one transmission end time for each node already in the tree. In this scenario, it is assumed that the transmission time from node to node should be fixed for the same transmission task; for example, at the present time, the transmission time from node a to node B, C, D is set to 1; when node B needs to add a child node E, it is equivalent to adding a transmission task, so the transmission end time of the node is +1. In a spanning tree, the overall required transmission time of the node cluster is equivalent to the longest transmission end time of the nodes in the spanning tree.
The selection of each round of nodes follows the principle of shortest transmission time, so that the transmission time required by the overall cluster can be ensured to be the shortest as well; an optimal transmission strategy is provided for optimal data transmission efficiency requirements of the digital networking nodes.
Sub-step 3024, repeating the above steps until the number of target nodes added to the node topology set of the tree structure reaches the target threshold.
Stopping when the number of the target nodes of the set reaches a target threshold.
Repeating the steps, and circularly screening and adding the nodes for a plurality of times until the number of target nodes absorbed by the transmission tree reaches the target requirement of a user; in this process, the number of target nodes included in the transmission tree may include all the nodes identified in the digital networking system, or may be only a part of the nodes in the digital networking system.
Step 303, generating a tree network planning file according to the node topology set, where the tree network planning file includes: the identification of the target node, the node type and the position information of the target node in the tree structure.
This step may refer to step 103, and will not be described herein.
And step 304, distributing a target data packet and the tree network planning file to the target node according to the identification of the target node.
This step may refer to step 104, and will not be described herein.
Optionally, step 304 may further include:
And sub-step 3041, according to the identifier of the root node in the tree structure, sending the tree network planning file and the target data packet to the root node, so that the root node builds the target node in the digital network into the tree structure through the tree network planning file, and distributes the target data packet to each target node in the tree structure.
And the client realizes data distribution on the large-scale digital networking through the system according to the generated tree network planning file.
Specifically, the client firstly packages target data required to be transmitted by a user and a tree-shaped network planning file, then uploads the packaged target data and the tree-shaped network planning file to a root node of the tree-shaped transmission tree, the root node starts to send the target data and the tree-shaped network planning file to a first layer target node which directly establishes a connection relation with the root node, and then the target node subjected to the data confirms a lower layer target node object required to be transmitted by the client through analyzing the tree-shaped network planning file. The tree-like network planning file contains detailed information of all aspects of the tree-like transmission structure of the whole cluster, which is equivalent to a blueprint of the transmission network structure, is an important basis of the whole cluster system in the data transmission process, and is different from the traditional S/C network transmission structure in which all service nodes acquire data from a central server.
Step 305, performing failure detection on the target node; wherein the failure detection comprises: and establishing connection with the target node, and acquiring a connection response result.
The introduction of a failure detection mechanism is a necessary option to perform dynamic adjustment. For an operation and maintenance system capable of managing the whole cluster, it is necessary to ensure that the whole cluster is in high availability as much as possible; meanwhile, the failure node should be monitored and reported in time, so that various distributed tasks depending on the cluster can be adjusted in time, and stable operation is ensured. In the prior art, failure detection judgment can be carried out on nodes in batches by sending heartbeat packets regularly, and certain tolerance can exist on the connection time of the nodes according to the situation due to factors such as network fluctuation; in the embodiment of the application, the state of the target node is judged by adopting a method for connecting the response time threshold.
Optionally, step 305 may further include:
and step 3051, judging according to the corresponding connection result, and judging that the target node is a failure node under the condition that the connection response time of the target node is larger than a first threshold value.
Specifically, after the Treeops client of the node in the cluster establishes connection to its child node through the secure shell protocol, if the corresponding node cannot be obtained within 200s after the connection request is initiated, the program will issue an abnormal warning, and identify that the node is in a failure state.
In the actual use situation, all nodes in the cluster should be kept in a communication state at any time so that the system can continuously run, and service requests from different users can be processed at any time; by periodically performing failure detection, the operation faults of each node in the cluster caused by possible hardware/software faults can be timely found, and then the target node in the failure state can be dynamically regulated in a targeted manner, so that the system can still work normally under the condition of losing the node.
And step 306, according to the corresponding connection result, sending the identification and the position information of the failure node.
When the child node is connected overtime, the system judges the node as a failure node, and performs dynamic adjustment for matching with the system, and firstly, node information of the failure node and other node conditions with topological connection relation with the failure node are required to be specified; when judging that the node fails and throws out the exception, the method directly enters a processing program of the node failure. Since the task of a child node in a spanning tree depends on the transfer of a parent node, once the parent node fails, the entire subtree thereof falls into failure; thus requiring the problem of handling failed nodes in steps.
Optionally, step 306 may further include:
sub-step 3061, according to the tree network planning file, sending the node information of the failure node to a father node of the failure node; and the father node continuously transmits the node information of the failure node to the root node.
Specifically, firstly, the information of the failure node is reported to the root node according to the path of the spanning tree, and because the tree network planning file records the specific topological connection relation of the transmission tree, each node can reversely transmit the information to the own upper parent node according to the transmission path.
Unlike the execution of data distribution, the transmission of the failure node information is single and ultimately only points to the root node of the entire tree-like transmission network.
Sub-step 3062, after the root node receives the node information of the failed node, notifying the root node to mark the failed node as abnormal.
The root node marks the failure node as an abnormal state according to the received failure node information and feeds the abnormal state back to the client; therefore, a user or operation and maintenance personnel can be informed of related operations such as overhauling the failure node.
And step 307, performing node failure dynamic adjustment operation on the failed node.
And dynamically adjusting the sub-node tasks of the failure node, and transmitting the sub-node tasks to other nodes for continuous execution. In the design of the embodiment of the application, all the child node transmission tasks of the failure node are dynamically adjusted to the parent node of the node, and the child node exists as a new child node of the parent node from the topology. If the root node fails, the subsequent task is not executed and is directly returned to wait for repair and re-execute. Therefore, the mechanism can ensure that all normal nodes except the failure node can execute corresponding tasks, and the overall usability of the system is ensured.
Optionally, step 307 may further include:
and 3071, re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a father node of the failure node so as to update the position of the descendant node in the tree structure.
For the failed node, the primary processing step is to transfer the connection relation of the descendant node to other normal nodes so as to ensure that the nodes can work normally; in the embodiment of the application, a method for re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file and connecting the topological relation of the descendant node to a father node of the failure node is adopted.
Specifically, referring to fig. 4, fig. 4 is a schematic diagram of a node failure dynamic processing step according to an embodiment of the present application; as shown in fig. 4, 401 is a reference diagram for distinguishing different types of nodes, in a block 402, a node No. 2 is judged as a failure node by a system, in a topological relation of the node No. 2, a parent node of an upper layer is a node No. 1, a node No. 3 and a node No. 4 are normal nodes in the same hierarchical relation with the node No. 2, offspring nodes of the node No. 2 comprise nodes No. 5, no. 6 and No. 8, and the node No. 8 is also a child node of the node No. 5, and at this time, the nodes No. 5, no. 6 and No. 8 are still in a normal state; and starting to execute a planning algorithm, and reapplying the connection relation of the descendant nodes of the No. 2 node, wherein the parent node of the No. 2 node is the No. 1 node in the tree relation, so that the topological relation of the 3 nodes is transferred to the No. 1 node, as shown in a block 403, and the effect of the transferred relation is shown in a block 404.
Sub-step 3072, adjusting the transmission task of the descendant node to be executed in the parent node.
As shown in fig. 4, after the dynamic adjustment and transfer, the nodes No. 5 and No. 6 are directly connected with the node No. 1, and the node No. 8 is still used as a child node of the node No. 5; therefore, when the cluster executes the next data transmission task, the target data packet and the tree network planning file are directly distributed to the No. 5 node and the No. 6 node by the No. 1 node, and even if the No. 2 node is invalid, the normal work of the No. 5 node, the No. 6 node and the No. 8 node is not influenced.
In summary, according to the data transmission method provided by the embodiment of the present application, a planning algorithm is executed by a local client, and a tree topology set is generated by planning a part of nodes in the digital network based on node information of each node in the digital network, where the set can realize that mesh connection is established between each node in the digital network, so as to generate a tree network transmission tree; generating a tree network planning file according to a transmission tree structure through a local client, packaging and uploading the tree network planning file and a target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a digital network by means of a transmission relation of the tree network; in addition, the client side carries out state detection on all nodes in the logarithmic networking at regular intervals, and enables a dynamic adjustment mechanism for the node with invalid detection result to transfer task content on the original node to other nodes for execution; the tree-shaped transmission tree is established to distribute data to all nodes in the digital networking system, so that the information transmission efficiency between the nodes can be greatly improved; meanwhile, a dynamic regulation mechanism is introduced to fill the gap of a fault remedy mechanism of the cluster system, so that the fault tolerance of the system is greatly improved.
Referring to fig. 5, fig. 5 is a relational block diagram of a data distribution device according to an embodiment of the present application, which is applied to a client in the digital network; as shown in fig. 5, the apparatus includes: an information acquisition module 501, a topology set construction module 502, a planning file generation module 503 and a data distribution module 504;
The information obtaining module 501 is configured to obtain node information of each node in the digital network, where the node information is used to characterize attribute features of the node;
the topology set construction module 502 is configured to select at least a part of target nodes from all the nodes according to the node information and a preset planning algorithm, and construct all the target nodes into a node topology set with a tree structure;
the planning file generating module 503 is configured to generate a tree network planning file according to the node topology set, where the tree network planning file includes: the identification of the target node, the node type and the position information of the target node in the tree structure;
the data distribution module 504 is configured to distribute, according to the identifier of the target node, a target data packet and the tree network planning file to the target node.
In an alternative implementation, the apparatus further includes:
the node detection module is used for performing failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node, and acquiring a connection response result;
the result judging module is used for judging that the target node is a failure node under the condition that the connection response time of the target node is larger than a first threshold value;
The failure node information transmission module is used for acquiring the identification of the failure node and the position information of the failure node in the tree structure according to the tree network planning file, and sending the identification and the position information of the failure node to a root node in the target node;
and the dynamic adjustment module is used for executing node failure dynamic adjustment operation on the failed node.
In an alternative implementation manner, the failure node information transmission module includes:
the failure node information transmission sub-module is used for transmitting the node information of the failure node to the father node of the failure node according to the tree network planning file; the father node continues to transmit the node information of the failure node to the root node;
and the abnormal node marking sub-module is used for notifying the root node to mark the invalid node as abnormal after the root node receives the node information of the invalid node.
In an alternative implementation, the dynamic adjustment module includes:
the topology relation updating sub-module is used for re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topology relation of the descendant node to a father node of the failure node so as to update the position of the descendant node in a tree structure;
And the task adjustment sub-module is used for adjusting the transmission task of the descendant node to the father node for execution.
In an alternative implementation, the data distribution module includes:
and the data distribution sub-module is used for sending the tree network planning file and the target data packet to the root node according to the identification of the root node in the tree structure so that the root node can construct the target node in the digital network into the tree structure through the tree network planning file, and distributing the target data packet to each target node in the tree structure.
In an alternative implementation, the node information includes:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the digital network node; the parsing operation includes: data structure storage of format check and key information of user input content; the composition of the standard data structure format includes: the node's internet protocol address, secure shell protocol user name, login password, and node description.
In an alternative implementation, the topology set construction module includes:
The relative distance calculation sub-module is used for calculating the geographic distance between the nodes according to the standard data structure obtained by analyzing the node information and the geographic position of the nodes;
the optimal distance selecting sub-module is used for selecting one target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
the target node selection sub-module is used for selecting a target node with the shortest transmission time from the digital network to other nodes in the digital network at the current moment as a descendant node of the root node and adding the node into a node topology set of the tree structure according to the standard data structure obtained by analyzing the node information;
and the set construction circulation submodule is used for repeating the steps until the number of target nodes added to the node topology set of the tree structure reaches a target threshold value.
In an alternative implementation, before the topology assembly construction module starts to perform a function, the apparatus further includes:
and the node information analysis module is used for calling an analysis function to analyze and process the node information according to the acquired node information and converting the node information into a standard data structure format.
In summary, according to the data transmission device provided by the embodiment of the present application, the local client executes the planning algorithm, and based on the node information of each node in the digital network, part of the nodes in the digital network are planned to generate the tree topology set, and the set can realize that mesh connection is established between each node in the digital network, so as to generate the tree network transmission tree; generating a tree network planning file according to a transmission tree structure through a local client, packaging and uploading the tree network planning file and a target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a digital network by means of a transmission relation of the tree network; in addition, the client side periodically detects states of all nodes in the cluster, and starts a dynamic adjustment mechanism for the nodes with invalid detection results to transfer task contents on the original nodes to other nodes for execution; the data distribution is carried out to all nodes in the cluster system by means of building a tree-shaped transmission tree, so that the information transmission efficiency between the nodes can be greatly improved; meanwhile, a dynamic regulation mechanism is introduced to fill the gap of a fault remedy mechanism of the cluster system, so that the fault tolerance of the system is greatly improved.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (5)

1. A data distribution method applied to a client in digital networking, the method comprising:
acquiring node information of each node in the Internet of things, wherein the node information is used for representing attribute characteristics of the nodes;
according to the node information, selecting at least part of target nodes from all the nodes according to a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure;
generating a tree network planning file according to the node topology set, wherein the tree network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure;
distributing a target data packet and the tree network planning file to the target node according to the identification of the target node, wherein the tree network planning file comprises the following components: according to the identification of a root node in the tree structure, the tree network planning file and the target data packet are sent to the root node so that the root node can construct the target node in the digital network into the tree structure through the tree network planning file, and the target data packet is distributed to each target node in the tree structure;
The method further comprises the steps of:
performing failure detection on the target node, including: when a Treeops client of a node in the cluster establishes connection to a child node of the node through a secure shell protocol, under the condition that a response cannot be obtained within 200s after a connection request is initiated, a program sends out an abnormal warning, and the node is determined to be in a failure state; wherein the failure detection comprises: establishing connection with the target node, and acquiring a connection response result;
judging the target node as a failure node under the condition that the connection response time of the target node is larger than a first threshold value;
acquiring the identification of the failure node and the position information of the failure node in a tree structure according to the tree network planning file, and transmitting the identification and the position information of the failure node to a root node in the target node;
and executing node failure dynamic adjustment operation on the failure node, wherein the node failure dynamic adjustment operation comprises the following steps: re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a father node of the failure node so as to update the position of the descendant node in a tree structure; and adjusting the transmission task of the descendant node to be executed in the father node.
2. The method according to claim 1, wherein the obtaining the identification of the failed node and the location information of the failed node in the tree structure according to the tree network planning file, and sending the identification of the failed node and the location information to the root node in the target node, comprises:
according to the tree network planning file, node information of the failure node is sent to a father node of the failure node; the father node continues to transmit the node information of the failure node to the root node;
and after the root node receives the node information of the failure node, notifying the root node to mark the failure node as abnormal.
3. The method of claim 1, wherein the node information comprises:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the digital network node;
converting the node information into a standard data structure format through analysis processing operation; wherein the parsing operation includes: data structure storage of format check and key information of user input content;
The composition of the standard data structure format includes: the node's internet protocol address, secure shell protocol user name, login password, and node description.
4. The method according to claim 1, wherein the selecting at least part of the target nodes from all the nodes according to the node information of the nodes and the preset planning algorithm, and constructing all the target nodes as a node topology set of a tree structure includes:
according to the standard data structure format obtained by analyzing the node information, the geographic distance between the nodes is obtained by calculating the geographic position of the nodes;
selecting one target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
and selecting the target node with the shortest transmission time from the digital network to other nodes in the current time as a descendant node of the root node according to the standard data structure obtained by analyzing the node information, adding the target node into the node topology set of the tree structure, and stopping until the number of the target nodes added into the node topology set of the tree structure reaches a target threshold.
5. The utility model provides a high-efficient distribution device of data on digital networking, is applied to digital networking software customer end, its characterized in that, the device includes:
the information acquisition module is used for acquiring node information of each node in the digital network, wherein the node information is used for representing attribute characteristics of the nodes;
the topology set construction module is used for selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set of a tree structure;
the planning file generation module is used for generating a tree network planning file according to the node topology set, and the tree network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure;
the data distribution module is used for distributing the target data packet and the tree network planning file to the target node according to the identification of the target node, and comprises the following steps: according to the identification of a root node in the tree structure, the tree network planning file and the target data packet are sent to the root node so that the root node can construct the target node in the digital network into the tree structure through the tree network planning file, and the target data packet is distributed to each target node in the tree structure;
The apparatus further comprises:
the node detection module is configured to perform failure detection on the target node, and includes: when a Treeops client of a node in the cluster establishes connection to a child node of the node through a secure shell protocol, under the condition that a response cannot be obtained within 200s after a connection request is initiated, a program sends out an abnormal warning, and the node is determined to be in a failure state; wherein the failure detection comprises: establishing connection with the target node, and acquiring a connection response result;
the result judging module is used for judging that the target node is a failure node under the condition that the connection response time of the target node is larger than a first threshold value;
the failure node information transmission module is used for acquiring the identification of the failure node and the position information of the failure node in the tree structure according to the tree network planning file, and sending the identification and the position information of the failure node to a root node in the target node;
the dynamic adjustment module is used for executing node failure dynamic adjustment operation on the failure node, and comprises the following steps: re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a father node of the failure node so as to update the position of the descendant node in a tree structure; and adjusting the transmission task of the descendant node to be executed in the father node.
CN202211194855.3A 2022-09-23 2022-09-29 Method and device for efficiently distributing data on digital networking Active CN115623000B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211164068 2022-09-23
CN2022111640684 2022-09-23

Publications (2)

Publication Number Publication Date
CN115623000A CN115623000A (en) 2023-01-17
CN115623000B true CN115623000B (en) 2023-11-14

Family

ID=84860085

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211194855.3A Active CN115623000B (en) 2022-09-23 2022-09-29 Method and device for efficiently distributing data on digital networking

Country Status (1)

Country Link
CN (1) CN115623000B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701410B (en) * 2023-08-07 2023-11-14 北京大学 Method and system for storing memory state data for data language of digital networking
CN116932614B (en) * 2023-09-18 2023-12-26 北京大学 Distributed searching method and system oriented to digital networking

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105306417A (en) * 2014-06-25 2016-02-03 深圳市华夏双赢通信有限公司 Network data distribution method based on super router and network system
CN110519365A (en) * 2019-08-26 2019-11-29 网宿科技股份有限公司 A kind of method and business change system changing appliance services
CN110597598A (en) * 2019-09-16 2019-12-20 电子科技大学广东电子信息工程研究院 Control method for virtual machine migration in cloud environment
US10614261B1 (en) * 2019-02-05 2020-04-07 Cadence Design Systems, Inc. Honoring pin insertion delay during clock tree synthesis
CN112769580A (en) * 2020-12-31 2021-05-07 阳江市链点创新科技发展有限公司 Block chain layered excitation consensus algorithm
CN114760135A (en) * 2022-04-19 2022-07-15 浙江大学 Optimization method of block chain fault-tolerant consensus scheme

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220210200A1 (en) * 2015-10-28 2022-06-30 Qomplx, Inc. Ai-driven defensive cybersecurity strategy analysis and recommendation system
US11281862B2 (en) * 2019-05-03 2022-03-22 Sap Se Significant correlation framework for command translation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105306417A (en) * 2014-06-25 2016-02-03 深圳市华夏双赢通信有限公司 Network data distribution method based on super router and network system
US10614261B1 (en) * 2019-02-05 2020-04-07 Cadence Design Systems, Inc. Honoring pin insertion delay during clock tree synthesis
CN110519365A (en) * 2019-08-26 2019-11-29 网宿科技股份有限公司 A kind of method and business change system changing appliance services
CN110597598A (en) * 2019-09-16 2019-12-20 电子科技大学广东电子信息工程研究院 Control method for virtual machine migration in cloud environment
CN112769580A (en) * 2020-12-31 2021-05-07 阳江市链点创新科技发展有限公司 Block chain layered excitation consensus algorithm
CN114760135A (en) * 2022-04-19 2022-07-15 浙江大学 Optimization method of block chain fault-tolerant consensus scheme

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Classification of SSH Attacks using Machine Learning Algorithms;Gokul Kannan Sadasivam;IEEE;全文 *
一种基于测量的启发式网络拓扑匹配优化算法;廖怡;盛益强;王劲林;;计算机学报(09);全文 *
中央核心处理式多层级流量管理系统网络结构设计;张明伟;王匀;;指挥信息系统与技术(03);全文 *

Also Published As

Publication number Publication date
CN115623000A (en) 2023-01-17

Similar Documents

Publication Publication Date Title
CN115623000B (en) Method and device for efficiently distributing data on digital networking
KR100791429B1 (en) Network attached storage snmp single system image
JP3966598B2 (en) Server selection system
US6253243B1 (en) Automated trap control for a distributed network management system
CN110209492B (en) Data processing method and device
CN102447585B (en) Method and device for converting network configuration protocol response message into command line
US20020032769A1 (en) Network management method and system
US20050047350A1 (en) Apparatus and methods for discovery of network elements in a network
CN107172176B (en) APP method for connecting network, equipment and configuration server based on configuration management
US8667113B2 (en) Bitmap network management object change detection approach
JP6539341B2 (en) Providing router information according to the programmatic interface
CN105323085A (en) Processing method and device for establishing performance measurement task and performance measurement result
CN102387028A (en) Network system, network management server, and OAM test method
CN108540505B (en) Content updating method and device
WO2020010906A1 (en) Method and device for operating system (os) batch installation, and network device
US20090150564A1 (en) Per-user bandwidth availability
CN110740355A (en) Equipment monitoring method and device, electronic equipment and storage medium
CN102845023B (en) For registering the apparatus and method of transmission capacity in broadband access network
US20130159509A1 (en) Method and system for controlling data communication within a network
US20230267326A1 (en) Machine Learning Model Management Method and Apparatus, and System
JP2016144186A (en) Communication information controller, relay system, communication information control method, and communication information control program
US10447807B1 (en) Dynamic middleware source selection for optimizing data retrieval from network nodes
CN107493308B (en) Method and device for sending message and distributed equipment cluster system
CN104717176A (en) Access control method, access control system, and server
CN109947451B (en) Cluster application file updating method, system, medium and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant