CN115623000A - Method and device for efficiently distributing data on digital network - Google Patents

Method and device for efficiently distributing data on digital network Download PDF

Info

Publication number
CN115623000A
CN115623000A CN202211194855.3A CN202211194855A CN115623000A CN 115623000 A CN115623000 A CN 115623000A CN 202211194855 A CN202211194855 A CN 202211194855A CN 115623000 A CN115623000 A CN 115623000A
Authority
CN
China
Prior art keywords
node
tree
target
nodes
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211194855.3A
Other languages
Chinese (zh)
Other versions
CN115623000B (en
Inventor
黄罡
蔡华谦
李影
郭京申
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peking University
Original Assignee
Peking University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peking University filed Critical Peking University
Publication of CN115623000A publication Critical patent/CN115623000A/en
Application granted granted Critical
Publication of CN115623000B publication Critical patent/CN115623000B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00Data switching networks
    • H04L12/28Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
    • H04L12/44Star or tree networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The embodiment of the application provides a method and a device for efficiently distributing data on a digital network, wherein the method comprises the following steps: executing a planning algorithm through a local client, and selecting at least part of target nodes from all other nodes of the network to construct a node topology set with a tree structure; at a local client, generating a tree-shaped network planning file according to a transmission tree structure, packaging the tree-shaped network planning file and a target data file to be distributed, uploading the tree-shaped network planning file and the target data file to a transmission tree root node, and distributing the data packet to all nodes in a digital network by means of the transmission relation of the tree-shaped network; data distribution is carried out to all nodes of the data networking by establishing the tree transmission tree, so that the information transmission efficiency among the nodes of the data networking can be greatly improved.

Description

Method and device for efficiently distributing data on digital network
Technical Field
The present application relates to the field of digital networking, and in particular, to a method and an apparatus for efficiently distributing data on a digital networking.
Background
Data networking is a "virtual/data" network formed over the "physical/machine" internet by connecting large-scale, heterogeneous data platforms and systems through an open software architecture and standardized interoperability protocols that are data-centric, based on software definitions. Nowadays, data interactive transmission between large-scale cluster servers mainly depends on server-Client (C/S) type architecture networking.
The client is connected with the central server through the local area network, receives a user request, makes a request to the server through the network and operates the database; the C/S framework can fully exert the processing capacity of the client computer, so that a plurality of jobs can be submitted to the server after the processing of the local equipment of the client is finished, and the load of running data of the server is reduced.
However, in the prior art, as the demand for cluster service operation increases, the number of local devices for data processing also increases; the data transmission mode of networking is totally dependent on a C/S architecture, for a large-scale cross-cloud and cross-region data networking system, the file data transmission and operation task distribution rate among all nodes is limited by the upper limit of the uploading bandwidth of a central server, and the data distribution efficiency is continuously reduced along with the continuous increase of the scale of the data networking system.
Disclosure of Invention
The embodiment of the application provides a method and a device for efficiently distributing data on a digital network, electronic equipment and a computer readable storage medium, and aims to solve the problems that in the process of file data transmission and operation task distribution between nodes of a large-scale cross-cloud and cross-region digital network system in the related technology, the distribution rate is limited by the upper limit of the uploading bandwidth of a central server, and the data transmission efficiency is low.
In a first aspect, an embodiment of the present application provides a data distribution method, which is applied to a client in a data networking system, and the method includes:
acquiring node information of each node in the data networking, wherein the node information is used for representing attribute characteristics of the node;
selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set with a tree structure;
generating a tree-shaped network planning file according to the node topology set, wherein the tree-shaped network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure;
and distributing a target data packet and the tree-shaped network planning file to the target node according to the identification of the target node.
In an alternative implementation, the method further comprises:
carrying out failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node and acquiring a connection response result;
under the condition that the connection response time of the target node is greater than a first threshold value, judging that the target node is a failure node;
acquiring the identifier of the failed node and the position information of the failed node in a tree structure according to the tree network planning file, and sending the identifier and the position information of the failed node to a root node in the target node;
and executing node failure dynamic adjustment operation aiming at the failed node.
In an optional implementation manner, the obtaining, according to the tree-like network planning file, the identifier of the failed node and the location information of the failed node in the tree structure, and sending the identifier of the failed node and the location information to the root node in the target node includes:
sending the node information of the failure node to a father node of the failure node according to the tree-shaped network planning file; the father node transmits the node information of the failed node to a root node continuously;
and after the root node receives the node information of the failure node, informing the root node to mark the failure node as abnormal.
In an optional implementation, the performing a node failure dynamic adjustment operation for the failed node includes:
according to the tree-shaped network planning file, re-executing a preset planning algorithm on at least one descendant node of the failure node, and connecting the topological relation of the descendant node to the father node of the failure node, so as to update the position of the descendant node in the tree-shaped structure;
and adjusting the transmission task of the descendant node to the parent node for execution.
In an optional implementation, the distributing the target data packet and the tree network planning file to the tree network transmission tree includes:
and sending the tree-like network planning file and the target data packet to the root node according to the identifier of the root node in the tree-like structure, so that the root node constructs the target nodes in the data network into a tree-like structure through the tree-like network planning file, and distributes the target data packet to each target node in the tree-like structure.
In an optional implementation manner, the node information includes:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the plurality of networking nodes; converting the node information into a standard data structure format through analysis processing operation; wherein the parsing processing operation comprises: checking the format of the content input by the user and storing the data structure of the key information; the standard data structure format comprises the following components: the internet protocol address, the security shell protocol user name, the login password and the node description of the node.
In an optional implementation manner, the selecting at least part of target nodes from all the nodes according to node information of the nodes and a preset planning algorithm, and constructing all the target nodes as a node topology set of a tree structure includes:
calculating the geographical distance between each node according to the standard data structure obtained by analyzing the node information and the geographical position of each node;
selecting a target node in the data network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
selecting a target node with the shortest transmission time from the node in the digital network at the current moment as a descendant node of the root node according to the standard data structure obtained by analyzing the node information, and adding the target node into the node topology set of the tree structure;
and repeating the steps until the number of the target nodes added into the node topology set of the tree structure reaches a target threshold value.
In an optional implementation manner, before the selecting at least a part of target nodes from all the nodes according to the node information of the nodes and a preset planning algorithm, and constructing all the target nodes as a node topology set of a tree structure, the method further includes:
and calling an analysis function to carry out analysis processing operation on the node information according to the acquired node information, and converting the node information into a standard data structure format.
In a second aspect, an embodiment of the present application provides a data distribution apparatus, which is applied to a client in a data networking, where the apparatus includes:
the information acquisition module is used for acquiring node information of each node in the data network, and the node information is used for representing attribute characteristics of the node;
the topology set construction module is used for selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set with a tree structure;
a planning file generation module, configured to generate a tree-like network planning file according to the node topology set, where the tree-like network planning file includes: the identification of the target node, the node type and the position information of the target node in the tree structure;
and the data distribution module is used for distributing the target data packet and the tree-shaped network planning file to the target node according to the identification of the target node.
In an alternative implementation, the apparatus further includes:
the node detection module is used for carrying out failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node and acquiring a connection response result;
the result judging module is used for judging that the target node is a failure node under the condition that the connection response time of the target node is greater than a first threshold value;
the invalid node information transmission module is used for acquiring the identifier of the invalid node and the position information of the invalid node in a tree structure according to the tree network planning file and sending the identifier and the position information of the invalid node to a root node in the target node;
and the dynamic adjusting module is used for executing node failure dynamic adjusting operation aiming at the failed node.
In an optional implementation manner, the failed node information transmission module includes:
the invalid node information transmission submodule is used for sending the node information of the invalid node to a father node of the invalid node according to the tree-shaped network planning file; the father node transmits the node information of the failed node to a root node continuously;
and the abnormal node marking submodule is used for informing the root node of marking the failed node as abnormal after the root node receives the node information of the failed node.
In an optional implementation manner, the dynamic adjustment module includes:
the topological relation updating submodule is used for re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a father node of the failure node, so that the position of the descendant node in the tree structure is updated;
and the task adjusting submodule is used for adjusting the transmission task of the descendant node to the father node for execution.
In an optional implementation manner, the data distribution module includes:
and the data distribution submodule is used for sending the tree-shaped network planning file and the target data packet to the root node according to the identifier of the root node in the tree-shaped structure, so that the root node constructs the target nodes in the data network into a tree-shaped structure through the tree-shaped network planning file, and distributes the target data packet to each target node in the tree-shaped structure.
In an optional implementation manner, the node information includes:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the plurality of networking nodes; the parsing processing operation includes: checking the format of the content input by the user and storing the data structure of the key information; the standard data structure format comprises the following components: the internet protocol address, the security shell protocol user name, the login password and the node description of the node.
In an optional implementation manner, the topology set building module includes:
the relative distance calculation submodule is used for calculating the geographical distance between the nodes according to the geographical position of the nodes according to the standard data structure obtained by analyzing the node information;
the optimal distance selection submodule is used for selecting a target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
the target node selection submodule is used for selecting a target node with the shortest transmission time from the digital network to other nodes at the current moment as a descendant node of the root node according to the standard data structure obtained by analyzing the node information and adding the descendant node into the node topology set of the tree structure;
and the set construction circulation submodule is used for repeating the steps until the number of the target nodes added into the node topology set of the tree structure reaches a target threshold value.
In an optional implementation, before the topology set building module starts to perform the function, the apparatus further includes:
and the node information analysis module is used for calling an analysis function to carry out analysis processing operation on the node information according to the acquired node information and converting the node information into a standard data structure format.
In the embodiment of the application, a local client executes a planning algorithm, and part of nodes in the digital network are planned to generate a tree topology set based on node information of each node in the digital network system, wherein the set can realize the establishment of mesh connection among the nodes in the digital network and generate a tree network transmission tree; generating a tree-shaped network planning file according to a transmission tree structure through a local client, packaging the tree-shaped network planning file and a target data file to be distributed, uploading the tree-shaped network planning file and the target data file to a transmission tree root node, and distributing the data packet to all nodes in a digital network system by means of the transmission relation of a tree-shaped network; data distribution is carried out to all nodes in the digital network system by establishing a tree transmission tree, so that the information transmission efficiency among the nodes is greatly improved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
Various additional advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a flow chart of an implementation of brief steps of a data distribution method provided in an embodiment of the present application;
FIG. 2 is a diagram illustrating a network architecture for data distribution in a network according to an embodiment of the present application;
fig. 3 is a flowchart illustrating specific steps of a data distribution method according to an embodiment of the present application;
fig. 4 is a schematic diagram of a node failure dynamic processing step according to an embodiment of the present application;
fig. 5 is a relational block diagram of a data distribution apparatus according to an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a flowchart of brief steps of a data distribution method provided in an embodiment of the present application, and is applied to a client in a data networking; as shown in fig. 1, the method may include:
step 101, obtaining node information of each node in the digital network, wherein the node information is used for representing attribute characteristics of the node.
In the step, firstly, a client acquires node information of each node to be processed in the data networking, wherein the node information is specifically used for representing attribute characteristics of the node; the information such as the relative physical distance between the nodes, the information transmission delay and the like can be obtained according to the attribute characteristics of the nodes, and the method has important significance for the subsequent generation process of the cluster tree transmission tree.
Specifically, an operation user or operation and maintenance personnel is connected to the information base of the digital networking system through a local client, and starts to acquire node information of all nodes to be operated in the whole digital networking system according to a preset target; it is worth to be noted that, in this process, all target nodes in the digital internetwork are peer-to-peer device nodes, and there is no up-down relationship between physical links or transmission layers.
Referring to fig. 2, fig. 2 is a diagram illustrating a network transmission relationship structure according to an embodiment of the present application; as shown in fig. 2, in a conventional Server-Client (C/S) architecture, all Client host nodes 202 are linked to a central Server 201, and a data distribution speed is limited by a network upload bandwidth of the central Server 201; the method establishes the service cluster by depending on a Peer-to-Peer computer network (P2P, peer-to-Peer), the link transmission relationship among all nodes is tree-like rhizome mesh transmission, the Peer-to-Peer relationship among all nodes exists, and a central server in the traditional sense does not exist; referring to fig. 2, all nodes in the P2P architecture may be divided into: root node 203, intermediate node 204, leaf node 205; the root node 203 is used as a data input node of the whole transmission network, and is only responsible for file transmission to other nodes after receiving the data file, the leaf node 205 is only responsible for receiving the data file because no lower link node exists, and the intermediate node 204 in the middle part is responsible for downloading and uploading the file at the same time.
And 102, selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set with a tree structure.
After the node information of all target nodes in the to-be-operated networking is acquired, the client starts to execute a preset planning algorithm according to the acquired node information to generate a tree-shaped transmission network framework; according to the node attribute characteristics represented by the node information, the client side at least selects part of target nodes in the data networking to construct a node topology set of a tree structure; the basic idea of the planning algorithm is to perform value addition of the nodes which have received instructions at the highest speed so as to enable the final total transmission time of the whole node cluster.
Specifically, the real geographical position of each node is obtained through the obtained node information, and the relative distance between each node is obtained through calculation; selecting a node with the shortest average distance to other nodes in the cluster as a root node through a clustering algorithm; the root node is used as an initial node of the whole tree topology set and is used as an access point of external data in the tree transmission network; after the root node is selected, the transmission rate and the network transmission delay among all the nodes are obtained by continuing to obtain information such as a cloud service provider of the urban service where the node is located in the obtained node information; according to the transmission parameter characteristics among the nodes, firstly, selecting a node with the shortest current transmission ending time as a father node, and then selecting a node with the shortest distance from the node to be selected from the rest nodes to be selected as a child node of the node to be selected to be added into the topology set; continuously repeating the step until the number of the selected nodes in the topology set reaches a preset threshold value; the large-scale cluster deployment and operation and maintenance system formed by the tree transmission network can be called a Treeops system.
In the step, based on the node information of the node, gradually screening the optimal target node by using a topological algorithm and adding the optimal target node into a topological set to generate a tree-shaped network transmission tree; constructing a topology set with an optimal structure through Torontal loop screening based on the idea of rapid value addition; the generation principle of the spanning tree is to achieve the technical effect that the transmission time of the whole data transmission working cluster is shortest, so that in the process, information such as the geographic position, the transmission rate and the like contained in the node information is used as an important basis for algorithm execution; the network transmission tree constructed by the method can achieve the optimal transmission rate aiming at the data distribution and transmission of multiple nodes in actual use, and improves the data transmission efficiency in a large-scale data-networking service network.
Step 103, generating a tree-shaped network planning file according to the node topology set, wherein the tree-shaped network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure.
After the client executes a planning algorithm to generate a tree-shaped transmission network, generating a tree-shaped network planning file according to the connection relation among nodes and the position information of a single node in the whole transmission network; the tree network planning file comprises the following information: the identity of each target node in the transmission tree, the node type of the target node in the tree structure, the interconnection relationship among each target node and the like.
The network planning file is used as a tree-shaped network transmission tree blueprint formed by the whole number of networking nodes, and the identity information corresponding to each node is recorded in detail, so that on one hand, the network planning file is used for correctly identifying the identity of the node in the network; on the other hand, in the later data transmission process of the file, the position of each node in the current tree structure can be clearly identified, so that data distribution and transfer are correctly carried out; the method can correctly identify the upper and lower layer relations of the links among the nodes, and is an important guarantee for ensuring correct transmission of data information.
And 104, distributing a target data packet and the tree-shaped network planning file to the target node according to the identifier of the target node.
After the client generates the tree-shaped network planning file, the system can be started to realize the data distribution of the large-scale data networking; the types of data that can be distributed for transmission include: application source packets, functional instruction script files, audio and video media files, encrypted documents, and the like.
Specifically, a client packages target data and a tree-shaped network planning file which are designated by a user to be transmitted and uploads the target data and the tree-shaped network planning file to a root node of a tree-shaped transmission tree at the same time, the root node starts to send the target data and the tree-shaped network planning file to a first layer target node which directly establishes a connection relation with the root node at the same time, and then the target node which receives the data analyzes the tree-shaped network planning file to determine a lower layer target node object which needs to be transmitted.
The tree-shaped network planning file contains all information of the whole cluster tree-shaped transmission structure in detail, is equivalent to a blueprint of a transmission network structure, is an important basis of the whole cluster system in the data transmission process, is different from the situation that all service nodes in the traditional S/C network transmission architecture acquire data from a central server, and except leaf nodes at the tail end of a transmission tree, other nodes of the tree-shaped transmission network established in the embodiment of the application all bear two responsibilities of receiving and distributing the data; for distributing the action, the service cluster as a whole has more data uploading amount and is not limited by the limited uploading bandwidth of the single central server.
To sum up, in the data transmission method provided in the embodiment of the present application, a local client executes a planning algorithm, and plans a part of nodes in a data networking to generate a tree topology set based on node information of each node in the data networking, where the set can realize establishment of mesh connection among the nodes in the data networking and generate a tree network transmission tree; generating a tree-shaped network planning file according to a transmission tree structure through a local client, packaging the tree-shaped network planning file and a target data file to be distributed, uploading the tree-shaped network planning file and the target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a data network by means of the transmission relation of the tree-shaped network; data distribution is carried out to all nodes in the digital network system by establishing a tree transmission tree, so that the information transmission efficiency among the nodes is greatly improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating specific steps of a data transmission method according to an embodiment of the present application; as shown in fig. 3, the steps specifically include:
step 301, obtaining node information of each node in the digital network, wherein the node information is used for representing attribute characteristics of the node.
This step may specifically refer to step 101, which is not described herein again.
Optionally, step 301 may further include:
and a substep 3011, calling an analysis function to perform analysis processing operation on the node information according to the obtained node information, and converting the node information into a standard data structure format.
Wherein the node information includes: the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the plurality of networking nodes; the parsing processing operation includes: checking the format of the user input content and storing the data structure of the key information; the standard data structure format consists of: the internet protocol address, the security shell protocol user name, the login password and the node description of the node.
Specifically, before the cluster task starts, the tree network planning module shown in fig. 3 reads the data networking node information input by the user, and analyzes the input file containing the node information by calling an analysis function, so as to convert the user input into a standard data format (SSH Machine) under a Secure Shell protocol and convert the corresponding information into a standard data structure table; the types of data contained therein include: an SSH user name, an Internet Protocol Address (IP), an SSH port, an SSH password, a node city, a node cloud service provider, a node description and a node intranet IP.
It is worth to be noted that SSHMachine can provide SSH connection and instruction execution of the class execution target node responsible for SSH connection; the SSHMachine is stored in a list in a Virtual Machine (VMInfo) class and always exists when the program runs.
The node information input by the user is analyzed and standardized to generate a data structure table, so that the node information corresponding to the huge number of nodes in the digital network can be managed more uniformly and conveniently, and meanwhile, the node information is used as an important basis for standard input and algorithm execution in the generation of a later-stage tree transmission tree.
And 302, selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set with a tree structure.
This step may specifically refer to step 102, which is not described herein again.
Optionally, step 302 may further include:
and a substep 3021, calculating the geographical distance between the nodes according to the geographical position of the node according to the standard data structure obtained by analyzing the node information.
In the standard SSHMachine structure data obtained by calling the analysis operation, the relative geographic positions of the target nodes can be obtained according to information such as cloud service providers of the target nodes, the cities where the nodes are located and the like, and therefore the geographic distances among the nodes can be obtained through calculation.
The relative distance between the target nodes in the real environment will also have a certain influence on data transmission, and the data information transmitted over a long distance may have multiple data transfers of the switch, which will generate higher transmission delay. In the method, the optimal transmission network generation strategy is selected by calculating the relative distance, so that unnecessary transmission time delay between the nodes can be removed as much as possible according to the basic attribute characteristics of the nodes.
Substep 3022, selecting a target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network.
The root node is used as an access point of the whole number networking and the outside, and the average distance from the root node to each node in the cluster needs to be ensured to be the shortest to the greatest extent.
And a substep 3023, similarly selecting a target node with the shortest transmission time to other nodes in the digital network at the current time as a descendant node of the root node according to the standard data structure obtained by analyzing the node information, and adding the target node to the node topology set of the tree structure.
The algorithm design idea of the transmission tree is that the proliferation of the received instruction nodes is carried out at the highest speed, so that the final total transmission time of the node cluster is reduced as much as possible; in the receiving substep 3022, after the root node is selected, a target node is selected for the other nodes in the data network by comparing the transmission time in the tree in the current structure and added to the transmission tree, and the transmission tree generated by the node with the shortest transmission time is selected through multiple cycles, so that the total transmission time required by the cluster is also the shortest.
Specifically, there is one transmission end time for each node already in the tree. In this scenario, it is assumed that the node-to-node transmission time should be fixed for the same transmission task; for example, at the current time, the transmission time from node a to node B, C, D is set to 1; when the node B needs to add a child node E, it is equivalent to add a transmission task, so the transmission end time of the node is +1. In a spanning tree, the total transmission time required by the node cluster is equal to the longest transmission end time of the nodes in the spanning tree.
Each round of node selection follows the principle of shortest transmission time, so that the transmission time required by the cluster as a whole can be ensured to be also shortest; optimal transmission strategies are provided for optimal data transmission efficiency requirements of several networked nodes.
And a substep 3024 of repeating the above steps until the number of target nodes added to the node topology set of the tree structure reaches a target threshold.
And stopping when the number of the target nodes of the set reaches a target threshold value.
Repeating the steps, and stopping the cyclic screening and adding of the nodes for multiple times until the number of the target nodes absorbed by the transmission tree reaches the target requirement of the user; in this process, the number of target nodes included in the transmission tree may include all the nodes identified in the data networking system, or may be only a part of all the nodes in the data networking system.
Step 303, generating a tree-shaped network planning file according to the node topology set, where the tree-shaped network planning file includes: the identification of the target node, the node type and the position information of the target node in the tree structure.
This step may specifically refer to step 103, which is not described herein again.
And 304, distributing a target data packet and the tree-shaped network planning file to the target node according to the identifier of the target node.
This step may specifically refer to step 104, which is not described herein again.
Optionally, step 304 may further include:
substep 3041, according to the identifier of the root node in the tree structure, sending the tree network planning file and the target data packet to the root node, so that the root node constructs the target nodes in the data network into a tree structure through the tree network planning file, and distributes the target data packet to each target node in the tree structure.
And the client side realizes data distribution on the large-scale data networking through the system according to the generated tree-shaped network planning file.
Specifically, a client packages target data and a tree-shaped network planning file which are designated by a user to be transmitted and uploads the target data and the tree-shaped network planning file to a root node of a tree-shaped transmission tree at the same time, the root node starts to send the target data and the tree-shaped network planning file to a first layer target node which directly establishes a connection relation with the root node at the same time, and then the target node which receives the data analyzes the tree-shaped network planning file to determine a lower layer target node object which needs to be transmitted. The tree-shaped network planning file contains all information of the whole cluster tree-shaped transmission structure in detail, is equivalent to a blueprint of a transmission network structure, is an important basis of the whole cluster system in the data transmission process, is different from the situation that all service nodes in the traditional S/C network transmission structure acquire data from a central server, except leaf nodes at the tail end of a transmission tree, other nodes of the tree-shaped transmission network established in the embodiment of the application all bear two responsibilities of receiving and distributing the data, and for the action of distributing, the whole data networking body has more data uploading amount and is not limited by the limited uploading bandwidth of a single central server.
Step 305, performing failure detection on the target node; wherein the failure detection comprises: and establishing connection with the target node and acquiring a connection response result.
The introduction of the failure detection mechanism is a necessary option to perform dynamic adjustment. For an operation and maintenance system capable of managing the whole cluster, it is necessary to ensure that the whole cluster is in high availability as much as possible; meanwhile, the failure nodes should be monitored and reported in time, so that various distributed tasks depending on the cluster can be adjusted in time, and stable operation is ensured. In the prior art, failure detection and judgment can be performed on nodes in batches by methods such as Ping and the like by periodically sending heartbeat packets, and due to the existence of factors such as network fluctuation, certain tolerance can exist on the connection time of the nodes according to conditions; in the embodiment of the application, the state of the target node is judged by adopting a method of connecting a response time threshold.
Optionally, step 305 may further include:
and a substep 3051, determining according to a corresponding connection result, and determining that the target node is a failed node when the connection response time of the target node is greater than a first threshold.
Specifically, after the Treeops client of the node in the cluster establishes connection to its child node through the secure shell protocol, if no correspondence is obtained within 200s after the connection request is initiated, the program will issue an exception warning to determine that the node is in a failure state.
In an actual use situation, all nodes in the cluster should keep a communication state all the time so that the system can continuously operate and respond to service requests from different users at any time; and by periodically carrying out failure detection, the operation failure of each node in the cluster caused by the possible hardware/software failure can be timely found, and then the target node in the failure state can be pertinently and dynamically adjusted, so that the system can still normally work under the condition of losing the node.
And step 306, sending the identification and the position information of the failure node according to the corresponding connection result.
When the connection of the child node is overtime, the system judges that the node is a failure node, and firstly, the node information of the failure node and the conditions of other nodes having topological connection relation with the failure node need to be detailed for the dynamic adjustment of the system; when judging that the node fails and throws the exception, directly entering a node failure processing program. As the tasks of the child nodes in the spanning tree depend on the transmission of the father node, once the father node fails, the whole child tree of the father node can be in failure; it is therefore necessary to step through the problem of failed nodes.
Optionally, step 306 may further include:
substep 3061, sending the node information of the failed node to the father node of the failed node according to the tree network planning file; for the father node to continue transmitting the node information of the failed node to the root node.
Specifically, the information of the failed node is reported to the root node according to the path of the spanning tree, and since the tree-like network planning file records the specific topological connection relationship of the transmission tree, each node can reversely transmit the information to the parent node at the upper layer thereof according to the transmission path.
Different from the execution process of data distribution, the transmission of the information of the failed node has unicity and finally only points to the root node of the whole tree transmission network.
And a substep 3062 of, after the root node receives the node information of the failed node, notifying the root node to mark the failed node as abnormal.
After the root node marks the failure node as an abnormal state according to the received failure node information, and feeds back the abnormal state to the client; therefore, the user or operation and maintenance personnel can be informed to carry out related operations such as maintenance and the like on the failed node.
And 307, executing node failure dynamic adjustment operation aiming at the failure node.
And dynamically adjusting the child node tasks of the failed nodes, and handing over the child node tasks to other nodes for continuous execution. In the design of the embodiment of the present application, all the child node transmission tasks of the failed node are dynamically adjusted to the parent node of the node, and exist as a new child node of the parent node in topological view. If the root node fails, the subsequent tasks are not executed and return directly, and the recovery and the re-execution are waited. Therefore, the mechanism can ensure that all normal nodes except the failed node can execute corresponding tasks, and the overall availability of the system is ensured.
Optionally, step 307 may further include:
and a substep 3071, re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a parent node of the failure node, thereby updating the position of the descendant node in the tree structure.
For the failed nodes, the primary processing step is to transfer the connection relation of the descendant nodes to other normal nodes so as to ensure that the nodes can work normally; in the embodiment of the application, a method of re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file and connecting the topological relation of the descendant node to a parent node of the failure node is adopted.
Specifically, referring to fig. 4, fig. 4 is a schematic diagram of a node failure dynamic processing step provided in the embodiment of the present application; as shown in fig. 4, 401 is a reference legend for distinguishing different types of nodes, in a block diagram 402, a node 2 is determined as a failed node by a system, in a topological relationship of the node 2, a parent node at an upper layer is a node 1, nodes 3 and 4 are normal nodes in a same-level relationship with the node 2, descendant nodes of the node 2 include nodes 5, 6 and 8, and a node 8 is also a child node of the node 5, at which time the nodes 5, 6 and 8 are still in a normal state; starting to execute the planning algorithm, and endowing the descendant node connection relationship of the node 2 again, because the parent node of the node 2 in the tree-like relationship is the node 1, the topological relationship of the above 3 nodes is transferred to the node 1, as shown in a block 403, and the effect of the relationship after the transfer is shown in a block 404.
And a substep 3072, adjusting the transmission task of the descendant node to the father node for execution.
As shown in fig. 4, after the dynamic adjustment and transfer, the nodes No. 5 and No. 6 are directly connected with the node No. 1, and the node No. 8 is still used as a child node of the node No. 5; therefore, when the cluster executes the next data transmission task, the target data packet and the tree-shaped network planning file are directly distributed to the No. 5 and No. 6 nodes by the No. 1 node, and even if the No. 2 node fails, the normal work of the No. 5, no. 6 and No. 8 nodes is not influenced.
To sum up, in the data transmission method provided in the embodiment of the present application, a local client executes a planning algorithm, and plans a part of nodes in a digital network to generate a tree topology set based on node information of each node in the digital network, where the set can implement mesh connection between each node in the digital network and generate a tree network transmission tree; generating a tree-shaped network planning file according to a transmission tree structure through a local client, packaging the tree-shaped network planning file and a target data file to be distributed, uploading the tree-shaped network planning file and the target data file to a transmission tree root node, and distributing the data packet to all nodes in a digital network by means of the transmission relation of the tree-shaped network; in addition, the client side carries out state detection on all nodes in the log-on network regularly, and a dynamic regulation mechanism is started for the nodes with invalid detection results to transfer task contents on the original nodes to other nodes for execution; data distribution is carried out to all nodes in the digital networking system by establishing a tree transmission tree, so that the information transmission efficiency among the nodes can be greatly improved; meanwhile, a dynamic regulation mechanism is introduced to fill the vacancy of a fault remediation mechanism of the cluster system, and the fault tolerance of the system is greatly improved.
Referring to fig. 5, fig. 5 is a relational block diagram of a data distribution apparatus provided in an embodiment of the present application, and is applied to a client in a data networking; as shown in fig. 5, the apparatus includes: the system comprises an information acquisition module 501, a topology set construction module 502, a planning file generation module 503 and a data distribution module 504;
the information obtaining module 501 is configured to obtain node information of each node in the digital network, where the node information is used to represent attribute characteristics of the node;
the topology set constructing module 502 is configured to select at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and construct all the target nodes as a node topology set of a tree structure;
the planning file generating module 503 is configured to generate a tree-shaped network planning file according to the node topology set, where the tree-shaped network planning file includes: the identification of the target node, the node type and the position information of the target node in the tree structure;
the data distribution module 504 is configured to distribute the target data packet and the tree-like network planning file to the target node according to the identifier of the target node.
In an alternative implementation, the apparatus further includes:
the node detection module is used for carrying out failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node and acquiring a connection response result;
the result judging module is used for judging that the target node is a failure node under the condition that the connection response time of the target node is greater than a first threshold value;
the invalid node information transmission module is used for acquiring the identifier of the invalid node and the position information of the invalid node in a tree structure according to the tree network planning file and sending the identifier and the position information of the invalid node to a root node in the target node;
and the dynamic adjusting module is used for executing node failure dynamic adjusting operation aiming at the failed node.
In an optional implementation manner, the failed node information transmission module includes:
the invalid node information transmission submodule is used for sending the node information of the invalid node to a father node of the invalid node according to the tree-shaped network planning file; the father node transmits the node information of the failed node to a root node continuously;
and the abnormal node marking submodule is used for informing the root node of marking the failed node as abnormal after the root node receives the node information of the failed node.
In an optional implementation, the dynamic adjustment module includes:
the topological relation updating submodule is used for re-executing a preset planning algorithm on at least one descendant node of the failure node according to the tree network planning file, and connecting the topological relation of the descendant node to a father node of the failure node, so that the position of the descendant node in the tree structure is updated;
and the task adjusting submodule is used for adjusting the transmission task of the descendant node to the father node for execution.
In an optional implementation, the data distribution module includes:
and the data distribution submodule is used for sending the tree-shaped network planning file and the target data packet to the root node according to the identifier of the root node in the tree-shaped structure, so that the root node constructs the target nodes in the data network into a tree-shaped structure through the tree-shaped network planning file, and distributes the target data packet to each target node in the tree-shaped structure.
In an optional implementation manner, the node information includes:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the plurality of networking nodes; the parsing processing operation includes: checking the format of the user input content and storing the data structure of the key information; the standard data structure format comprises the following components: the internet protocol address, the security shell protocol user name, the login password and the node description of the node.
In an optional implementation manner, the topology set building module includes:
the relative distance calculation submodule is used for calculating the geographical distance between the nodes according to the geographical position of the nodes according to the standard data structure obtained by analyzing the node information;
the optimal distance selection submodule is used for selecting a target node in the digital network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
a target node selection submodule, configured to select, according to the standard data structure obtained by analyzing the node information, a target node with the shortest transmission time from the node in the digital network at the current time as a descendant node of the root node, and add the selected target node to the node topology set of the tree structure;
and the set construction circulation submodule is used for repeating the steps until the number of the target nodes added into the node topology set of the tree structure reaches a target threshold value.
In an optional implementation manner, before the topology set building module starts to execute the function, the apparatus further includes:
and the node information analysis module is used for calling an analysis function to carry out analysis processing operation on the node information according to the acquired node information and converting the node information into a standard data structure format.
To sum up, the data transmission device provided in the embodiment of the present application performs a planning algorithm through a local client, and plans a part of nodes in a data network to generate a tree topology set based on node information of each node in the data network, where the set can realize establishment of mesh connection among the nodes in the data network to generate a tree network transmission tree; generating a tree-shaped network planning file according to a transmission tree structure through a local client, packaging the tree-shaped network planning file and a target data file to be distributed, uploading the tree-shaped network planning file and the target data file to be distributed to a transmission tree root node, and distributing the data packet to all nodes in a data network by means of the transmission relation of the tree-shaped network; in addition, the client side regularly carries out state detection on all nodes in the cluster, and starts a dynamic regulation mechanism for the nodes with invalid detection results to transfer task contents on the original nodes to other nodes for execution; data distribution is carried out on all nodes in the cluster system by establishing a tree-shaped transmission tree, so that the information transmission efficiency among the nodes can be greatly improved; meanwhile, a dynamic regulation mechanism is introduced to fill the vacancy of a fault remediation mechanism of the cluster system, and the fault tolerance of the system is greatly improved.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (8)

1. A data distribution method is applied to a client in a data networking, and is characterized by comprising the following steps:
acquiring node information of each node in the data networking, wherein the node information is used for representing attribute characteristics of the node;
selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set with a tree structure;
generating a tree-shaped network planning file according to the node topology set, wherein the tree-shaped network planning file comprises: the identification of the target node, the node type and the position information of the target node in the tree structure;
and distributing a target data packet and the tree-shaped network planning file to the target node according to the identifier of the target node.
2. The method of claim 1, further comprising:
carrying out failure detection on the target node; wherein the failure detection comprises: establishing connection with the target node and acquiring a connection response result;
under the condition that the connection response time of the target node is greater than a first threshold value, judging that the target node is a failure node;
acquiring the identifier of the failed node and the position information of the failed node in a tree structure according to the tree network planning file, and sending the identifier and the position information of the failed node to a root node in the target node;
and executing node failure dynamic adjustment operation aiming at the failure node.
3. The method of claim 2, wherein the obtaining the identifier of the failed node and the location information of the failed node in the tree structure according to the tree network planning file, and sending the identifier of the failed node and the location information to the root node in the target node comprises:
sending the node information of the failure node to a father node of the failure node according to the tree-shaped network planning file; the father node transmits the node information of the failed node to a root node continuously;
and after the root node receives the node information of the failure node, informing the root node to mark the failure node as abnormal.
4. The method of claim 2, wherein performing a node failure dynamic adjustment operation on the failed node comprises:
according to the tree-shaped network planning file, re-executing a preset planning algorithm on at least one descendant node of the failure node, and connecting the topological relation of the descendant node to the father node of the failure node, so as to update the position of the descendant node in the tree-shaped structure;
and adjusting the transmission task of the descendant node to the father node for execution.
5. The method of claim 1, wherein the distributing the target data package and the tree-shaped network plan file to the tree-shaped network transmission tree comprises:
and sending the tree-like network planning file and the target data packet to the root node according to the identifier of the root node in the tree-like structure, so that the root node constructs the target nodes in the data network into a tree-like structure through the tree-like network planning file, and distributes the target data packet to each target node in the tree-like structure.
6. The method of claim 1, wherein the node information comprises:
the host name, the login password, the geographic position of the city and the cloud provider of each host connected with the plurality of networking nodes;
converting the node information into a standard data structure format through analysis processing operation; wherein the parsing processing operation comprises: checking the format of the user input content and storing the data structure of the key information;
the standard data structure format comprises the following components: the internet protocol address, the security shell protocol user name, the login password and the node description of the node.
7. The method according to claim 1, wherein the selecting at least a part of the target nodes from all the nodes according to the node information of the nodes and a preset planning algorithm, and constructing all the target nodes as a node topology set of a tree structure, comprises:
calculating the geographical distance between each node according to the standard data structure format obtained by analyzing the node information and the geographical position of each node;
selecting a target node in the data network as a root node through a clustering algorithm; the target node is the node with the shortest average distance to other nodes in the digital network;
selecting a target node with the shortest transmission time to other nodes in the digital network at the current moment as a descendant node of the root node according to the standard data structure obtained by analyzing the node information, and adding the descendant node into a node topology set of the tree structure;
and repeating the steps until the number of the target nodes added into the node topology set of the tree structure reaches a target threshold value.
8. An efficient data distribution device on a digital network, which is applied to a digital network software client, and is characterized in that the device comprises:
the information acquisition module is used for acquiring node information of each node in the data network, and the node information is used for representing attribute characteristics of the node;
the topology set construction module is used for selecting at least part of target nodes from all the nodes according to the node information and a preset planning algorithm, and constructing all the target nodes into a node topology set with a tree structure;
a planning file generation module, configured to generate a tree-like network planning file according to the node topology set, where the tree-like network planning file includes: the identification of the target node, the node type and the position information of the target node in the tree structure;
and the data distribution module is used for distributing the target data packet and the tree-shaped network planning file to the target node according to the identifier of the target node.
CN202211194855.3A 2022-09-23 2022-09-29 Method and device for efficiently distributing data on digital networking Active CN115623000B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211164068 2022-09-23
CN2022111640684 2022-09-23

Publications (2)

Publication Number Publication Date
CN115623000A true CN115623000A (en) 2023-01-17
CN115623000B CN115623000B (en) 2023-11-14

Family

ID=84860085

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211194855.3A Active CN115623000B (en) 2022-09-23 2022-09-29 Method and device for efficiently distributing data on digital networking

Country Status (1)

Country Link
CN (1) CN115623000B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701410A (en) * 2023-08-07 2023-09-05 北京大学 Method and system for storing memory state data for data language and digital networking system
CN116932614A (en) * 2023-09-18 2023-10-24 北京大学 Distributed searching method and system oriented to digital networking

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105306417A (en) * 2014-06-25 2016-02-03 深圳市华夏双赢通信有限公司 Network data distribution method based on super router and network system
CN110519365A (en) * 2019-08-26 2019-11-29 网宿科技股份有限公司 A kind of method and business change system changing appliance services
CN110597598A (en) * 2019-09-16 2019-12-20 电子科技大学广东电子信息工程研究院 Control method for virtual machine migration in cloud environment
US10614261B1 (en) * 2019-02-05 2020-04-07 Cadence Design Systems, Inc. Honoring pin insertion delay during clock tree synthesis
US20200349228A1 (en) * 2019-05-03 2020-11-05 Sap Se Significant correlation framework for command translation
CN112769580A (en) * 2020-12-31 2021-05-07 阳江市链点创新科技发展有限公司 Block chain layered excitation consensus algorithm
US20220210200A1 (en) * 2015-10-28 2022-06-30 Qomplx, Inc. Ai-driven defensive cybersecurity strategy analysis and recommendation system
CN114760135A (en) * 2022-04-19 2022-07-15 浙江大学 Optimization method of block chain fault-tolerant consensus scheme

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105306417A (en) * 2014-06-25 2016-02-03 深圳市华夏双赢通信有限公司 Network data distribution method based on super router and network system
US20220210200A1 (en) * 2015-10-28 2022-06-30 Qomplx, Inc. Ai-driven defensive cybersecurity strategy analysis and recommendation system
US10614261B1 (en) * 2019-02-05 2020-04-07 Cadence Design Systems, Inc. Honoring pin insertion delay during clock tree synthesis
US20200349228A1 (en) * 2019-05-03 2020-11-05 Sap Se Significant correlation framework for command translation
CN110519365A (en) * 2019-08-26 2019-11-29 网宿科技股份有限公司 A kind of method and business change system changing appliance services
CN110597598A (en) * 2019-09-16 2019-12-20 电子科技大学广东电子信息工程研究院 Control method for virtual machine migration in cloud environment
CN112769580A (en) * 2020-12-31 2021-05-07 阳江市链点创新科技发展有限公司 Block chain layered excitation consensus algorithm
CN114760135A (en) * 2022-04-19 2022-07-15 浙江大学 Optimization method of block chain fault-tolerant consensus scheme

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
GOKUL KANNAN SADASIVAM: "Classification of SSH Attacks using Machine Learning Algorithms", IEEE *
廖怡;盛益强;王劲林;: "一种基于测量的启发式网络拓扑匹配优化算法", 计算机学报, no. 09 *
张明伟;王匀;: "中央核心处理式多层级流量管理系统网络结构设计", 指挥信息系统与技术, no. 03 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701410A (en) * 2023-08-07 2023-09-05 北京大学 Method and system for storing memory state data for data language and digital networking system
CN116701410B (en) * 2023-08-07 2023-11-14 北京大学 Method and system for storing memory state data for data language of digital networking
CN116932614A (en) * 2023-09-18 2023-10-24 北京大学 Distributed searching method and system oriented to digital networking
CN116932614B (en) * 2023-09-18 2023-12-26 北京大学 Distributed searching method and system oriented to digital networking

Also Published As

Publication number Publication date
CN115623000B (en) 2023-11-14

Similar Documents

Publication Publication Date Title
US11706102B2 (en) Dynamically deployable self configuring distributed network management system
US11088903B2 (en) Hybrid cloud network configuration management
JP3966598B2 (en) Server selection system
KR100791429B1 (en) Network attached storage snmp single system image
CN115623000B (en) Method and device for efficiently distributing data on digital networking
CN110209492B (en) Data processing method and device
EP1682994B1 (en) Adaptive load balancing
US7257817B2 (en) Virtual network with adaptive dispatcher
JP6539341B2 (en) Providing router information according to the programmatic interface
CN100479367C (en) Program arranging method and system based on distributing network
CN108540505B (en) Content updating method and device
US20090316600A1 (en) Telecommunications system and server apparatus
US8233480B2 (en) Network clustering
US20090150564A1 (en) Per-user bandwidth availability
WO2020010906A1 (en) Method and device for operating system (os) batch installation, and network device
JP6540063B2 (en) Communication information control apparatus, relay system, communication information control method, and communication information control program
CN110740355A (en) Equipment monitoring method and device, electronic equipment and storage medium
CN111404719B (en) Network topology information acquisition method and system
CN107493308B (en) Method and device for sending message and distributed equipment cluster system
KR20190114126A (en) Software-defined network controller for controlling message transmission to switch based on processing completion delay time of message and method thereof
CN104717176A (en) Access control method, access control system, and server
CN107231394A (en) A kind of building method of data source address distribution tree and the method for replicate data
CN114531393B (en) Method, device, equipment and medium for issuing segment routing strategy
CN113472572B (en) Network recovery method and device based on rail transit signal system
Tan et al. Optimizing all-to-all data transmission in WANs

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant