CN106411605A - Node network self-organizing method, apparatus, server and system - Google Patents

Abstract

The invention discloses a node network self-organization method, device, server and system. Wherein, the method includes: receiving the task start command sent by the master control node; selecting the port number used for the task; returning the host name and the selected port number of the slave node to the master control node, so that the master control node Generate a node network map according to the tasks started by each slave node and the returned host name and port number; receive the node network map sent by the master control node. This technical solution changes the allocation of port numbers from the master control node to each slave node, avoiding the conflict between the port numbers allocated by the master control node and the port numbers already used by the slave nodes, and the generated node network diagram can be used for the node network Management and establishment of connections between nodes not only meets the usage requirements, but also improves the success rate of node network construction.

Description

A node network self-organization method, device, server and system

technical field

The invention relates to the technical field of computer networks, in particular to a node network self-organization method, device, server and system.

Background technique

A distributed cluster has multiple nodes that can run the same task in parallel. In many cases, parallel execution of tasks also requires communication between nodes, so the organization of the node network is a very critical issue. In the prior art, when the task is established, the master control node usually assigns the port number used by the process of executing the task to each slave node executing the task, so that each slave node can know the port number used by the process of executing the task on other slave nodes. The port number used by the process to establish connections with other slave nodes. However, the allocated port numbers are actually one-to-one correspondence with each slave node. If the master control node does not know the port number already used on a certain node, and allocates the port number already used by the node when allocating the port number, it will Will affect the start of the task. For example, task A running on node 1 uses port 8080, and port 8080 is specified when the master control node allocates the port number used by task B to node 1, which causes a port conflict.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a node network ad hoc method, device, server and system for overcoming the above problems or at least partially solving the above problems.

According to one aspect of the present invention, a node network self-organization method is provided, including:

Receive the task start command sent by the master control node;

Select the port number to use for this task;

Return the host name and the selected port number of this slave node to the master control node, so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number;

Receive the node network diagram sent by the master control node.

Optionally, the port number selected for the task includes:

Randomly select a port number from the currently unoccupied port numbers of the slave node.

Optionally, the receiving the node network graph sent by the master control node includes:

Periodically send a request for obtaining a node network map to the master control node, and receive a node network map returned by the master control node according to the request;

and / or,

and receiving the node network diagram actively delivered by the master control node.

Optionally, the method also includes:

According to the node network map sent by the master control node, a connection is established with one or more other slave nodes in the node network map.

Optionally, the task startup instruction is a startup instruction of a deep learning subtask; the deep learning subtask includes: a parameter server subtask and/or a worker subtask.

According to another aspect of the present invention, a node network self-organization method is provided, including:

Send task start instructions to one or more slave nodes according to the input task information;

Receive the host name and port number returned by each slave node;

Generate a node network diagram according to the tasks started by each slave node and the returned host name and port number;

Send the node network graph to one or more slave nodes.

Optionally, the sending the node network graph to one or more slave nodes includes:

When receiving a request for acquiring a node network diagram sent by a slave node, sending the node network diagram to the slave node;

and / or,

Send the node network diagram to all slave nodes connected to the master control node.

Optionally, the task information is task information of a deep learning task; the task information includes: the number of nodes used to execute the deep learning task, the type of deep learning subtasks, and the number of subtasks of each type.

Optionally, the sending a task start instruction to one or more slave nodes according to the input task information includes:

Select slave nodes equivalent to the number of nodes used to perform deep learning tasks from all slave nodes connected to the master control node;

Determine the tasks to start on each slave node according to the type of deep learning subtasks and the number of subtasks of each type;

A task start command corresponding to the task started on the slave node is sent to each selected slave node.

According to yet another aspect of the present invention, a node network ad hoc device is provided, wherein the device is deployed on a slave node of a distributed cluster, including:

The communication unit is adapted to receive the task start instruction sent by the self-organizing server of the node network;

a port selection unit adapted to select a port number for the task;

The communication unit is also suitable for returning the host name and the selected port number of the slave node where the device is located to the node network self-organizing server, so that the node network self-organizing server The hostname and port number returned by the network ad hoc device generate a node network map; and the node network map is adapted to receive the node network self-organizing server.

Optionally, the port selection unit is adapted to randomly select a port number from currently unoccupied port numbers of the slave node where the device is located.

Optionally, the communication unit is adapted to periodically send a request for acquiring a node network graph to the node network ad hoc server, and receive the node network graph returned by the node network ad hoc server according to the request; and/or, receive The node network self-organizing server actively delivers the node network diagram.

Optionally, the communication unit is further adapted to establish a connection with the node network ad hoc device on one or more other slave nodes in the node network map sent by the node network self-organizing server.

Optionally, the task startup instruction is a startup instruction of a deep learning subtask; the deep learning subtask includes: a parameter server subtask and/or a worker subtask.

According to yet another aspect of the present invention, a node network self-organizing server is provided, wherein the server is deployed on a master control node of a distributed cluster, including:

The communication unit is adapted to send a task start instruction to one or more node network self-organizing devices on the slave node according to the input task information; receive the host name and port number returned by each node network self-organizing device;

The node network diagram generation unit is adapted to generate a node network diagram according to the tasks started by each slave node and the host name and port number returned by each node network self-organizing device;

The communication unit is further adapted to send the node network graph to one or more node network self-organizing devices on the slave nodes.

Optionally, the communication unit is adapted to send the node network map to the node network self-organizing device on the slave node when receiving a request for acquiring the node network map sent by the node network self-organizing device on the slave node device; and/or, sending the node network graph to the node network self-organizing device on all slave nodes connected to the master control node where the server is located

Optionally, the task information is task information of a deep learning task; the task information includes: the number of nodes used to execute the deep learning task, the type of deep learning subtasks, and the number of subtasks of each type.

Optionally, the server also includes:

The scheduling unit is adapted to select slave nodes equivalent to the number of nodes used to perform deep learning tasks from all slave nodes connected to the master control node where the server is located; according to the type of deep learning subtasks and the number of subtasks of each type, Determine the tasks to start on each slave node;

The communication unit is adapted to send a task start instruction corresponding to the task started on the slave node to the node network self-organizing device on the selected slave node.

According to another aspect of the present invention, a node network ad hoc system is provided, wherein the system includes one or more node network ad hoc devices as described in any one of the above and the node network as described in any one of the above Network Ad Hoc Server.

As can be seen from the above, in the technical solution of the present invention, after the slave node receives the task start command sent by the master control node, it actively selects the port number used for the task, and returns the host name and the selected port number of the slave node to the master control node. node, so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number. This technical solution changes the allocation of port numbers from the master control node to each slave node, avoiding the conflict between the port numbers allocated by the master control node and the port numbers already used by the slave nodes, and the generated node network diagram can be used for the node network Management and establishment of connections between nodes not only meets the usage requirements, but also improves the success rate of node network construction.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:

Fig. 1 shows a schematic flow chart of a node network self-organization method according to an embodiment of the present invention;

Fig. 2 shows a schematic flow chart of another node network self-organization method according to an embodiment of the present invention;

Fig. 3 shows a schematic structural diagram of a node network self-organizing device according to an embodiment of the present invention;

Fig. 4 shows a schematic structural diagram of a node network self-organizing server according to an embodiment of the present invention;

Fig. 5 shows a schematic structural diagram of a node network ad hoc system according to an embodiment of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

Fig. 1 shows a schematic flow chart of a node network self-organization method according to an embodiment of the present invention. As shown in Fig. 1, the method includes:

Step S110, receiving a task start instruction sent by the master control node.

Step S120, selecting the port number used for the task.

Step S130, returning the host name and selected port number of the slave node to the master control node, so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number.

Step S140, receiving the node network diagram sent by the master control node.

It can be seen that in the method shown in Figure 1, the slave node actively selects the port number used for the task after receiving the task start instruction sent by the master node, and returns the host name and the selected port number of the slave node to the master node , so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number. This technical solution changes the allocation of port numbers from the master control node to each slave node, avoiding the conflict between the port numbers allocated by the master control node and the port numbers already used by the slave nodes, and the generated node network diagram can be used for the node network Management and establishment of connections between nodes not only meets the usage requirements, but also improves the success rate of node network construction.

In an embodiment of the present invention, in the method shown in FIG. 1 , selecting the port number used for the task includes: randomly selecting a port number from the port numbers not currently occupied by the slave node.

The port number on the slave node is usually 65535, which is randomly selected from unoccupied port numbers, which is efficient and does not cause port conflicts.

In one embodiment of the present invention, in the method shown in FIG. 1, receiving the node network map sent by the master control node includes: regularly sending a request to the master control node to obtain the node network map, and receiving the request returned by the master control node according to the request. A node network diagram; and/or, receiving a node network diagram actively issued by the master control node.

For example, a request to obtain the node network map is sent to the master control node every 10 minutes, or the master control node sends the updated node network map to the relevant slave nodes after the node network map changes.

An important reason for each slave node to obtain a node network map is that in many cases, processes running on slave nodes need to communicate with processes on other slave nodes. Therefore, in an embodiment of the present invention, the method shown in FIG. 1 further includes: according to the node network map sent by the master control node, establishing a connection with one or more other slave nodes in the node network map.

In an embodiment of the present invention, in the method shown in FIG. 1 , the task start command is a start command of a deep learning subtask; the deep learning subtask includes: a parameter server subtask and/or a worker subtask. In this example, the parameter server acts as a parameter server and needs to receive the calculated parameters submitted by the worker subtask.

Fig. 2 shows a schematic flow chart of another node network self-organization method according to an embodiment of the present invention. As shown in Fig. 2, the method includes:

Step S210, sending a task start instruction to one or more slave nodes according to the input task information.

Step S220, receiving the host name and port number returned by each slave node.

Step S230, generating a node network diagram according to the tasks initiated by each slave node and the returned host name and port number.

Step S240, sending the node network map to one or more slave nodes.

In one embodiment of the present invention, in the method shown in FIG. 2, sending the node network diagram to one or more slave nodes includes: when receiving a request for obtaining the node network diagram sent by the slave node, sending the node network diagram Send to the slave node; and/or, send the node network map to all slave nodes connected to the master control node.

In this embodiment, two distribution methods of the node network diagram are provided, which can be used in combination, but this does not represent a limitation on the distribution method. When the node network diagram changes, the updated node network diagram can only be sent to Nodes related to this update. For example, task A adds two execution nodes, node 13 and node 14, and the original execution nodes of task A are node 1 and node 2, then only the updated node network diagram needs to be sent to nodes 1, 3, 13 and 14 . Of course, the node network diagram can also generate corresponding sub-graphs according to each task, so that when the node network diagram is delivered, it only needs to be distributed to the relevant nodes in the diagram.

In one embodiment of the present invention, in the method shown in Figure 2, the task information is the task information of the deep learning task; the task information includes: the number of nodes used to perform the deep learning task, the type of deep learning subtask, each type of The number of subtasks.

Deep learning tasks are submitted in the form of graphs. These tasks are further divided into multiple jobs during execution, and each job includes one or more subtasks. Subtask types include one or more of the following: parameter server subtask, worker subtask.

For example, TensorFlow is an open source deep learning library. Tensor (tensor) means an N-dimensional array, Flow (flow) means calculation based on a data flow graph, and TensorFlow is the calculation process of tensor flowing from one end of the image to the other end. The deep learning library can be integrated with the Spark big data computing framework, that is, submit a TensorFlow task as a Spark task, which is the deep learning task referred to above. The deep learning task information can also include one or more of the following: the calculation graph for performing deep learning; the deep learning library interface that needs to be called to execute the deep learning task; the data address for the deep learning task; the storage address of the execution result data .

In an embodiment of the present invention, in the above method, according to the input task information, sending a task start command to one or more slave nodes includes: selecting from all slave nodes connected to the master control node and using the execution depth Slave nodes with the same number of learning task nodes; according to the deep learning subtask type and the number of subtasks of each type, determine the tasks started on each slave node; send to each selected slave node the task started on the slave node The corresponding task start command.

Taking a deep learning task as an example, if according to the task information of the deep learning task, it is necessary to call the deep learning library to start 2 parameter server subtasks and 2 worker subtasks, and these four subtasks are executed on four slave nodes respectively , then first determine the subtasks executed on each task, and then send instructions to each slave node to start the corresponding subtasks.

As mentioned above, the deep learning library can be integrated with the Spark big data computing framework, that is, the distributed cluster can be a Spark cluster. Spark clusters can also perform task scheduling, job management, and resource management through Yarn. Yarn can provide users with a front-end page for task submission, so in an embodiment of the present invention, the submitted deep learning task can be input through the front-end page. After the task is started, the user can also view the processing status of the task in real time and perform operations such as killing the task according to the front-end page provided by Yarn. Since the Spark cluster can also perform task scheduling, job management, and resource management through Yarn, in the above-mentioned embodiment, select slave nodes equivalent to the number of nodes used to perform deep learning tasks from all slave nodes connected to the master control node. Nodes can also obtain relatively idle nodes to perform deep learning tasks by sending requests to Yarn. That is: send the number of nodes used to execute the deep learning task to the node scheduler of the distributed cluster, and receive the information of multiple nodes returned by the node scheduler.

An example of a node network graph generated for a deep learning task is shown below:

{PS:[node1:8080,node2:8080]worker:[node3:9090,node4:9090]}

This means that the parameter server subtask is started on port 8080 of node 1, the parameter server subtask is started on port 8080 of node 2; the worker subtask is started on port 9090 of node 3, and the subtask of worker is started on port 9090 of node 4 The worker subtask is started on . Next, it is necessary to actively send the node network map to these slave nodes, or send the node network map to these nodes according to the request for acquiring the self-benefit network list sent by each slave node. For example, the worker subtask started on port 9090 of node 3 can establish connections with the parameter server subtask started on port 8080 of node 1 and the parameter server subtask started on port 8080 of node 2 respectively.

After the deep learning task is submitted, Spark starts a Driver process and a Scheduler scheduling process at the same time, which realizes the construction, management and distribution of the node network graph.

Specifically, obtaining the data used for the deep learning task from the file system of the distributed cluster includes: according to the data address used for the deep learning task, constructing the data used for the deep learning task in the file system of the distributed cluster as Elastic distributed data set RDD object; pushing the obtained data for the deep learning task to the corresponding subtask for execution includes: pushing the RDD object to each node, and each node pushes the RDD object to the node on subtasks started in .

Taking the Spark distributed cluster as an example, its data is stored on HDFS (Hadoop Distributed File System, Hadoop Distributed File System). When operating data, it is correspondingly constructed as an RDD (resilient distributed dataset, resilient distributed dataset) object. RDD objects can be reused. If the data used by deep learning tasks has been constructed as RDD objects, then this step is naturally unnecessary. When these data are used, they are pushed to the slave nodes where each task is located through a pipe, and each node pushes the RDD object to the subtask started in the slave node. The deep learning task in the above example includes two worker subtasks as an example. Part of the RDD object needs to be pushed to node 3, and the other part is pushed to node 4, thus realizing distributed processing of deep learning tasks.

Fig. 3 shows a schematic structural diagram of a node network self-organizing device according to an embodiment of the present invention, and the device can be deployed on slave nodes of a distributed cluster. As shown in Figure 3, the node network ad hoc device 300 includes:

The communication unit 310 is adapted to receive a task start instruction sent by the node network ad hoc server.

The port selecting unit 320 is adapted to select the port number used for the task.

The communication unit 310 is also suitable for returning the host name and the selected port number of the slave node where the device is located to the node network self-organizing server, so that the node network self-organizing server returns the The host name and port number of the node network are generated to generate a node network diagram; and the node network diagram is suitable for receiving the node network self-organizing server.

It can be seen that in the device shown in Figure 3, through the mutual cooperation of each unit, the slave node actively selects the port number used for the task after receiving the task start command sent by the master control node, and uses the host name of the slave node and the selected The port number is returned to the master control node, so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number. This technical solution changes the allocation of port numbers from the master control node to each slave node, avoiding the conflict between the port numbers allocated by the master control node and the port numbers already used by the slave nodes, and the generated node network diagram can be used for the node network Management and establishment of connections between nodes not only meets the usage requirements, but also improves the success rate of node network construction.

In an embodiment of the present invention, in the device shown in FIG. 3 , the port selection unit 320 is adapted to randomly select a port number from currently unoccupied port numbers of the slave node where the device is located.

In one embodiment of the present invention, in the device shown in FIG. 3 , the communication unit 310 is adapted to periodically send a request for obtaining a node network map to the node network ad hoc server, and receive the node network graph returned by the node network ad hoc server according to the request. A network diagram; and/or, receiving a node network diagram actively delivered by the node network self-organizing server.

In one embodiment of the present invention, in the device shown in FIG. 3 , the communication unit 310 is further adapted to communicate with other one or more slave nodes in the node network graph according to the node network graph sent by the node network self-organizing server. The nodes on the network ad hoc device 300 establish connections.

In an embodiment of the present invention, in the apparatus shown in FIG. 3 , the task start command is a start command of a deep learning subtask; the deep learning subtask includes: a parameter server subtask and/or a worker subtask.

Fig. 4 shows a schematic structural diagram of a node network ad hoc server according to an embodiment of the present invention, and the server can be deployed on a master control node of a distributed cluster. As shown in Figure 4, the node network self-organizing server 400 includes:

The communication unit 410 is adapted to send a task start instruction to one or more node network self-organizing devices on the slave node according to the input task information; receive the host name and port number returned by each node network self-organizing device;

The node network diagram generation unit 420 is adapted to generate a node network diagram according to the tasks started by each slave node and the host name and port number returned by each node network self-organizing device;

The communication unit 410 is further adapted to send the node network graph to the node network self-organizing device on one or more slave nodes.

In one embodiment of the present invention, in the server shown in FIG. 4 , the communication unit 410 is adapted to send the node network graph to To the node network self-organizing device on the slave node; and/or, sending the node network map to the node network self-organizing device on all slave nodes connected to the master node where the server is located

In one embodiment of the present invention, in the server shown in Figure 4, the task information is the task information of the deep learning task; the task information includes: the number of nodes used to perform the deep learning task, the type of deep learning subtask, each type of The number of subtasks.

In one embodiment of the present invention, the server shown in FIG. 4 also includes: a scheduling unit 430, adapted to select a node for performing deep learning tasks from all slave nodes connected to the master control node where the server is located A considerable number of slave nodes; according to the deep learning subtask type and the number of subtasks of each type, determine the task started on each slave node; the communication unit 310 is suitable for sending a message to the node network self-organizing device on the selected slave node The task start command corresponding to the task started on the slave node.

It should be noted that the specific implementation manners of the foregoing apparatus and server embodiments are similar to the specific implementation manners of the aforementioned corresponding method embodiments, and will not be repeated here. A slight difference is that not only node network self-organizing devices but also task execution devices can be deployed on each slave node. Not only the self-organizing server of the node network can be deployed on the master control node, but also the control server of the task can be deployed. Of course, these servers can be implemented as one server through functional integration, and each device on the same slave node can also be implemented as one device through functional integration.

Fig. 5 shows a schematic structural diagram of a node network self-organizing system according to an embodiment of the present invention. As shown in Fig. 5, the node network self-organizing system 500 includes one or more node networks as in any of the above-mentioned embodiments The ad hoc device 300 and the node network ad hoc server 400 as in any of the foregoing embodiments.

In summary, in the technical solution of the present invention, after receiving the task start instruction sent by the master control node, the slave node actively selects the port number used for the task, and returns the host name and the selected port number of the slave node to the master node. Control nodes, so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number. This technical solution changes the allocation of port numbers from the master control node to each slave node, avoiding the conflict between the port numbers allocated by the master control node and the port numbers already used by the slave nodes, and the generated node network diagram can be used for the node network Management and establishment of connections between nodes not only meets the usage requirements, but also improves the success rate of node network construction.

It should be noted:

The algorithms and displays presented herein are not inherently related to any particular computer, virtual appliance, or other device. Various general purpose devices can also be used with the teachings based on this. The structure required to construct such an apparatus will be apparent from the foregoing description. Furthermore, the present invention is not specific to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of some or all of the node network self-organizing devices, servers, and systems according to embodiments of the present invention. Full functionality. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

The embodiment of the present invention discloses A1, a node network self-organization method, wherein the method includes:

Receive the task start command sent by the master control node;

Select the port number to use for this task;

Return the host name and the selected port number of this slave node to the master control node, so that the master control node generates a node network diagram according to the tasks started by each slave node and the returned host name and port number;

Receive the node network diagram sent by the master control node.

A2. The method as described in A1, wherein the port number selected for the task includes:

Randomly select a port number from the currently unoccupied port numbers of the slave node.

A3. The method according to A1, wherein the receiving the node network diagram sent by the master control node includes:

Periodically send a request for obtaining a node network map to the master control node, and receive a node network map returned by the master control node according to the request;

and / or,

and receiving the node network diagram actively delivered by the master control node.

A4. The method as described in A1, wherein the method also includes:

According to the node network map sent by the master control node, a connection is established with one or more other slave nodes in the node network map.

A5. The method as described in A1, wherein,

The task startup instruction is a startup instruction of a deep learning subtask; the deep learning subtask includes: a parameter server subtask and/or a worker subtask.

The embodiment of the present invention also discloses B6, a node network self-organization method, wherein the method includes:

Send task start instructions to one or more slave nodes according to the input task information;

Receive the host name and port number returned by each slave node;

Generate a node network diagram according to the tasks started by each slave node and the returned host name and port number;

Send the node network graph to one or more slave nodes.

B7. The method as described in B6, wherein the sending the node network graph to one or more slave nodes includes:

When receiving a request for acquiring a node network diagram sent by a slave node, sending the node network diagram to the slave node;

and / or,

Send the node network diagram to all slave nodes connected to the master control node.

B8. The method as described in B6, wherein the task information is the task information of the deep learning task; the task information includes: the number of nodes used to perform the deep learning task, the deep learning subtask type, and various types of subtasks quantity.

B9. The method as described in B8, wherein, according to the input task information, sending a task start instruction to one or more slave nodes includes:

Select slave nodes equivalent to the number of nodes used to perform deep learning tasks from all slave nodes connected to the master control node;

Determine the tasks to start on each slave node according to the type of deep learning subtasks and the number of subtasks of each type;

A task start command corresponding to the task started on the slave node is sent to each selected slave node.

The embodiment of the present invention also discloses C10, a node network ad hoc device, wherein the device is deployed on the slave nodes of the distributed cluster, including:

The communication unit is adapted to receive the task start instruction sent by the self-organizing server of the node network;

a port selection unit adapted to select a port number for the task;

The communication unit is also suitable for returning the host name and the selected port number of the slave node where the device is located to the node network self-organizing server, so that the node network self-organizing server The hostname and port number returned by the network ad hoc device generate a node network map; and the node network map is adapted to receive the node network self-organizing server.

C111. The device of C110, wherein,

The port selection unit is adapted to randomly select a port number from currently unoccupied port numbers of the slave node where the device is located.

C112. The device of C110, wherein,

The communication unit is adapted to periodically send a request for acquiring a node network map to the node network self-organizing server, receive the node network map returned by the node network self-organizing server according to the request; and/or receive the node network map The node network diagram actively issued by the self-organizing server.

C113. The device of C110, wherein,

The communication unit is further adapted to, according to the node network map sent by the node network self-organizing server, establish a connection with other one or more node network self-organizing devices on the node network map.

C114. The device of C110, wherein,

The task startup instruction is a startup instruction of a deep learning subtask; the deep learning subtask includes: a parameter server subtask and/or a worker subtask.

The embodiment of the present invention also discloses D15, a node network self-organizing server, wherein the server is deployed on the master control node of the distributed cluster, including:

The communication unit is adapted to send a task start instruction to one or more node network self-organizing devices on the slave node according to the input task information; receive the host name and port number returned by each node network self-organizing device;

The node network diagram generation unit is adapted to generate a node network diagram according to the tasks started by each slave node and the host name and port number returned by each node network self-organizing device;

The communication unit is further adapted to send the node network graph to one or more node network self-organizing devices on the slave nodes.

D16. The server as described in D15, wherein,

The communication unit is adapted to send the node network map to the node network self-organizing device on the slave node when receiving a request for acquiring a node network map sent by the node network self-organizing device on the slave node; and/ Or, send the node network graph to the node network self-organizing device on all slave nodes connected to the master control node where the server is located

D17. The server as described in D15, wherein the task information is the task information of the deep learning task; the task information includes: the number of nodes used to perform the deep learning task, the type of deep learning subtasks, and various types of subtasks quantity.

D18. The server as described in D17, wherein the server further includes:

The scheduling unit is adapted to select slave nodes equivalent to the number of nodes used to perform deep learning tasks from all slave nodes connected to the master control node where the server is located; according to the type of deep learning subtasks and the number of subtasks of each type, Determine the tasks to start on each slave node;

The communication unit is adapted to send a task start instruction corresponding to the task started on the slave node to the node network self-organizing device on the selected slave node.

The embodiment of the present invention also discloses E19, a node network ad hoc system, wherein the system includes one or more node network ad hoc devices as described in any one of C10-C14 and claims D15-D18 The node network self-organizing server described in any one.

Claims (10)

1. a kind of meshed network self-organizing method, wherein, the method includes：

Receive the task start instruction that main controlled node sends；

Choose the port numbers for this task；

The port numbers of the host name of this from node and selection are returned to described main controlled node, so that described main controlled node is according to each Task and the host name of return and port numbers that from node starts, generate meshed network figure；

Receive the meshed network figure that described main controlled node sends.

2. the method for claim 1, wherein the described port numbers chosen for this task include：

From this from node currently unappropriated port numbers, randomly select a port number.

3. the method for claim 1, wherein the described meshed network figure receiving described main controlled node transmission includes：

Periodically send, to described main controlled node, the request obtaining meshed network figure, receive described main controlled node and returned according to this request Meshed network figure；

And/or,

Receive the meshed network figure that described main controlled node active issues.

4. a kind of meshed network self-organizing method, wherein, the method includes：

According to the mission bit stream of input, send task start instruction to one or more from nodes；

Receive host name and the port numbers that each from node returns；

Being started according to each from node of task and the host name of return and port numbers, generate meshed network figure；

Described meshed network figure is sent to one or more from nodes.

5. described meshed network figure wherein, described is sent to one or more from node bags by method as claimed in claim 4 Include：

When receiving the request of acquisition meshed network figure of from node transmission, described meshed network figure is sent to this from section Point；

And/or,

Described meshed network figure is sent to all from nodes being connected with this main controlled node.

6. a kind of meshed network self-organizing device, wherein, this device is deployed in the from node of distributed type assemblies, including：

Communication unit, is suitable to the task start instruction of receiving node self-organization of network server transmission；

Unit is chosen in port, is suitable to choose the port numbers for this task；

Described communication unit, the port numbers being further adapted for the host name of this device place from node and choosing return to described node Self-organization of network server, so that the described meshed network hoc service device task of being started according to each from node and each node net Host name and port numbers that network self-organizing device returns, generate meshed network figure；And be suitable to receive described meshed network from group Knit the meshed network figure of server transmission.

7. device as claimed in claim 6, wherein,

Unit is chosen in described port, is suitable to from the currently unappropriated port numbers of this device place from node, randomly selects A port number.

8. a kind of meshed network hoc service device, wherein, this server disposition, on the main controlled node of distributed type assemblies, wraps Include：

Communication unit, is suitable to the mission bit stream according to input, to the meshed network self-organizing device in one or more from nodes Send task start instruction；Receive host name and the port numbers that each meshed network self-organizing device returns；

Meshed network figure signal generating unit, be suitable to according to each from node start task and each meshed network self-organizing device return Host name and port numbers, generate meshed network figure；

Described communication unit, is further adapted for for described meshed network figure being sent to meshed network in one or more from nodes from group Knit device.

9. server as claimed in claim 8, wherein,

Described communication unit, the acquisition meshed network figure that the meshed network self-organizing device being suitable on receiving from node sends Request when, described meshed network figure is sent to the meshed network self-organizing device in this from node；And/or, by described section Spot net figure is sent to the meshed network self-organizing device in all from nodes that the main controlled node being located with book server is connected.

10. a kind of meshed network self-organizing system, wherein, this system includes one or more such as any one of claims 6-7 Described meshed network self-organizing device and the meshed network hoc service device as any one of claim 8-9.