CN112199326B - Method and device for dynamically constructing software supernodes on array heterogeneous computing system - Google Patents
Method and device for dynamically constructing software supernodes on array heterogeneous computing system Download PDFInfo
- Publication number
- CN112199326B CN112199326B CN202011396788.4A CN202011396788A CN112199326B CN 112199326 B CN112199326 B CN 112199326B CN 202011396788 A CN202011396788 A CN 202011396788A CN 112199326 B CN112199326 B CN 112199326B
- Authority
- CN
- China
- Prior art keywords
- node
- super
- daemon
- physical
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/76—Architectures of general purpose stored program computers
- G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
- G06F15/7807—System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
- G06F15/7825—Globally asynchronous, locally synchronous, e.g. network on chip
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/76—Architectures of general purpose stored program computers
- G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
- G06F15/7807—System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
Abstract
The application relates to a method, a device, computer equipment and a storage medium for dynamically constructing a software supernode on an array heterogeneous computing system. The method comprises the following steps: a daemon process is started in a background of each physical node in the array heterogeneous computing system, information such as a host name, a node identity, a daemon process number and a communication port address of each physical node is exchanged inside the super-junction point through a super-junction point configuration process, the information is stored in a shared memory space under the daemon process, a software super-junction point is obtained through construction, after the software super-junction point is constructed, the super-junction point configuration process exits, and the daemon process resides in the background and is responsible for managing information inside the super-junction point. By the method, when the number of hardware nodes is increased or decreased in the array heterogeneous computing system, the corresponding software nodes can be dynamically constructed in the software super nodes.
Description
Technical Field
The present application relates to the field of heterogeneous computer technologies, and in particular, to a method, an apparatus, a computer device, and a storage medium for dynamically constructing a software supernode on an array heterogeneous computing system.
Background
Heterogeneous computing is continuously developed in the Field of high-performance computing due to its characteristics of high performance and high energy efficiency, and more special accelerators such as a GPU (Graphics Processing Unit), an FPGA (Field-programmable Gate Array), a DSP (Digital Signal Processor), an AI (Artificial Intelligence) accelerator, and the like are emerging. Generally, an accelerator is connected to a host CPU (Central Processing Unit) in the form of a PCIe (Peripheral Component Interconnect Express) Peripheral device to form tightly coupled heterogeneous computing nodes, and then a parallel system is constructed by a plurality of nodes. This approach is computationally efficient but not easily scalable, making it difficult to add to a well-built high performance computing system when new accelerators are present.
Therefore, the prior art has the problem that the nodes are not easy to expand.
Disclosure of Invention
In view of the above, there is a need to provide a method, an apparatus, a computer device, and a storage medium for dynamically constructing a software supernode on an array heterogeneous computing system, which can solve the problem of inflexible node expansion.
A method of dynamically building software supernodes on an array heterogeneous computing system, the method comprising:
acquiring a physical node list of a supernode required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system;
acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes;
starting a daemon process on each physical node in the physical node list so that the daemon process creates and initializes a super-junction data structure;
running a super-junction point configuration process on the physical node, obtaining identity judgment information for judging whether the physical node is a main node according to the super-junction point configuration process number and the main node number of the super-junction point configuration process, and writing the average slave node number information and the identity judgment information into the super-junction point data structure under the daemon of the physical node;
according to the identity judgment information, when the physical node is judged to be a master node, super-junction point organization information sent by the slave node is sequentially received, and when the physical node is judged to be a slave node, super-junction point organization information sent by the master node is received; the super node organization information comprises a host name of the physical node, a process number of a daemon process and a communication port address;
and writing the super-node organization information into a corresponding super-node data structure to complete the construction of the software super-node.
In one embodiment, the method further comprises the following steps: acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list; wherein the number of master nodes and the number of slave nodes satisfy the constraint of:
wherein the content of the first and second substances,
representing the number of the slave nodes;
representing the number of the main nodes;
representing the number of physical nodes in the physical node list;
is an integer;
obtaining the average number of the slave nodes under each master node according to the number of the master nodes and the number of the slave nodes as follows:
wherein the content of the first and second substances,
representing the average slave node number information.
In one embodiment, a daemon process is started on each physical node in the physical node list, and whether the daemon process runs in the physical node is determined;
if the daemon process runs in the physical node, ending the current daemon process, and exiting the program for starting the daemon process;
if no daemon process runs in the physical node, creating a daemon process log file, writing a process number of the daemon process into the daemon process log file, and locking the daemon process log file;
initializing the super junction point data structure through the daemon process;
the method further comprises the following steps:
establishing and initializing a communication port through the daemon process to obtain a communication port address;
creating and initializing a shared memory space through the daemon process, wherein the shared memory space comprises parameters: the communication state marks of the physical node daemon and a super junction point configuration process, the identity judgment information of the physical node, the average slave node number information, the communication port address of the physical node and the process number of the physical node daemon;
acquiring a daemon process number through the daemon process, writing the daemon process number into the shared memory space, and setting the communication state mark as a first state;
suspending the daemon process so that the daemon process enters a wait wakeup state.
In one embodiment, a super junction point configuration process number is obtained through the super junction point configuration process;
when the super node configuration process number is smaller than the number of the main nodes, judging that the current physical node is the main node, and setting the identity judgment information as TRUE;
when the super node configuration process number is not less than the number of the main nodes, judging that the current physical node is a slave node; and setting the identity judgment information as FALSE.
In one embodiment, after the information of the average slave node number is acquired by the super junction point configuration process, the communication state flag is set to be in a second state;
acquiring a daemon number of a daemon corresponding to the physical node from the shared memory space through the super-junction point configuration process, and sending a pairing signal to the corresponding daemon according to the daemon number through the super-junction point configuration process so that the daemon receives the pairing signal and writes the average slave node number information and the identity judgment information in the shared memory space into a corresponding super-junction point data structure when judging that the communication state is marked as a second state;
acquiring the communication port address of the physical node through the daemon process, and marking the communication state as a third state after writing the communication port address into a shared memory space;
suspending the daemon process so that the daemon process enters a wait wakeup state.
In one embodiment, after detecting that the communication state is marked as a third state through the super junction point configuration process, acquiring the communication port address from the shared memory space corresponding to the physical node;
splicing the host name, the process number and the communication port address of the physical node into a cache buffer of supernode organization information;
according to the identity judgment information, if the physical node is a main node, sequentially sending the cache buffers to slave nodes in the super nodes, and sequentially receiving the cache buffers sent by the slave nodes; wherein, the process number of the slave node is:
wherein the content of the first and second substances,
a process number representing the slave node;
the process number representing the super junction point configuration process,
；
if the physical node is a slave node, sending the cache buffer to a master node in the super node, and receiving the cache buffer sent by the master node; wherein, the process number of the main node is:
wherein the content of the first and second substances,
a process number representing the master node.
In one embodiment, the remote host name, the process number of the remote daemon process and the remote communication port address in the super node organization information are analyzed from the received cache buffer through the super node configuration process of the physical node, and are written into the shared memory space;
setting the communication state flag to be a fourth state, and sending a pairing signal to the daemon process;
after the daemon process receives the pairing signal and judges that the communication state is marked as a fourth state, the remote host name, the process number of the remote daemon process and the remote communication port address in a shared memory space are written into a super-junction data structure under the physical node;
setting the communication state mark under the physical node to be a first state;
suspending the daemon process so that the daemon process enters a wait wakeup state;
and after detecting that the communication state is marked as a first state, the super node configuration process disconnects the shared memory space, quits the super node configuration process, and completes the construction of the software super node after all the super node configuration processes of all the physical nodes in the physical node list quit.
An apparatus for dynamically building software supernodes on an array heterogeneous computing system, the apparatus comprising:
the physical node list acquisition module is used for acquiring a physical node list of the supernode required to be constructed in the array heterogeneous computing system; the super node is composed of a plurality of physical nodes in the array heterogeneous computing system.
The average slave node number information acquisition module is used for acquiring a physical node list of the super nodes required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system; acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes;
the daemon starting module is used for starting a daemon on each physical node in the physical node list so as to enable the daemon to create and initialize a super-junction data structure;
a super-node configuration process running module, configured to run a super-node configuration process on the physical node, obtain identity judgment information for judging whether the physical node is a main node according to a super-node configuration process number and the main node number of the super-node configuration process, and write the average slave node number information and the identity judgment information into the super-node data structure under the daemon process of the physical node;
the super-node organization information exchange module is used for sequentially receiving super-node organization information sent by the slave node when the physical node is judged to be the master node according to the identity judgment information, and receiving the super-node organization information sent by the master node when the physical node is judged to be the slave node; the super node organization information comprises a host name of the physical node, a process number of a daemon process and a communication port address;
and the super-junction point data structure writing module is used for writing the super-junction point organization information into the corresponding super-junction point data structure to complete the software super-junction point construction.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring a physical node list of a supernode required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system;
acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes;
starting a daemon process on each physical node in the physical node list so that the daemon process creates and initializes a super-junction data structure;
running a super-junction point configuration process on the physical node, obtaining identity judgment information for judging whether the physical node is a main node according to the super-junction point configuration process number and the main node number of the super-junction point configuration process, and writing the average slave node number information and the identity judgment information into the super-junction point data structure under the daemon of the physical node;
according to the identity judgment information, when the physical node is judged to be a master node, super-junction point organization information sent by the slave node is sequentially received, and when the physical node is judged to be a slave node, super-junction point organization information sent by the master node is received; the super node organization information comprises a host name of the physical node, a process number of a daemon process and a communication port address;
and writing the super-node organization information into a corresponding super-node data structure to complete the construction of the software super-node.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring a physical node list of a supernode required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system;
acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes;
starting a daemon process on each physical node in the physical node list so that the daemon process creates and initializes a super-junction data structure;
running a super-junction point configuration process on the physical node, obtaining identity judgment information for judging whether the physical node is a main node according to the super-junction point configuration process number and the main node number of the super-junction point configuration process, and writing the average slave node number information and the identity judgment information into the super-junction point data structure under the daemon of the physical node;
according to the identity judgment information, when the physical node is judged to be a master node, super-junction point organization information sent by the slave node is sequentially received, and when the physical node is judged to be a slave node, super-junction point organization information sent by the master node is received; the super node organization information comprises a host name of the physical node, a process number of a daemon process and a communication port address;
and writing the super-node organization information into a corresponding super-node data structure to complete the construction of the software super-node.
According to the method, the device, the computer equipment and the storage medium for dynamically constructing the software super node on the array heterogeneous computing system, a daemon process is started in a background of each physical node in the array heterogeneous computing system, information such as a host name, a node identity, a daemon process number, a communication port address and the like of each physical node is exchanged inside the super node through a super node configuration process and stored in a shared memory space under the daemon process to construct the software super node, after the software super node is constructed, the super node configuration process exits, and the daemon process resides in the background and is responsible for managing information inside the super node. By the method, when hardware nodes are added or reduced in the array heterogeneous computing system, corresponding software nodes and shared memory space maintained by the daemon process can be dynamically constructed in the software supernode, the daemon process of the physical node is supported to communicate with other processes in the same node, and communication between the daemon process and any node in the supernode is also supported.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating a method for dynamically building software supernodes on an array heterogeneous computing system, according to one embodiment;
FIG. 2 is a block diagram of an apparatus for dynamically building software supernodes on an array heterogeneous computing system, in accordance with an embodiment;
FIG. 3 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The method for dynamically constructing the software super-node on the array heterogeneous computing system can be applied to the application environment under the graph. The method comprises the steps of designing and constructing an array heterogeneous computer system, independently forming an array by accelerators on a hardware level, and when the novel accelerators are replaced or appear in the array, virtualizing separated hardware nodes into software supernodes from a software level.
In one embodiment, as shown in FIG. 1, there is provided a method for dynamically building software supernodes on an array heterogeneous computing system, comprising the steps of:
The super node is composed of a plurality of physical nodes in the array heterogeneous computing system. The physical node may be a general-purpose Processor, or may be a plurality of special accelerators, such as a GPU (Graphics Processing Unit), an FPGA (Field-programmable Gate Array), a DSP (Digital Signal Processor), an AI (Artificial Intelligence) accelerator, and so on. On the hardware level, the physical nodes independently form an array and are interconnected with the existing system through a high-speed interconnection network.
And 104, acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes.
On the array heterogeneous computing system, one or more software super nodes can be constructed, and in one software super node, one master node and a plurality of slave nodes are arranged. In the embodiment, the number of slave nodes in each software supernode is the same.
And 106, starting a daemon process on each physical node in the physical node list so that the daemon process creates and initializes the super-junction data structure.
The daemon is a process that runs in the background and is not controlled by any terminal. In this embodiment, a daemon process resident in a background needs to be run on each physical node, and is responsible for managing information inside a super junction point to be constructed, where the information mainly includes information such as the number of physical nodes in the super junction point, the identity of a node (a master node or a slave node) where the node is located, the number of the node where the node is located inside the super junction point, and communication port numbers ep of all nodes in the super junction point. Therefore, when the daemon process is started, some initialization operations are required, including creating and initializing a super junction point data structure for maintaining software super junction point information.
And 108, operating a super-junction point configuration process on the physical nodes, obtaining identity judgment information for judging whether the physical nodes are main nodes according to the super-junction point configuration process number and the main node number of the super-junction point configuration process, and writing the average slave node number information and the identity judgment information into a super-junction point data structure under a daemon process of the physical nodes.
The physical node number of the main node in the software super node is in front, and the identity judgment information for judging whether the physical node is the main node or not can be obtained according to the super node configuration process number and the main node number of the super node configuration process.
And step 110, sequentially receiving super-junction organization information sent by the slave node when the physical node is judged to be the master node according to the identity judgment information, and receiving the super-junction organization information sent by the master node when the physical node is judged to be the slave node.
The super node organization information comprises a host name of the physical node, a process number of the daemon process and a communication port address.
When the physical node is a main node in the super nodes, the main node sequentially sends super node organization information of the main node to slave nodes in the super nodes according to the process number, and sequentially receives the slave node super node organization information sent by the slave nodes; and when the physical node is a slave node in the super nodes, the slave node sends the slave node super node organization information to the master node according to the process number and receives the master node super node organization information sent by the master node.
And step 112, writing the super-junction point organization information into the corresponding super-junction point data structure to complete the construction of the software super-junction point.
And the super-node configuration process needs to write the received super-node organization information of the remote node into the corresponding super-node data structure by the daemon process every time the super-node organization information is received. When the master node receives and stores information such as the communication address of the slave node, and the slave node receives and stores information such as the communication address of the master node, the physical nodes on the array heterogeneous computing system can directly communicate through a daemon process running in a background, and the construction of the software supernode is completed.
In the method for dynamically constructing the software super node on the array heterogeneous computing system, a daemon process is started in a background of each physical node in the array heterogeneous computing system, information such as a host name, a node identity, a daemon process number, a communication port address and the like of each physical node is exchanged inside the super node through a super node configuration process and stored in a shared memory space under the daemon process to construct the software super node, after the construction of the software super node is completed, the super node configuration process exits, and the daemon process resides in the background and is responsible for managing information inside the super node. By the method, when hardware nodes are added or reduced in the array heterogeneous computing system, corresponding software nodes and shared memory space maintained by the daemon process can be dynamically constructed in the software supernode, the daemon process of the physical node is supported to communicate with other processes in the same node, and communication between the daemon process and any node in the supernode is also supported.
In one embodiment, a daemon process is started on each physical node in the physical node list, and whether the daemon process runs in the physical node is determined; if the daemon process runs in the physical node, ending the current daemon process, and exiting the program for starting the daemon process; if no daemon process runs in the physical node, creating a daemon process log file, writing a process number of the daemon process into the daemon process log file, and locking the daemon process log file; initializing a super-junction point data structure through a daemon process; further comprising: establishing and initializing a communication port through a daemon process to obtain a communication port address; establishing and initializing a shared memory space through a daemon process, wherein the shared memory space comprises parameters: communication state marks of the physical node daemon process and the super-junction point configuration process, identity judgment information of the physical nodes, average slave node number information, communication port addresses of the physical nodes and process numbers of the physical node daemon process; acquiring a daemon process number through a daemon process, writing the daemon process number into a shared memory space, and setting a communication state mark as a first state; and suspending the daemon process, so that the daemon process enters a wait wakeup state.
Specifically, S1: starting a daemon process, appointing a physical node list for running the daemon process, wherein the number of nodes is n, and executing the following steps on each node:
s1.1: starting a daemon process, and checking whether a/tmp/log _ daemon file exists in the system or not by the daemon process:
when a/tmp/log _ daemon file exists in the system, the fact that the daemon process exists in the node and is running is shown, resource management conflict can be caused when a plurality of copies are run, and therefore the current daemon process is ended, and the program exits;
when the/tmp/log _ daemon file does not exist in the system, the file is created, the process number pid of the daemon process is written into the/tmp/log _ daemon file, and the file is locked;
s1.2: the daemon process establishes a parallel initialization communication port to obtain a communication port address ep for communication between nodes in the software supernode;
s1.3: the daemon process creates and initializes a shared memory space for communicating with the super-junction point configuration process, wherein the shared memory space mainly comprises the following information:
s1.4: the daemon process converts the current process into the current process
Writing into shared memory, and storing the data in the shared memory
Setting to 0;
s1.5: daemon process initialized super-junction point data structure
The system is used for maintaining software super-junction point information;
s1.6: the daemon suspends itself and waits for a specific signal to wake up.
In one embodiment, the method further comprises the following steps: acquiring the number of preset master nodes and the number of preset slave nodes in a physical node list; wherein, the number of the master nodes and the number of the slave nodes meet the constraint:
wherein the content of the first and second substances,
representing the number of slave nodes;
indicating the number of main nodes;
representing the number of physical nodes in the physical node list;
is an integer;
obtaining the average number of the slave nodes under each master node according to the number of the master nodes and the number of the slave nodes as follows:
wherein the content of the first and second substances,
indicating average slave node number information.
Acquiring a super junction point configuration process number through a super junction point configuration process; when the super-junction configuration process number is smaller than the number of the main nodes, judging that the current physical node is the main node, and setting identity judgment information to be TRUE; when the super-junction configuration process number is not less than the number of the main junctions, judging that the current physical junction is a slave junction; and setting the identity judgment information to FALSE.
Specifically, S2: starting a supernode configuration process supernode, and designating the same node list as the first step, wherein the physical node number of the main node in the software supernode is in front of the supernode, and designating the number of the main nodes and the number of the slave nodes, wherein the number of the main nodes and the number of the slave nodes need to satisfy:
wherein the content of the first and second substances,
representing the number of slave nodes;
indicating the number of main nodes;
representing the number of physical nodes in the physical node list;
is an integer;
s3: the super junction point configuration process judges the identity of the physical node in which the super junction point configuration process is located in the super junction point;
s3.1: the super junction point configuration process obtains the process number of the super junction point configuration process
If, if
If the physical node is the main node in the super-junction, the main node will share the memory
Setting as TRUE, otherwise the physical node is slave node in supernode, sharing memory
Setting as FALSE;
s3.2: the super node configuration process obtains the number of average slave nodes in the super node
Comprises the following steps:
and will share memory
Setting as 1;
s3.3: the super junction point configuration process obtains the process number of the node daemon process from the shared memory
And according to the process number
Sending a pairing signal SIGUSR1 to the daemon;
s3.4: supernode configuration process loop waiting
Becomes 2.
In one embodiment, after the average slave node number information is acquired through the super node configuration process, the communication state flag is set to be in a second state; acquiring a daemon number of a daemon process corresponding to the physical node from the shared memory space through the super-junction point configuration process, and sending a pairing signal to the corresponding daemon process through the super-junction point configuration process according to the daemon number so that the daemon process receives the pairing signal, and when the communication state is judged to be marked as a second state, writing average slave node number information and identity judgment information in the shared memory space into a corresponding super-junction point data structure through the daemon process; acquiring a communication port address of the physical node through a daemon process, and marking the communication state as a third state after the communication port address is written into a shared memory space; and suspending the daemon process, so that the daemon process enters a wait wakeup state.
Specifically, S4: the daemon writes the super-junction point information into a super-junction point data structure;
s4.1: the daemon receives the SIGURSR 1 signal and judges
1, will share memory
And
variable write super junction data structure
；
S4.2: daemon process for communication port address
Write to the shared memory and will
Setting the value as 2;
s4.3: the daemon suspends itself again, waiting for a wake-up signal.
In one embodiment, after detecting that the communication state is marked as a third state through the supernode configuration process, acquiring a communication port address from a shared memory space corresponding to a physical node; splicing the host name, the process number and the communication port address of the physical node into a cache buffer of the supernode organization information; according to the identity judgment information, if the physical node is the main node, sequentially sending cache buffers to slave nodes in the super nodes, and sequentially receiving the cache buffers sent by the slave nodes; wherein, the process number of the slave node is:
wherein the content of the first and second substances,
a process number representing a slave node;
the process number representing the super junction point configuration process,
；
if the physical node is a slave node, sending a cache buffer to a main node in the supernode, and receiving the cache buffer sent by the main node; wherein, the process number of the main node is:
wherein the content of the first and second substances,
indicating the process number of the master node.
Specifically, S5: each node inside the super node exchanges communication port numbers and writes the communication port numbers into the shared memory, and the steps comprise:
s5.1: supernode configuration process discovery
Set to 2, obtain the communication port address of this node from the shared memory, and make the host name of this node and the process number of the daemon process
And communication port address
The cache buffers are spliced together to perform network transceiving according to the following rules:
when the physical node is a main node in the super nodes, the main node sequentially sends buffers to slave nodes in the super nodes according to the process number, and sequentially receives the buffers sent by the slave nodes, wherein the process number of the slave nodes is
Wherein
Is a positive integer;
when the physical node is a slave node in the supernode, the slave node sends a buffer to the master node according to the process number and receives the buffer sent by the master node, and the master nodeThe process number is
；
S5.2: the super-junction point configuration process analyzes the remote host name and the process number of the remote daemon process from the received buffer
And remote communication port address
Write to the shared memory, will
Setting to be 3, sending a pairing signal SIGUSR1 to a daemon process;
s5.3: supernode configuration process loop waiting
Becomes 0.
In one embodiment, the remote host name, the process number of the remote daemon process and the remote communication port address in the super node organization information are analyzed from the received cache buffer through the super node configuration process of the physical node, and the super node configuration information is written into the shared memory space; setting a communication state mark as a fourth state, and sending a pairing signal to a daemon process; after the daemon receives the pairing signal and judges that the communication state is marked as a fourth state, writing a remote host name, a process number of a remote daemon and a remote communication port address in the shared memory space into a super-junction data structure under a physical node; setting a communication state mark under a physical node as a first state; suspending the daemon process so that the daemon process enters a waiting awakening state; and after detecting that the communication state is marked as a first state, the super-junction point configuration process disconnects the shared memory space, exits the super-junction point configuration process, and completes the construction of the software super-junction point after all the super-junction point configuration processes of all the physical nodes in the physical node list exit.
Specifically, S6: the daemon writes the super-junction point organization information into a super-junction point data structure;
s6.1: the daemon receives the pairing signal SIGUSR1 and judges
3, the remote host name and the process number of the remote daemon process in the shared memory
And remote communication port address
Write super junction data structure
Completing the construction of the super junction point, and
setting to 0;
s6.2: the daemon suspends the daemon process again and waits for a wake-up signal;
s7: supernode configuration process discovery
And setting the state to be 0, disconnecting the connection with the shared memory and exiting.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in FIG. 2, there is provided an apparatus for dynamically building software supernodes on an array heterogeneous computing system, comprising: a physical node list obtaining module 202, an average slave node number information obtaining module 204, a daemon starting module 206, a super junction point configuration process running module 208, a super junction point organization information exchange module 210, and a super junction point data structure writing module 212, wherein:
a physical node list obtaining module 202, configured to obtain a physical node list of supernodes that need to be constructed in the array heterogeneous computing system; the super node is composed of a plurality of physical nodes in the array heterogeneous computing system.
The average slave node number information acquisition module 204 is used for acquiring a physical node list of the supernodes required to be constructed in the array heterogeneous computing system; the super node is composed of a plurality of physical nodes in the array heterogeneous computing system; and acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes.
A daemon starting module 206, configured to start a daemon on each physical node in the physical node list, so that the daemon creates and initializes the super junction data structure.
And a super node configuration process running module 208, configured to run a super node configuration process on the physical node, obtain identity judgment information for judging whether the physical node is a main node according to the super node configuration process number and the main node number of the super node configuration process, and write the average slave node number information and the identity judgment information into a super node data structure under a daemon process of the physical node.
A super node organization information exchange module 210, configured to sequentially receive, according to the identity determination information, super node organization information sent from the node when the physical node is determined to be the master node, and receive, when the physical node is determined to be the slave node, super node organization information sent from the master node; the super node organization information comprises a host name of the physical node, a process number of the daemon process and a communication port address.
And a super junction point data structure writing module 212, configured to write the super junction point organization information into the corresponding super junction point data structure, so as to complete software super junction point construction.
The average slave node number information obtaining module 204 is further configured to obtain the number of master nodes preset in the physical node list and the number of slave nodes preset in the physical node list; wherein, the number of the master nodes and the number of the slave nodes meet the constraint:
wherein the content of the first and second substances,
representing the number of slave nodes;
indicating the number of main nodes;
representing the number of physical nodes in the physical node list;
is an integer;
obtaining the average number of the slave nodes under each master node according to the number of the master nodes and the number of the slave nodes as follows:
wherein the content of the first and second substances,
indicating average slave node number information.
The super-junction configuration process running module 208 is further configured to obtain a super-junction point configuration process number through a super-junction point configuration process; when the super-junction configuration process number is smaller than the number of the main nodes, judging that the current physical node is the main node, and setting identity judgment information to be TRUE; when the super-junction configuration process number is not less than the number of the main junctions, judging that the current physical junction is a slave junction; and setting the identity judgment information to FALSE.
The super-node configuration process running module 208 is further configured to set the communication state flag to be the second state after obtaining the average slave node number information through the super-node configuration process; acquiring a daemon number of a daemon process corresponding to the physical node from the shared memory space through the super-junction point configuration process, and sending a pairing signal to the corresponding daemon process through the super-junction point configuration process according to the daemon number so that the daemon process receives the pairing signal, and when the communication state is judged to be marked as a second state, writing average slave node number information and identity judgment information in the shared memory space into a corresponding super-junction point data structure through the daemon process; acquiring a communication port address of the physical node through a daemon process, and marking the communication state as a third state after the communication port address is written into a shared memory space; and suspending the daemon process, so that the daemon process enters a wait wakeup state.
The super-node organization information exchange module 210 is further configured to obtain a communication port address from the shared memory space corresponding to the physical node after detecting that the communication state is marked as a third state through the super-node configuration process; splicing the host name, the process number and the communication port address of the physical node into a cache buffer of the supernode organization information; according to the identity judgment information, if the physical node is the main node, sequentially sending cache buffers to slave nodes in the super nodes, and sequentially receiving the cache buffers sent by the slave nodes; wherein, the process number of the slave node is:
wherein the content of the first and second substances,
a process number representing a slave node;
the process number representing the super junction point configuration process,
；
if the physical node is a slave node, sending a cache buffer to a main node in the supernode, and receiving the cache buffer sent by the main node; wherein, the process number of the main node is:
wherein the content of the first and second substances,
indicating the process number of the master node.
The software super-junction point constructing module 212 is further configured to analyze a remote host name, a process number of a remote daemon process, and a remote communication port address in the super-junction organization information from the received cache buffer through a super-junction point configuration process of the physical node, and write the remote host name, the process number, and the remote communication port address into the shared memory space; setting a communication state mark as a fourth state, and sending a pairing signal to a daemon process; after the daemon receives the pairing signal and judges that the communication state is marked as a fourth state, writing a remote host name, a process number of a remote daemon and a remote communication port address in the shared memory space into a super-junction data structure under a physical node; setting a communication state mark under a physical node as a first state; suspending the daemon process so that the daemon process enters a waiting awakening state; and after detecting that the communication state is marked as a first state, the super-junction point configuration process disconnects the shared memory space, exits the super-junction point configuration process, and completes the construction of the software super-junction point after all the super-junction point configuration processes of all the physical nodes in the physical node list exit.
For the specific limitation of the device for dynamically constructing the software supernode on the array heterogeneous computing system, reference may be made to the above limitation on the method for dynamically constructing the software supernode on the array heterogeneous computing system, and details are not described here. All or part of each module in the device for dynamically constructing the software super node on the array heterogeneous computing system can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 3. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program when executed by a processor implements a method for dynamically building software supernodes on an array heterogeneous computing system. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 3 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for dynamically building software supernodes on an array heterogeneous computing system, the method comprising:
acquiring a physical node list of a supernode required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system;
acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes;
starting a daemon process on each physical node in the physical node list so that the daemon process creates and initializes a super-junction data structure;
running a super-junction point configuration process on the physical node, obtaining identity judgment information for judging whether the physical node is a main node according to the super-junction point configuration process number and the main node number of the super-junction point configuration process, and writing the average slave node number information and the identity judgment information into the super-junction point data structure under the daemon of the physical node;
according to the identity judgment information, when the physical node is judged to be a master node, super-junction point organization information sent by the slave node is sequentially received, and when the physical node is judged to be a slave node, super-junction point organization information sent by the master node is received; the super node organization information comprises a host name of the physical node, a process number of a daemon process of the physical node and a communication port address of the physical node;
and writing the super-node organization information into a corresponding super-node data structure to complete the construction of the software super-node.
2. The method according to claim 1, wherein obtaining a preset number of master nodes and a preset number of slave nodes in the physical node list, and obtaining information on an average number of slave nodes per master node according to the number of master nodes and the number of slave nodes comprises:
acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list; wherein the number of master nodes and the number of slave nodes satisfy the constraint of:
wherein the content of the first and second substances,
representing the number of the slave nodes;
representing the number of the main nodes;
representing the number of physical nodes in the physical node list;
is an integer;
obtaining the average number of the slave nodes under each master node according to the number of the master nodes and the number of the slave nodes as follows:
wherein the content of the first and second substances,
representing the average slave node number information.
3. The method of claim 1, wherein starting a daemon on each physical node in the list of physical nodes to cause the daemon to create and initialize a superjunction data structure comprises:
starting a daemon process on each physical node in the physical node list, and determining whether the daemon process runs in the physical node;
if the daemon process runs in the physical node, ending the current daemon process, and exiting the program for starting the daemon process;
if no daemon process runs in the physical node, creating a daemon process log file, writing a process number of the daemon process into the daemon process log file, and locking the daemon process log file;
initializing the super junction point data structure through the daemon process;
the method further comprises the following steps:
establishing and initializing a communication port through the daemon process to obtain a communication port address;
creating and initializing a shared memory space through the daemon process, wherein the shared memory space comprises parameters: the communication state marks of the physical node daemon and a super junction point configuration process, the identity judgment information of the physical node, the average slave node number information, the communication port address of the physical node and the process number of the physical node daemon;
acquiring a daemon process number through the daemon process, writing the daemon process number into the shared memory space, and setting the communication state mark as a first state;
suspending the daemon process so that the daemon process enters a wait wakeup state.
4. The method according to claim 3, wherein the step of running a super junction configuration process on the physical node and obtaining identity judgment information for judging whether the physical node is a master node according to the process number of the super junction configuration process and the number of the master nodes comprises:
acquiring a super junction point configuration process number through the super junction configuration process;
when the super node configuration process number is smaller than the number of the main nodes, judging that the current physical node is the main node, and setting the identity judgment information as TRUE;
when the super node configuration process number is not less than the number of the main nodes, judging that the current physical node is a slave node; and setting the identity judgment information as FALSE.
5. The method of claim 4, wherein writing the average slave node number information and the identity determination information into the super junction point data structure under the daemon of the physical node comprises:
after the information of the average slave node number is obtained through the super junction point configuration process, setting the communication state mark as a second state;
acquiring a daemon number of a daemon corresponding to the physical node from the shared memory space through the super-junction point configuration process, and sending a pairing signal to the corresponding daemon according to the daemon number through the super-junction point configuration process so that the daemon receives the pairing signal and writes the average slave node number information and the identity judgment information in the shared memory space into a corresponding super-junction point data structure when judging that the communication state is marked as a second state;
acquiring the communication port address of the physical node through the daemon process, and marking the communication state as a third state after writing the communication port address into a shared memory space;
suspending the daemon process so that the daemon process enters a wait wakeup state.
6. The method according to claim 5, wherein according to the identity determination information, when the physical node is determined to be a master node, sequentially receiving superjunction organization information sent by the slave node, and when the physical node is determined to be a slave node, receiving superjunction organization information sent by the master node, comprises:
after detecting that the communication state is marked as a third state through the super junction point configuration process, acquiring the communication port address from the shared memory space corresponding to the physical node;
splicing the host name, the process number and the communication port address of the physical node into a cache buffer of supernode organization information;
according to the identity judgment information, if the physical node is a main node, sequentially sending the cache buffers to slave nodes in the super nodes, and sequentially receiving the cache buffers sent by the slave nodes; wherein, the process number of the slave node is:
wherein the content of the first and second substances,
a process number representing the slave node;
the process number representing the super junction point configuration process,
；
if the physical node is a slave node, sending the cache buffer to a master node in the super node, and receiving the cache buffer sent by the master node; wherein, the process number of the main node is:
wherein the content of the first and second substances,
a process number representing the master node.
7. The method of claim 6, wherein writing the superjunction organization information into the corresponding superjunction data structure to complete software superjunction point construction, comprises:
resolving a remote host name, a process number of a remote daemon process and a remote communication port address in the super node organization information from the received cache buffer through a super node configuration process of the physical node, and writing the remote host name, the process number of the remote daemon process and the remote communication port address into the shared memory space;
setting the communication state flag to be a fourth state, and sending a pairing signal to the daemon process;
after the daemon process receives the pairing signal and judges that the communication state is marked as a fourth state, the remote host name, the process number of the remote daemon process and the remote communication port address in a shared memory space are written into a super-junction data structure under the physical node;
setting the communication state mark under the physical node to be a first state;
suspending the daemon process so that the daemon process enters a wait wakeup state;
and after detecting that the communication state is marked as a first state, the super node configuration process disconnects the shared memory space, quits the super node configuration process, and completes the construction of the software super node after all the super node configuration processes of all the physical nodes in the physical node list quit.
8. An apparatus for dynamically building a software super junction point on an array heterogeneous computing system, the apparatus comprising:
the physical node list acquisition module is used for acquiring a physical node list of the supernode required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system;
the average slave node number information acquisition module is used for acquiring a physical node list of the super nodes required to be constructed in the array heterogeneous computing system; the super node consists of a plurality of physical nodes in the array heterogeneous computing system; acquiring the number of preset main nodes and the number of preset slave nodes in the physical node list, and acquiring the average slave node number information under each main node according to the number of the main nodes and the number of the slave nodes;
the daemon starting module is used for starting a daemon on each physical node in the physical node list so as to enable the daemon to create and initialize a super-junction data structure;
a super-node configuration process running module, configured to run a super-node configuration process on the physical node, obtain identity judgment information for judging whether the physical node is a main node according to a super-node configuration process number and the main node number of the super-node configuration process, and write the average slave node number information and the identity judgment information into the super-node data structure under the daemon process of the physical node;
the super-node organization information exchange module is used for sequentially receiving super-node organization information sent by the slave node when the physical node is judged to be the master node according to the identity judgment information, and receiving the super-node organization information sent by the master node when the physical node is judged to be the slave node; the super node organization information comprises a host name of the physical node, a process number of a daemon process of the physical node and a communication port address of the physical node;
and the super-junction point data structure writing module is used for writing the super-junction point organization information into the corresponding super-junction point data structure to complete the software super-junction point construction.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011396788.4A CN112199326B (en) | 2020-12-04 | 2020-12-04 | Method and device for dynamically constructing software supernodes on array heterogeneous computing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011396788.4A CN112199326B (en) | 2020-12-04 | 2020-12-04 | Method and device for dynamically constructing software supernodes on array heterogeneous computing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112199326A CN112199326A (en) | 2021-01-08 |
| CN112199326B true CN112199326B (en) | 2021-02-19 |
Family
ID=74033762
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011396788.4A Active CN112199326B (en) | 2020-12-04 | 2020-12-04 | Method and device for dynamically constructing software supernodes on array heterogeneous computing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112199326B (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1324470C (en) * | 2004-03-16 | 2007-07-04 | 联想(北京)有限公司 | Method for executing parallel virtual machine process in Quadrics network |
| US8293207B2 (en) * | 2009-09-18 | 2012-10-23 | Valery Zavarzin | Radiometal-labeled amino acid analogs, imaging and therapeutic agents incorporating the same, and methods using the same |
| CN101938514A (en) * | 2010-08-26 | 2011-01-05 | 青海西部矿业科技有限公司 | Heterogeneous multi-channel wireless sensor network based underground monitoring system |
| CN103098015B (en) * | 2010-09-30 | 2015-11-25 | 日本电气株式会社 | Storage system |
| CN102323917B (en) * | 2011-09-06 | 2013-05-15 | 中国人民解放军国防科学技术大学 | Shared memory based method for realizing multiprocess GPU (Graphics Processing Unit) sharing |
| CN108846000A (en) * | 2018-04-11 | 2018-11-20 | 中国科学院软件研究所 | A kind of common sense semanteme map construction method and device based on supernode and the common sense complementing method based on connection prediction |
| CN110611672B (en) * | 2019-09-17 | 2021-08-13 | 中国人民解放军战略支援部队信息工程大学 | Network space safety protection method, server equipment, node equipment and system |
| CN110955535B (en) * | 2019-11-07 | 2022-03-22 | 浪潮(北京)电子信息产业有限公司 | Method and related device for calling FPGA (field programmable Gate array) equipment by multi-service request process |
-
2020
- 2020-12-04 CN CN202011396788.4A patent/CN112199326B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112199326A (en) | 2021-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102074468B1 (en) | A computer cluster arragement for processing a computation task and method for operation thereof | |
| CN107959582B (en) | Slice instance management method and device | |
| US10943324B2 (en) | Data processing method, apparatus, and electronic device | |
| US20140245295A1 (en) | Providing Dynamic Topology Information in Virtualized Computing Environments | |
| KR102386495B1 (en) | Data accessing method and apparatus, device and medium | |
| CN103605577A (en) | Cross-process resource sharing method and equipment | |
| CN111506386A (en) | Virtual machine online migration method, device, equipment and computer readable storage medium | |
| EP2128759A1 (en) | Starting-up control method for operating system and information processing device | |
| US20110265093A1 (en) | Computer System and Program Product | |
| WO2020052379A1 (en) | Method and apparatus for processing metadata of object in distributed storage system | |
| WO2021042733A1 (en) | Blockchain transaction processing method and apparatus, computer device, and storage medium | |
| CN101216781B (en) | Multiprocessor system, device and method | |
| CN110187923A (en) | A kind of CPU starting method and apparatus applied to multi -CPU board | |
| CN111159090B (en) | Information processing method and device and electronic equipment | |
| CN104714792A (en) | Multi-process shared data processing method and device | |
| CN112199326B (en) | Method and device for dynamically constructing software supernodes on array heterogeneous computing system | |
| CN116662039B (en) | Industrial information parallel detection method, device and medium based on shared memory | |
| CN116450184B (en) | System upgrading method and device, electronic equipment and storage medium | |
| CN115114042A (en) | Storage data access method and device, electronic equipment and storage medium | |
| CN117130571A (en) | Display method, device, chip and storage medium based on multi-core heterogeneous system | |
| US20130007768A1 (en) | Atomic operations on multi-socket platforms | |
| US9690619B2 (en) | Thread processing method and thread processing system for setting for each thread priority level of access right to access shared memory | |
| CN116010093A (en) | Data processing method, apparatus, computer device and readable storage medium | |
| CN114741165A (en) | Processing method of data processing platform, computer equipment and storage device | |
| CN112433869B (en) | Software supernode-oriented OpenCL programming framework construction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |