RU2744591C2 - Method and system for controlling computer communications in multilevel composite computer cluster - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: invention relates to computer engineering. Method of controlling computer connections in a multilevel composite computer cluster comprising computer controllers - objects ^kO_i and communication modules ^kMS, characterized by that at each hierarchical level equal to k = 0 and more, leader - one of objects ^kO_i, directly connected to communication module ^kMS of hierarchy level k, sends command through ^kMS to communication modules of next level of hierarchy ^k+1MS and directly connected to them objects ^k+1O_i, wherein ^k+1MS modules are sent in ^kMS signal of beginning of barrier synchronization and interrupt transmission of signals from ^kMS to objects ^k+1O_i, interacting only with their ^k+1MS under control of one of objects ^k+1O_i - leader of level hierarchy k+1, which, after completion of actions specified by command, stops transmission from ^k+1MS to ^kMS of barrier signal and restores communication between ^kMS and ^k+1O_i.

EFFECT: technical result consists in wider range of the same purpose tools.

2 cl, 5 dwg

Description

The invention relates to the field of computer technology, in particular, to methods and technical means for organizing the interaction of computers in computer clusters.

A multilevel composite computer cluster performing distributed computing by network means is understood as a cluster consisting of groups of simple computer clusters. At the same time, in a simple cluster, its computers exchange messages using tools that belong only to this cluster. In a composite cluster, computers belonging to different simple clusters for messaging must use means external to these clusters. The multilevel structure of a composite cluster means the ability to combine simple clusters into a hierarchical structure so that the clusters higher in the hierarchy can control the actions of the lower-standing clusters, and for any two simple clusters there is a cluster hierarchy, with the help of which the messages of the specified clusters will be exchanged.

Modern supercomputers and computer clusters use groups of computers that, on their own initiative, transmit messages and tasks to perform distributed computing to other computers in the system. The execution of such calculations is used directly by network means (In-Network Computing), which for this purpose contain computing means - network computers. It is also possible to carry out a number of calculations by network tools that do not have network computers (see Computer network with a fast distributed restructuring of its structure and data processing in the process of their transfer, Control problems No. 1. 2017. P. 47-56; Network information and computational support interaction of mobile robots, Control problems No. 5. 2018. pp. 56-65.)

The second publication is taken as a prototype for the method of the present invention. The prototype is focused on the systems of mobile robots with computers, in which computers are connected by wireless non-directional optical or radio channels and communication modules.

The solutions proposed in the patent are focused on providing the capabilities of the prototype, but for systems with a different structure. Therefore, we will describe in more detail the main features of the prototype.

In the prototype, each computer contains a network controller for managing the exchange of messages with other computers in the system; contains a communication module and directional wireless optical or radio communication channels of computers with a communication module. Each network controller contains a source and a receiver for optical or radio signals. The network controller also contains a device for controlling the interaction of the computer with the sources and receivers of messages in the system.

The purpose of the present invention is to develop a method and system for managing communications between computers in a multi-level composite computer cluster performing distributed computing by network means.

The following components of the prototype cluster structure and the main operations in which they participate are important for the proposed in the patent management of computers in a multilevel composite computer cluster. These components are a simple cluster structure, network nodes, a communication module. Operations - the procedure for starting a cluster, synchronization, resolution of access conflicts, asynchronous barrier synchronization, calculations in the network tools of the cluster.

In what follows, the following notation will be used. The communication module is designated MS. The technical means of a computer - a controller performing the interaction of computers by means of MS, will be called an object O _i .

A. Network structure of a simple cluster, network nodes - communication module, objects. Each computer in the proposed cluster contains a network controller, which is the first of the host types for the network — an object that represents that computer in the computer cluster.

FIG. 1 objects O _i are designated as 1. The object initiates and ensures the execution of all the necessary distributed operations in the cluster.

The second type of node is the MS communication module, referred to here as 2. There is only one communication module in the network. Objects transmit signals to the communication module via channels 3 and receive signals from the communication module via channels 4. Both types of channels can be directional and non-directional, wired and wireless, transmitting optical or radio signals. Interaction between objects via MS is carried out as follows.

Objects must send signals of frequency ₁ to transmit binary one and frequency ƒ _0. to transmit binary zero to the MS. The MS module converts these signals respectively into ^* _{сигналы 1} and ^* ƒ ₀ signals of other frequencies and sends them to all objects. These signals have the same duration for the entire network.

Objects use signals ƒ ₂ and ^* ƒ ₂ to control asynchronous processes. Signals ƒ ₂ send objects. Each ƒ ₂ has a duration required by a particular process. The MS module sends signals ^* ƒ ₂ objects if it receives at least one signal ƒ ₂ .

Cluster computers are stationary. Therefore, it is possible to determine the transit time of the signal between each cluster object and the MS once. This time is required to obtain high performance for all distributed computing. It can be omitted if fast distributed operations are not required.

B. Synchronization. The task of synchronization is delivery to MS of a group of object messages in the form of one general message containing messages of objects following one after another without time pauses between them, as well as in the form of one message with the coincidence of identical bits of object messages. There are three types of synchronization: special command synchronization, MS access conflict resolution, and barrier synchronization.

In the first variant, one of the objects sends a synchronization command via MS to all objects. After receiving it, the objects send messages to the MS with a delay, taking into account the distance of the object from the MS. The object knows the time interval T _max - the interval is not less than the propagation time of the signal between the most distant object and the MS. After receiving the command, the object sends a message with a delay D _i = T _max -T _i , where T _i is the signal propagation time between O _i and MS.

In this case, message bits with the same names arrive at the MS simultaneously from all objects, and all objects receive a common message from the MS - the result of overlapping signals from message objects.

In the second variant, in the absence of signals from the MS, the objects begin to transmit messages, detect a collision in the MS, and send a binary scale to the MS. The scale has the number of bits equal to the number of objects, and has a digit one in the bits corresponding to the objects that must send the message. Having received this scale from MS, the sequencing objects send messages. This removes the conflict for all conflicting objects.

In asynchronous barrier synchronization, the participating entities send signals to the MS at a frequency ƒ ₂ . The MS module returns a signal to the object with a frequency ^* ƒ ₂ . Upon completion of work, the object removes signal ƒ ₂ . The disappearance of the signal ^* ƒ ₂ in objects is the moment of synchronization.

C. Presentation of data in a cluster. The distributed data is processed either entirely in the communication module, or with a significant part of such calculations being performed in it.

For this, paraphase coding of binary digits and the representation of the digits of numbers in the form of logical scales are used.

In paraphase coding, the digit "one" is represented by the sequence of signals ƒ ₁ ƒ ₀ , the digit “zero” - by the sequence ƒ ₀ ƒ ₁ .

When using a logical scale, each digit of a number given in the base p number system is represented by a binary scale, the number of bits in which is equal to p. In the scale, only one bit, corresponding to the value of the digit, is equal to 1, and the rest are equal to 0. For example, for p = 10 and the digit 7, the scale is 001000000.

Examples of calculations.

D. Logical operations AND, OR. Objects synchronously transmit data bits to the MS using the paraphase representation of the binary digits 1 and 0. If the AND operation is performed, the combination of signals ƒ ₁ ƒ ₀ , which results from the superposition of signals in the MS, gives the operation result equal to 1, otherwise the result is equal to 0. For the OR operation, the combination ƒ ₀ ƒ ₁ gives the result of the operation 0, otherwise 1.

These operations are performed in the MS without signal delay; The settlement time does not depend on the number of participants in the operation. Logic gates are not used for calculations in MS.

E. Search for MAX and MIN. To calculate MAX, objects transmit to the MS the largest digit of their number, represented as a logical scale, and upon receipt of a scale with several units from the MS, the largest value of this digit is determined for all objects at the same time. The next digit is transmitted by the entities that passed the largest leading digit. MAX will be received after all digits of the numbers have been transmitted. To calculate MIN, objects determine the minimum values of the digits. Increasing the base p speeds up the operation because the number of message exchanges decreases.

F. Analog-to-digital operations. The prototype shows how, if there is an analog-to-digital converter of signal levels in the communication module, fast network summing operations are performed directly in the network module. The summation time does not depend on the number of objects participating in the operation.

The disadvantage of the method in the prototype is that it does not provide for the ability to divide all the computers in the system into hierarchically organized groups - clusters so that in each cluster independent (autonomous) data processing is simultaneously performed, with the exchange of messages within the cluster. This reduces the performance of such a system.

As a prototype of the system, the system given in the above article is taken - Network information and computational support for the interaction of mobile robots, Control problems No. 5. 2018.S. 56-65.

The disadvantage of the system in the prototype is that it does not allow the simultaneous use of several hierarchically linked cluster communication modules for simultaneous autonomous execution of work in several clusters with independent messaging in the cluster. This slows down the processing of data.

The object of the present invention for the method is to eliminate the disadvantage of the prototype - the impossibility of processing data simultaneously by several groups of computers interacting within the group and to develop for this a method for managing hierarchical connections of computers in a multi-level composite computer cluster using a group of communication modules to perform distributed computing by network means.

The technical result for the method is to include in it methods of interaction of computers with a group of communication modules and methods for managing hierarchical connections of computers in a multilevel composite computer cluster using a group of communication modules to perform distributed computing by network means.

The technical result of the method is achieved by the fact that at each level of the hierarchy equal to k = 0 and more, the leader, one of the objects ^k O _i , directly connected with the communication module ^k MS of the hierarchy level k, sends a command through the ^k MS to the communication modules of the next hierarchy level ^{k + 1} MS and directly related objects ^{k + 1} O _i , while the ^{k + 1} MS modules send the barrier synchronization start signal to the ^k MS and interrupt the signal transmission from ^k MS to the objects ^{k + 1} O _i associated with these ^{k + 1} MS, which is subsequently restored by a command from ^k O _i , and these ^{k + 1} O _i continue to work autonomously, interacting only with their ^{k + 1} MS under the control of one of the objects ^{k + 1} O _i - the leader of the hierarchy level k +1, which, after the completion of the actions given by the command, stops the transmission from ^{k + 1} MS to ^k MS of the barrier synchronization signal and restores communication between ^k MS and ^{k + 1} O _i, as a result of which ^k MS, ^{k + 1} MS, ^k O _i and ^{k +1} O _i begin to send signals to their associated direct about communication modules and objects and receive signals from them.

The objective of the present invention for a system of devices is to eliminate the indicated drawback of the system of devices of the prototype - the inability to use several communication modules simultaneously and to create an organized multi-level composite computer cluster using this group of communication modules to perform distributed computing by network means.

The technical result for a system of devices is that it solves the specified problem of creating a hierarchically organized multi-level composite computer cluster using a group of communication modules to perform distributed computing by network means.

The technical result for the system is achieved by the fact that at each level of the hierarchy k≥0, it contains communication modules ^k MS and controllers in computers - objects ^k O _i , and wireless or wired communication lines make connections between ^k MS and ^{k + 1} MS, respectively, between ^k MS and ^k O _i , between ^{k + 1} MS and ^{k + 1} O _i , and each of these devices has receivers of signals sent to it and sources of signals sent by them to receivers of these devices, and communication modules ^{k + 1} MS contain blocks , breaking and restoring communication between devices of hierarchy levels k and k + 1, and a control unit for them, and ^k O _i also contain a control unit for exchanging commands and signals with a computer.

The technical essence and principle of operation of the proposed system of devices are illustrated by drawings.

FIG. 1. Simple cluster.

FIG. 2. Composite cluster.

FIG. 3. The structure of the communication module.

FIG. 4. Block of isolation of objects of neighboring levels of the hierarchy.

FIG. 5. The structure of the object.

The devices shown in the drawings should be considered as examples of the implementation of the devices proposed in the patent.

Brief description of the proposed method.

The main feature of the method of managing computer connections in a multilevel composite computer cluster is as follows. At each level k of the hierarchy of a composite cluster, other than the upper zero level, one of the objects of level k through the ^k MS module sends a command to communication modules of levels greater than k, which for the duration of its operation isolates in the modules indicated by the command all objects directly connected to them from signals previous levels. This allows dedicated computers to act interconnected and isolated from other devices, increasing the overall performance of the composite cluster.

Brief description of the proposed system of devices.

The main feature of the computer communications control system in a multilevel composite computer cluster is that at each level of the hierarchy k≥0, it contains communication modules ^k MS, and in computers it contains controllers - objects ^k O _i . Connections between ^k MS and ^{k + 1} MS, between ^k MS and ^k O _i , between ^{k + 1} MS and ^{k + 1} O _{i are} made by wireless or wired communication lines. Each of these devices has receivers of signals sent to it and sources of signals sent to them. Communication modules ^{k + 1} MS contain blocks that break and restore communication between devices of hierarchy levels k and k + 1, and a control unit for them. Objects ^k O _i contain a control unit for exchanging commands and signals with a computer.

Detailed description of the proposed method.

The method is applied to a multilevel composite cluster containing a central cluster and subordinate clusters. It focuses on the development of the structure of FIG. 1. In a composite cluster, a method is considered that consists of two interacting methods - synchronous and asynchronous (autonomous).

Synchronous way of functioning of a composite cluster.

The organization of the interaction of the nodes of a composite cluster, consisting of the central and subordinate clusters, is shown in Fig. 2. The following components of the composite cluster in FIG. 2 are assigned the numbers indicated in the text below in parentheses.

, к модулю ¹MS_k-объекты ¹O_k.l,…, ¹O_k.m. Модули ¹MS подключены к ⁰MS также каналами (9).Here ⁰ MS (5) is the main zero-level communication module belonging to the central cluster. Objects ⁰ О ₁ ,…, ⁰ O _i (6) are directly related to ⁰ MS. Subordinate clusters of the first level ¹ O _i (8) with numbers i = 1, ..., k are also connected to ⁰ MS through their communication modules of the first level ¹ MS ₁ , ..., ¹ MS _k (7). Fiber optic channels (9) connect objects to first-level modules: to module ¹ MS ₁ - objects

, to module ¹ MS _k -objects ¹ O _kl ,…, ¹ O _km . Modules ¹ MS are connected to ⁰ MS also by channels (9).

For all objects of the composite cluster, their distance from ⁰ MS and the greatest distance of all objects from ⁰ MS are known.

MS Module ⁰ operates like the Simple Cluster MS Module described above. Modules ¹ MS ₁ receive signals ƒ ₀ , ƒ ₁ and ƒ _{2 of the} objects connected to them and, without changing the frequency of the signals, transmit them to ⁰ MS. Therefore ⁰ MS treats ¹ MS modules as objects connected to channels (9).

Module ⁰ MS sends signals ^* ƒ ₁ , ^* ƒ ₀ and ^* ƒ ₂ not only to objects, but also to modules ¹ MS. The latter, like ⁰ MS, send these signals to their objects.

Thus, ¹ MS modules perform only the functions of repeaters (repeaters) of signals received at ⁰ MS from objects. A system with ¹ MS modules acts as a system containing only ⁰ MS. The only difference is that signals from objects of any cluster do not go directly to objects of other clusters.

The synchronization process in a composite cluster needs to be refined. Let, for simplicity, the clusters use synchronization scales of the same number of binary digits for synchronization. The number of subordinate clusters is known. The synchronization process includes all clusters that transmit their scales in a common message. From ⁰ MS, the general scale will return to all objects. This will allow the cluster objects to synchronously send their messages of the same duration.

In this process, an additional condition must be met. Object messages should arrive without conflicts not only at ⁰ MS, but also at ¹ MS. Since all distances between objects ⁰ MS and ¹ MS are known, for this each object must send any of its messages with an additional delay, taking into account the distance between ⁰ MS and its ¹ MS, as well as the order of messages in ⁰ MS from other ¹ MS preceding in sync scale ¹ MS of this object.

In such a composite cluster, all communication methods and digital operations are similar to those of a cluster that has no subordinate clusters. It is also easy to organize work with scales and messages of different but known lengths. You can work with dynamically changing message lengths, but for this you need to specify these lengths in the synchronization scale. Instead, a normal sync timeline can be sent, followed by a shorter timeline sent only by entities that transmit messages. The second scale reports the length of the messages.

The synchronous method can be applied independently, but it is also included in the next stand-alone method.

An autonomous way of functioning of a composite cluster.

Unlike the previous section, objects connected to ¹ MS modules work autonomously in this cluster, without delaying the operation of objects in other clusters. Let's go back to a simple cluster first. Let one of the U objects in the cluster send the job to other objects. Each of these objects starts an asynchronous operation, sends a ƒ ₂ barrier synchronization ^{signal at 0} MS, and completes transmission ƒ ₂ upon completion of task U. The ⁰ MS module responds to ƒ _{2 by} sending a signal ^* ƒ ₂ objects and completes transmission ^* ƒ ₂ when the arrival is terminated to it signal ƒ ₂ . ^{If *} ƒ _{2 is} absent, the U object can send a new job.

To simplify the presentation, the actions of the ¹ MS module are further presented to simplify the presentation in the form of interaction of its two parts ¹ MS _a and ¹ MS _b . Although these parts can be physically implemented on fully or partially overlapping technical means, their logical functioning is different. Taking the central cluster of module ¹ MS to offline mode consists of the following steps.

Stage 1. Arbitrary central cluster object - U sends command C1. Module ¹ MS, having received the command C1, informs the objects connected to it about the forthcoming autonomous operation. Modules ¹ MS and their objects, which receive the C1 command, wait for the C2 command from the U object. C2 lists the names of the C2 recipient objects. For each object, an action or sequence of actions (program) that the object must perform is indicated. MS module ¹ performs stage 2.

Stage 2. After receiving C2, the ¹ MS module prohibits the passage of signals from ⁰ MS to the objects of this module ¹ MS and signals of these objects to ⁰ MS. Part ¹ MS _a sends a signal ƒ ₂ - a sign of barrier synchronization at ⁰ MS. ^{Signal *} ₂ is received from ⁰ MS module. The objects of the ^{MS module 1} will asynchronously execute the tasks specified in the C2 command, completing step 3.

Step 3. The job received from ⁰ MS selects an arbitrary ^* U object among the ¹ MS objects. The ^* U object must control the actions of the remaining objects of the module ¹ MS. The ^* U object acts like the U object of the central cluster. Object ^* U can also be selected by joint actions of ¹ MS objects.

Each object that receives a command from ^* U sends signal ƒ ₂ ^{to part 1} MS _b , and ¹ MS _b converts signal ₂ into a signal ^* ƒ ₂ sent to the objects of module ¹ MS.

After completion of work, the object turns off its signal ƒ ₂ . If there are no signals ₂ , the ¹ MS _b module turns off the signal ^* ƒ ₂ . In response, the ^* U object either continues to work with ¹ MS objects, or terminates and sends a command to part ¹ MS _a to complete the sending of signal ƒ ₂ to module ⁰ MS. Object ^* U also removes the isolation of its objects from the central cluster.

Step 4. When the ⁰ MS module stops receiving signals ƒ ₂ , it will stop transmitting the signal ^* ƒ _{2 to} objects and to ¹ MS modules. After completing stages 1-4, the results of calculations in ¹ MS modules remain in the objects connected to these modules and are not available in the objects connected to MS ^0. To access them, step 5 is performed.

Stage 5. It is assumed that in the C2 command, the ^* U object received the task to collect data from objects connected to ¹ MS, and to form from them the data necessary for objects connected to ⁰ MS. Object ^* U performs these actions. Similarly, object U collects all data from objects ^{through module 0 MS.}

Thus, dividing the ¹ MS module into parts ¹ MS _a and ¹ MS _b allows two asynchronous processes to interact. Part ¹ MS _a provides connection of the ¹ MS module as an object to ⁰ MS. Part ¹ MS _b allows objects connected to the ¹ MS module to work autonomously from all objects connected to ⁰ MS.

Slave modules ¹ MS compared to ⁰ MS must have additional hardware to perform the above actions.

Stages 1-5 can be modified to include the number of hierarchy levels k exceeding the above k = 0, 1. Let's introduce ^k MS modules with k> 1 and connect them to the system like connecting ^{MS modules 1 to} MS module ⁰ . If any communication module ^{k + 1} MS receives a command that breaks its connection with the higher communication module ^k MS in the hierarchy, then this ^{k + 1} MS in relation to the lower communication modules and objects acts like ¹ MS in relation to ⁰ MS. This creates a layered structure.

If modules and objects do not use an autonomous way of interaction at the current time, then their actions do not differ from those in a synchronous way of functioning of a composite cluster.

In a composite cluster, any computer can be replaced by a composite cluster.

Detailed description of the proposed system for managing communications of computers in a multilevel composite computer cluster.

The proposed system is presented for simplicity only at the zero and first levels of the hierarchy. It is shown in FIG. 2 and contains a communication module of the zero level ⁰ MS (5), communication modules of the first level ¹ MS ₁ ... ¹ MS _k (7), objects of the zero level O ₁ - O _i (6), objects of the first level ¹ O ₁ - O _m ( 8). The listed devices are connected by communication channels (9).

FIG. 3 shows the arrangement of modules ^k MS 7, the first and higher levels of the hierarchy.

The design of communication modules ⁰ MS and MS is not considered, since it follows directly from the design of modules 7. FIG. 3 on the left, common channel 9 is represented by channels 3 and 4, connecting the communication module 7 with the previous level of hierarchy, similarly on the right, channels 3 and 4 connect the communication module with the next level of hierarchy.

Communication module 7 contains blocks 13 and 11 of receivers of signals сигналов ₀ , ƒ ₁ , ƒ ₂ and ^* ƒ ₀ , ^* ƒ ₁ , ^* ƒ _2, respectively, converting them into electrical signals; blocks 12 and 14 of signal sources ƒ ₀ , ƒ ₁ , ƒ ₂ and ^* ƒ ₀ , ^* ƒ ₁ , ^* ƒ _2, respectively, forming them in response to the arrival of electrical signals; contains a block 10 for isolating objects of neighboring levels of the hierarchy, the organization of which is shown in FIG. four; contains control unit 15.

In block 15, its inputs 16, 20 are connected to the outputs of blocks 11, 13, respectively, outputs 17, 19 are connected to the inputs of blocks 12, 14, output 18 is connected to the control input of block 10, which is shown in FIG. 4 is designated as 21. Here, block 21 contains keys 22, 23, 25. To perform the method steps for isolating objects of neighboring levels of the hierarchy, block 15 controls the keys 22, 23. To break the connections, the signal sent to the input 24 (Fig. 4) opens 22, 23 and closes 25.

The prototype uses analog-to-digital operations using an ADC. To have such a possibility in the proposed system, a node 26 is introduced, which includes an ADC. If an analog-to-digital operation is requested from the output 13 to the node 26, then during the execution of this operation and the transmission of its result to the system, the node opens the key 25.

The construction of the object is shown in FIG. 5. Here 27 is an object, i.e. network controller of a computer included in one of the clusters; 28 - control unit; 29 - block of sources of signals ƒ ₀ , ƒ ₁ , ƒ ₂ , sent to the communication module via channel 3; 30 - block of receivers of signals ^* ƒ ₀ , ^* ƒ ₁ , ^* ƒ ₂ , received from the communication module via channel 4. Control unit 28 is connected to 29 and 30, and is also connected to the cluster computer by channels 31 and 32, through which commands and signals providing the action of the proposed method.

Claims (2)

1. A method for managing computer connections in a multi-level composite computer cluster containing computer controllers - objects ^k O _i and communication modules ^k MS, characterized by the fact that at each level of the hierarchy equal to k = 0 and more, the leader is one of the objects ^k O _i directly connected to the communication module ^k MS of the hierarchy level k sends a command via ^k MS to the communication modules of the next hierarchy level ^{k + 1} MS indicated in the command and to the objects ^{k + 1} O _i directly connected with them, while the ^{k + 1} MS modules are sent to ^k MS signal of the beginning of barrier synchronization and interrupt the transmission of signals from ^k MS to objects ^{k + 1} O _i associated with these ^{k + 1} MS, which is subsequently restored by a command from ^k O _i , and these ^{k + 1} O _i continue to work autonomously, interacting only with its own ^{k + 1} MS under the control of one of the objects ^{k + 1} O _i - the leader of the hierarchy level k + 1, which, after completing the actions specified by the command, stops the transmission of the barrier synchronization signal ^{from k + 1} MS to ^{k MS and restores} communication between ^k MS and ^{k + 1} O _i , as a result of which ^k MS, ^{k + 1} MS, ^k O _i and ^{k + 1} O _i begin to send signals to and receive signals from directly related communication modules and objects. 2. A control system for computers in a multilevel composite computer cluster, characterized by the fact that at each level of the hierarchy k≥0, it contains communication modules ^k MS and controllers in computers - objects ^k О _i , and wireless or wired communication lines are made, respectively, between ^k MS and ^{k + 1} MS, between ^k MS and ^k O _i , between ^{k + 1} MS and ^{k + 1} O _i , and in each of these devices there are receivers of signals sent to it and sources of signals sent by them to receivers of these devices, and in communication modules ^{k + 1} MS contain blocks that break and restore communication between devices of hierarchy levels k and k + 1, and a control unit for them, and ^k O _i also contain a control unit for exchanging commands and signals with a computer.

Images