JPH08314744A - Fault tolerant system - Google Patents

Abstract

PURPOSE: To provide the constitution of a fault tolerant system suitable for distributive arrangement. CONSTITUTION: This fault tolerant system comprises plural processing nodes 11-1n, an output selection node 20 which selects data with highest reliability out of the data that are the result of data processing in each processing node to provide them as an output signal 250, and a network 3 which connects them. The processing nodes 11-1n send fault generation information representing the generating status of a fault detected by self-diagnostic functions 121-12n to the output selection node 20 with the data respectively. The judging function 230 of the output selection node 20 judges the data with the highest reliability based on matching/unmatching between the data and the fault generation information from the respective processing nodes 11-1n, and outputs them to a selection function 240.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a fault tolerant system in which subsystems are multiplexed.

[0002]

2. Description of the Related Art As a technique for improving the reliability of a control computer system or the like, a system in which subsystems are multiplexed is used.
Ruto tolerant systems are known.

Further, in such a fault tolerant system, each multiplexed subsystem is made to execute the same processing, and the most probable processing result is selected from the processing results output by each subsystem to be controlled. Is being output to.

Techniques for selecting the most probable output result include a technique for determining the most probable processing result by majority logic and a technique for selecting the most probable processing result based on the self-diagnosis result of each subsystem. Are known.

Further, as described in JP-A-3-15946, etc., the reliability of the processing result of each subsystem is estimated based on the syndrome (fault occurrence state), and the processing result with the highest reliability is obtained. Techniques for selecting are known. In this technology, each subsystem is connected to each other and an output selection circuit that selects a processing result to be output to a control target among processing results output from each system and each subsystem by independent communication paths, and The output selection circuit selects the processing result to be finally output while exchanging information.

[0006] As a technique for improving reliability, a technique for increasing the multiplicity of hardware and a technique for multiplexing processing in one piece of hardware are known.

[0007]

In such a fault-tolerant system, an output process for selecting a processing result to be output to the control target from among the processing results output from each subsystem and each subsystem. It may be necessary to disperse the circuits over a wide area. However, such a distributed arrangement is difficult to realize in the technique of Japanese Patent Laid-Open No. 3-15946 mentioned above, because the required communication path becomes huge.

Further, the technique of estimating the reliability of the processing result of each subsystem and selecting the processing result with the highest reliability, as described in the above-mentioned Japanese Patent Laid-Open No. 3-15946, is based on the technique of each subsystem. It is based on the assumption that the internal configurations are the same and the coverage of the failure detection / failure recovery functions (probability that there is no failure detection / failure recovery omission) is the same.

However, an actual system may have a configuration in which old and new subsystems are mixed due to system expansion and the like. In such a case, the coverage of the failure detection / failure recovery function may differ for each subsystem. Similarly, when subsystems intentionally manufactured by different manufacturers are mixed for the purpose of the effect of design diversity, the internal configuration and the coverage of the failure detection / failure recovery function may differ for each subsystem. There is. In such a case, the technique described in the above-mentioned Japanese Patent Laid-Open No. 3-15946 may not always select the processing result with the highest reliability.

Further, according to the technique of multiplexing the processing in one piece of hardware described above, an error that is correlated between the multiplexed processing does not occur and the influence of the error is not localized, so that the processing is performed. The result selection may not be performed normally.

Therefore, an object of the present invention is to provide a fault tolerant system suitable for distributed arrangement of subsystems and output selection circuits.

Further, even when the coverage of the failure detection / failure recovery function differs for each subsystem, the processing result with the highest reliability can be selected from the processing results output by each subsystem. -To provide a tolerant system.

It is another object of the present invention to localize an error of each process so that the error does not propagate when the processes in one hardware are multiplexed.

[0014]

For example, in order to achieve the first object, the present invention provides, for example, a plurality of processing nodes that execute the same processing, and at least one output selection node, The processing node has a network connected to the plurality of processing nodes and an output selection node, and each of the processing nodes generates data processing means for executing the processing and processing performed by the data processing means. A self-diagnosis means for detecting or recovering a failure, data which is the processing result of the processing executed by the data processing means, and failure occurrence information indicating the status of failure detection or recovery of the self-diagnosis means are provided to the network. And a transmission means for transmitting, wherein the output selection node is provided between the respective data transmitted from the respective processing nodes to the network. A data comparison means for detecting a match / mismatch, a match / mismatch between the detected data, and a failure in each processing node represented by the failure occurrence information transmitted from each processing node to the network. Based on the state of detection or recovery, the determining means for determining the most reliable data among the respective data transmitted from each process node to the network, and the data determined to have the highest reliability. A fault tolerant system is provided.

[0015]

According to the fault tolerant system according to the present invention, a system configuration and information transmission method that take advantage of the characteristics of the network are adopted, and even in a system distributed over a wide area, a huge communication path is not required and a failure occurs. The reliability estimation based on the generation information makes it possible to select the most reliable data.

[0016]

EXAMPLES Examples of the fault tolerant system according to the present invention will be described below.

First, the first embodiment will be described.

FIG. 1 shows the configuration of a fault tolerant system according to the first embodiment.

As shown in the figure, in the fault tolerant system according to the first embodiment, a plurality of processing nodes 11 to 1n for performing data processing and data 141 as a result of data processing in each of the processing nodes 11 to 1n. To 14n, the most reliable data is selected, and the output selection node 20 to be given to the controlled object (not shown) as the output signal 250, the output selection node 20, and the processing nodes 11 to 1n are connected. It consists of the network 3.

The processing nodes 11 to 1n have data processing functions 111 to 11n, self-diagnosis functions 121 to 12n for detecting a failure occurrence state, and processing result transmission functions 131 to 131n.
13n respectively. Data processing function 111-
Data 141 to 11n that are the results of data processing in 11n
14n and failure occurrence information 151 to 15n indicating the occurrence status of the failure detected by the self-diagnosis functions 121 to 12n are transmitted to the network 3 by the processing result transmission functions 131 to 13n.
Is transmitted to another node through.

On the other hand, the output selection node 20 has a processing result collection function 210, a data comparison function 220, and a judgment function 23.
0, has a selection function 240.

However, the internal configuration of the processing nodes 11 to 1n and the output selection node 20 shows the functional configuration, and in terms of hardware, the processing nodes 11 to 1n and the output selection node 20 are MPU and It may be a computer having a general configuration including a memory. In this case, the processing node 11 shown in FIG.
1n and each function inside the output selection node 20 are realized as a process of software executed on the computer.

The processing result collection function 210 is provided with data 141 to 141 sent by each processing node on the network 3.
14n and failure occurrence information 151 to 15n are collected. The data comparison function 220 compares and collates the collected data 141 to 14n, and reports the result of the comparison and collation, that is, the match / mismatch information to the determination function 230. In the determination function 230, based on the comparison result of the data 141 to 14n and the collected failure occurrence information 151 to 15n, the data 14
Each of 1 to 14n estimates a correct probability (reliability of data), and reports to the selection function 240 which data has the highest estimated reliability. The selection function 240 selects the data to be output as the output signal 250 from the data 141 to 14n based on the report from the determination function 230.

The reliability of the data in the determination function 230 can be estimated by using the technique described in Japanese Patent Laid-Open No. 3-16946.

The method of estimating the reliability of data will be described below.

The data reliability Rd is the data 141 to 14
It depends on the Syndrome consisting of n match / mismatch and failure occurrence information 151 to 15n. Therefore, Syndrome has a high reliability Rd of data in the order of Syndrome1, Syndrome2, ... SyndromeL (L: Syndro
That is, Rd (Syndrome1)> Rd (Syndrome2)>...> Rd (SyndromeL) However, Rd (Syndromei): It can be ordered with the reliability of the output when Syndromei is observed.

Here, Syndrome 1 represents the best symptom, that is, no abnormality is detected, while Syndrome L
Indicates the worst symptom, that is, the abnormality is detected in all the check items. For example, if there are only two check items, failure information and data comparison, Syndrome
1 indicates that no abnormality is detected in the self-diagnosis and that the data matches the data output by another processing node. Syndrome L indicates that the abnormality is detected in the self-diagnosis and the data is in another processing node. Indicates that the data output by the driver is inconsistent.

By defining such a Syndrome, the existing Syndrome is searched by gradually decreasing the grade of the symptom from the best symptom as follows, and the Syndrome that exists is searched.
By detecting the Syndrome corresponding to the highest reliability Rd among the romes and selecting the data output by the processing node having that Syndrome, the data output from each processing node is the most Reliability Rd
You can choose one.

If any subsystem has Syndrome1 then select the subsystem (s) which has Syndrome1, else if any subsystem has Syndrome2 then select the subsystem (s) which has Syndrome2, ・ ・ ・ else if any subsystem has SyndromeL-1 then select the subsystem (s) which has SyndromeL-1, else output fail safe signal. For simplicity, data 141 to 14n are matched / mismatched and one self-diagnosis result, that is, normal / abnormal, two-stage (1-bit information) From failure occurrence information 151 to 15n
A method of estimating the reliability of this data will be specifically described by taking the case where Syndrome is configured as an example.

Now, the missing rate of the self-diagnosis of each processing node (probability that a processing node is abnormal but is erroneously determined to be normal, that is, 1-coverage) is Pdε, an erroneous data. The probability that the data will match will be Paε, and the probability that an error will occur in the processing node will be Pε.

In this case, the probability of erroneous data matching is considered to be the probability of random two data matching assuming that the error is random, and Paε = 2. -Nth power However, it is expressed as n: bit length of data. When data is collated by software in a computer system or the like, the bit length n of the normal data is used.
Often takes a large value, so that Paε≈0, and it can be considered that Paε << Pdε.

Therefore, considering these things, each Synd
The reliability Rd of the data corresponding to rome is ascending from the highest reliability Rd. (1) As a result of self-diagnosis, it is diagnosed as normal
If it matches that of one processing node

[0033]

[Equation 1]

(2) When the data matches those of the other k subsystems

[0035]

[Equation 2]

(3) When the result of self-diagnosis is normal

[0037]

(Equation 3)

[0038]

Accordingly, the judgment function 230 determines that the data 14
Based on the result of the comparison and collation of 1 to 14n and the collected failure occurrence information 151 to 15n, following the procedure below,
The most reliable data can be selected. However, in the first embodiment, it is assumed that the failure rate Pdε of self-diagnosis of each processing node and the probability Pε of error occurrence in each processing node are all equal.

(1) As a result of self-diagnosis, a processing node which is diagnosed as normal and whose data matches those of other processing nodes is regarded as normal, and the data is selected.

When there is no processing node satisfying the condition (1), (2) a processing node whose data matches that of another processing node is regarded as normal and the data is selected. .

When there is no processing node satisfying the condition (2), (3) a processing node which is diagnosed as normal as a result of self-diagnosis is regarded as normal and its data is selected.

If there is no processing node satisfying the condition (3), the judgment function 230 either (4) stops the output from the selection function 240 or outputs a normal output from the selection function 240. It outputs a signal warning that it cannot be obtained.

In the above description, the case where the failure occurrence information 151 to 15n consists of one self-diagnosis result has been described for simplification, but the failure occurrence information 151 to 15n is composed of a combination of a plurality of self-diagnosis results. If
The data reliability Rd is as follows.

[0045]

[Equation 4]

When the calculation of Rd is complicated as shown in the equation 4, the above-mentioned judgment algorithm is complicated and there are multiple steps. Therefore, when the number of stages is large, the reliability Rd
It is also possible to arrange the syndromes and the data to be selected in a table form in the descending order of, and select the data corresponding to the first matching syndrome.

Note that the self-diagnosis functions 121 to 12n of the processing nodes 11 to 1n cause the failure occurrence information 151 to 1n.
Computation is a self-diagnosis performed to generate 5n.
For example, in the case of a data system, (1) ECC (Error Correcting) added to RAM etc.
Code: Error correction code) Information on detection and correction of bit errors in stored data by the encoder / decoder (2) Detection results by the error detection function of the microprocessor (bus error, address error, etc.) (3) Execution result of the hardware function diagnosis program (4) Collation result of distributed arrangement data (data multiplexed in the same processing node) (processing is performed when a discrepancy is detected by collation) (The correction is performed by majority logic in the node.) (5) Error detection result by error detection function added to arithmetic unit of microprocessor Next, the processing result transmission functions 131 to 13n of the processing nodes 11 to 1n will be described. To do.

FIG. 2 shows the configuration of the processing result transmission function 131.

In the figure, the register 1311 stores the failure occurrence information 15 sent from the self-diagnosis function 121 each time a failure occurs.
1 is accumulated during the processing of the data processing function 111. The period of accumulation is from the data processing function 111 to the previous data 141
Is sent next time, that is, the data 141 of the corresponding time
It is the period until is sent. That is, the data 141
Of the register 1311 serves as a trigger 1315 for controlling the accumulation operation of the register 1311.

In the packet editing function 1312, when the processing of the data processing function 111 is completed and the data 141 is obtained, the data 141 and the failure occurrence information 151 stored in the register are collectively edited, and the packet 131 is edited.
It is set to 3 and is passed to the transmission function 1314. The transmission function 1314 transmits the content of the packet 1313 as a message through the network 3 according to a predetermined protocol.

In FIG. 3, the processing by the data processing function 111, the self-diagnosis function 121, the packet editing function 1312,
The time relation of the operation of the transmission function 1314 is shown.

As shown in the figure, a failure occurring during the processing by the data processing function 111 is detected by the self-diagnosis function 121 as failure occurrence information 151, and the register 13 is updated at any time.
It is stored in 11. When the processing by the data processing function 111 is completed, it is edited into a packet 1313 by the packet editing function 1312 based on the data 141 and the failure occurrence information 151, and is passed to the transmission function 1314.

The first embodiment of the present invention has been described above.

As described above, according to the first embodiment, in the distributed system connected by the network, the data 1
Information of match / mismatch between 41 to 14n, failure occurrence information 1
The reliability of the data can be estimated from 51 to 15n and the data with the highest reliability can be selected as the final output of the system.

The second embodiment of the present invention will be described below.

FIG. 4 shows the configuration of the fault tolerant system according to the second embodiment.

As shown in the figure, in the second embodiment, in the fault tolerant system according to the first embodiment described above, the processing nodes 11 to 1n are responsible for the reliability, configuration, and fault detection / recovery of the processing nodes. Categories 161 to 16i that output categories representing coverage
Is added.

With this configuration, in the present fault tolerant system, each processing node 11 to 11 is
The processing result transmission functions 131 to 13i of the 1n process the data 14
In addition to 1 to 14n and failure occurrence information 151 to 15n, categories 161 to 16n representing reliability of the processing node, configuration, and coverage of failure detection / recovery are transmitted to the output selection node 20 through the network 3.

On the other hand, the judgment function 2 in the output selection node 20
In 30, the probability that each of the data 141 to 14n is correct (data reliability) is estimated based on the comparison and comparison result of the data 141 to 14n, the failure occurrence information 151 to 15n, and the categories 161 to 16n, and the reliability of the data is The highest value is reported to the selection function 240. The selection function 240 uses the data 141 to 14 based on the report from the determination function 230.
The output signal 250 is selected from n.

Here, the reliability of the processing nodes of the categories 161 to 16n represents the probability Pε that an error will occur at the processing node, and the probability Pa that the erroneous data will match depending on the configuration of the processing node.
ε can be obtained, and the failure rate Pdεi of each self-diagnosis is obtained from the fault detection / recovery coverage of categories 161 to 16n.
Thus, the category 161 can be obtained in this way.
By sending 16n to 16n to the output selection node 20, the determination function 230 can obtain the reliability Rd of each data sent from each processing node by the same equation as the above-mentioned equation 4. it can.

In the second embodiment, unlike the first embodiment, not only the number of bits of data but also the diversification of the configuration of the processing node causes the erroneous data of the two processing nodes to be one. The probability Pa ε is calculated.
The more different the configurations of the two processing nodes that execute the same process, the lower the probability of making the same error. Therefore, the determination function 230 determines the configuration of the two processing nodes having the same data from the configuration of each received processing node. Diversity is calculated, and the degree of diversity is large Probability Pa
Determine so that ε is small. Further, the reliability of the data is obtained by calculating the term of the probability Pa ε in the expression (4), which is the k−1 power, and the probability Pa between that data and each data that matches the data.
It is calculated by replacing with the product of ε.

That is, in the second embodiment, in addition to the embodiment shown in FIG. 1, even when the reliability of each processing node, the type configuration, and the probability of error detection omission of the self-diagnosis function are different, The most reliable data can be selected. Therefore, according to this embodiment, a highly reliable system can be constructed by combining various processing nodes.

FIG. 5 shows the configuration of the processing result transmission function 131 of the processing nodes 11 to 1n in the second embodiment. As shown in the figure, in the configuration of the processing result transmission function 131 according to the first embodiment shown in FIG. 3, the category is also input to the packet edit 1312.

The packet edit 1312 is, for example, as shown in FIG.
The packet 1313 having the format shown in is edited.

That is, the packet 1313 is input to the fault occurrence information 15x, the category 16x, and the data 14x.
It consists of. Here, the order of the failure occurrence information 15x, the category 16x, and the data 14x is set.

Further, in this example, the failure occurrence information 15x
Indicates that parity error is detected / non-detected, 1-bit error is detected / non-detected by ECC, 2-bit error is detected / non-detected, self-diagnosis result is normal / abnormal, duplicate comparison / matching result is match / mismatch Composed of information. Again, category 16x
Is dual comparison / verification coverage, ECC coverage parity leverage, self-diagnosis coverage, hardware diversity,
It is composed of information on software diversity. The hardware diversity and the software diversity are, for example, the hardware configuration of the processing node and the version number representing the software configuration.

In the figure, an example is shown in which the reliability of each processing node is not transmitted. In this case, the probability P that an error will occur at the processing node
ε is fixed in the determination function of the output selection node. As described above, in the second embodiment, categories 161 to
16n, processing node reliability, configuration,
Only one or two of the fault detection / recovery coverage may be used, and only one or two of the probability Pε, the probability Paε, and the probability Pdεi may be variable for each processing node. .

Next, the transmission function 1314 receives such a packet 1313, creates a message with a header-31 added thereto, as shown in FIG. 7, and selects an output via the network 3. Send to node. The header-31 represents information such as a source node, a destination node, a message type, etc. The included information depends on the type and protocol of the network 3.

By the way, the packet 1313 edited by the packet edit 1312 may be constructed as shown in FIG. The format of FIG. 8 is data 14x, failure occurrence information 15
x and category 16x are formed in this order.

In this case, from the transmission function 1312, the message shown in FIG.
Will be transmitted to the internet.

Such a message is a message exchanged in a normal network, that is, a message composed of a header 31 and data 14x in this order, followed by the second message.
The failure occurrence information 15x and category 16x peculiar to the embodiment are added. Therefore, even if there is a processing node in the system that does not have the function of transmitting the failure occurrence information 15x or the failure occurrence information 15x and the category 16x to the network, the output selection node 20 will be able to detect the processing node concerned. If only the first header 31 and the data 14x are referred to and the subsequent failure occurrence information 15x and the category 16x are ignored, it can be handled in the same manner as other processing nodes. That is, it is possible to allow the conventional processing nodes to be mixed in the system.
8 and 9, the same effect can be obtained when the order of the failure occurrence information 15x and the category 16x is opposite.

The categories omitted from the messages of FIGS. 7 and 9 correspond to the messages transmitted by each processing node in the first embodiment.

The third embodiment of the present invention will be described below.

FIG. 10 shows the configuration of the fault tolerant system according to the third embodiment.

In the third embodiment, the processing nodes 11 to 1n have a plurality of tasks 1 (41) to m (4).
m) is executed. Further, output selection nodes 21 to 2m are provided corresponding to each task.

In FIG. 10, the processing nodes 11 to 1 are shown.
Processing nodes 11 to 1i out of n are task 1 (4
1) is executed, and the processing nodes 1j to 1n are executing the task m (4m).

The output selection nodes 21 to 2m are provided corresponding to the tasks 1 (41) to m (4m), respectively, and are selected from the outputs of the processing nodes executing the corresponding tasks. The data with the highest reliability is selected and set as the outputs 251 to 25 m.

As an example, the operation of the output selection node 21 which outputs 251 the data of task 1 (41) will be described. Processing node 1 executing task 1 (41)
Data 141 to 14i from 1 to 1i, fault occurrence information 1
51 to 15i and categories 161 to 16i are transmitted through the network 3 and selected by the processing result collecting function in the output selection node 21. In the judgment function 231, as a result of comparison and collation of the data 141 to 14i, the failure occurrence information 1
Based on 51 to 15i and categories 161 to 16i, the probability that each of the data 141 to 14i is correct (data reliability) is estimated, and whether the data reliability is highest is reported to the selection function 241. The selection function 241 is the judgment function 23.
Based on the report from No. 1, the output signal 251 is selected from the data 141 to 14n.

According to the above-mentioned embodiment, it is possible to realize a multi-functional and highly reliable distributed system by utilizing the characteristics of the network without the need for a huge communication path.
Further, the network may be made redundant so that the entire system is not affected by the network failure.

The fourth embodiment of the present invention will be described below.

FIG. 11 shows the configuration of a fault tolerant system according to the fourth embodiment.

As shown in the figure, the fault tolerant system according to the fourth embodiment is a fault tolerant system according to the third embodiment.
In the configuration of the lto tolerant system (see FIG. 10), the processing result collecting functions 131 ′ to 13 n ′ and the data comparison functions 171 to 1 are included in each of the processing nodes 11 to 1 n.
7n, execution task determination functions 181 to 18n are added.

In such a configuration, each of the processing nodes 11 to 1n determines the task to be executed by itself,
Run.

By the way, the processing result collecting functions 131 'to 13n' of each processing node are processed by the processing node 1
Data 141 transmitted from 1 to 1n through the network
To 14i, failure occurrence information 151 to 15i, category 16
Collect 1-16i. Data comparison function 171 to 17n
Compares and collates the collected data 141 to 14n,
The result of comparison and matching, that is, the match / mismatch is reported to the execution task determination functions 181 to 18n. Execution task decision function 18
1 to 18n are failure occurrence information 151 to 15i, category 1
61 to 16i and the data comparison function 171 to 17n, the reliability of each task is estimated based on the match / mismatch information between the data 141 to 14n, and the task with the lowest reliability is determined to be the task to be executed. , Start running.

That is, the probability Pε, the probability Paε, and the probability Pd
εi are all equal, N1 processing nodes are executing the same task, and failure information of N2 processing nodes indicates that no failure has occurred, and N3 processing nodes When the data of the processing nodes match, the product of the probabilities Pε of the N1 processing nodes, the product of the probabilities Pdεi of the N2 processing nodes with no failure, and the N3 processing nodes. -The product of the probability that the data of erroneous coincides is obtained as the probability that the processing result of the task is erroneous, and the value subtracted from this is used as the reliability of the task. Alternatively, the number of processing nodes executing the task may be considered. It can be considered that the greater the number of processing nodes executing the task, the higher the reliability of the task.

Then, for each processing node, the reliability value of each task is corrected by giving the calculated reliability of the task an offset value previously determined for each task. The offset value applied to the same task is made different for each processing node. For example, a processing node mainly responsible for this task is defined for each task, and in this processing node, this task in this processing node is set so that the reliability of this task becomes relatively small. Determine the offset value of. By doing this, each processing node
In this mode, since the reliability of the task mainly handled by the processing node is relatively low, this processing node does not execute other tasks in place of this task.

The execution task decision functions 181 to 18n of the respective processing nodes have their own processing nodes.
If there is a task whose reliability is lower than that of the task that is running on the computer, control the data processing function to execute this task.

At this time, the offset value applied to the same task is made different for each processing node. Therefore, if each offset value is set appropriately, all processing nodes can execute the same task. It is possible to prevent a task that is not executed from occurring.

The fifth embodiment of the present invention will be described below.

The basic construction of the fault tolerant system according to the fifth embodiment is the same as that of the fault tolerant system according to the third embodiment (see FIG. 10),
A plurality of task 1 (4
1) -Execute task m (4m). Further, output selection nodes 21 to 2m are provided corresponding to each task.

However, as shown in FIG. 12, the processing nodes except one of a plurality of processing nodes that execute the same task are shown in FIG.
2 processing node 1i. That is, the processing result collection function 13i ′ and the data processing function 11
i, data comparison function 191i, self-diagnosis function 12i, category 16i, processing result failure occurrence information comparison function 193i
And a comparison information step heart function 13i ″ constitutes a processing node.

FIG. 12 shows only the node which performs the processing related to the task 1 (41) among the nodes of the fault tolerant system according to the fifth embodiment. In the figure, the processing nodes 11 to 1i execute task 1 (41) by the data processing functions 111 to 11i,
The output selection node 21 selects the data with the highest reliability from the data obtained by the processing nodes 11 to 1i executing the corresponding task 1 and outputs the selected data 251.
It is a node for

However, the processing nodes 12 to 1 (i-1) are not shown in FIG.

Now, in FIG. 12, it is assumed that the processing nodes 12 to 1i have the same configuration as the illustrated processing node 1i.

In such a configuration, task 1 (4
Among the processing nodes 11 to 1i executing 1), the processing node 11 has the data 141 and the failure occurrence information 15 as in the case of the third embodiment.
1, the category 161 is transmitted as a message through the network 3 by the processing result transmission function 131. However, the transmission destinations are the output selection node 21 and the processing nodes 12 to 1i.

The processing nodes 12 to 1i transmit the transmitted data 141, the failure occurrence information 151, the category 16
1 and data 142 to 14i of the processing result of the processing of task 1 performed by its own data processing functions 112 to 11i, and failure occurrence information 152 generated by the self-diagnosis functions 122 to 12i.
Data comparison function 1 for 15i and categories 162 to 16i
912 to 191i and failure occurrence information comparison function 1932
Data comparison information 1922-192.
i, failure occurrence information comparison information 1942-194i output from comparison information transmission function 132 "-13i" selection node 21
Send to. The data comparison information 1922 to 1922 is information indicating a match or a difference between the data 14i and the data 141 obtained by the own data processing functions 112 to 11i.
Similarly, failure occurrence information comparison information 1942-194.
2 is self-diagnosis functions 122 to 12i and failure occurrence information 15
It is information indicating the match or difference with 1 and the match or difference between the categories 162 to 16i and the category 161.

Here, the operation will be described focusing on the processing node 1i in order to facilitate understanding. The processing node 1i transmits the data 141, the failure occurrence information 151, the category 161, and the processing result data 1 of itself.
i, the failure occurrence information 15i, and the category 16i are compared by the data comparison function 191i and the failure occurrence information comparison function 193i to generate the data comparison information 192i and the failure occurrence information comparison information 194i. The output selection node 21 is transmitted through the step 3.

The data comparison information 192i and the failure occurrence information comparison information 194i are information indicating that they are a match if the above-mentioned comparison results are coincident, and information indicating a difference if they are not coincident.

In the judgment function 231 in the output selection node 21, the failure occurrence information 151, the category 161, the data comparison information 1922 to 192i, and the failure occurrence information comparison information 194.
2 to 194i, processing nodes 12 to 1
If all of the data comparison information 1922 to 192i and the failure occurrence information comparison information 1942 to 194i sent from i indicate coincidence and the failure occurrence information 151 does not indicate occurrence of a failure, the data 141 is selected. Report to select function 240. Data comparison information 1922-192
i, the failure occurrence information comparison information 1942 to 194i all indicate coincidence, and the failure occurrence information 151 indicates the occurrence of a failure, the predetermined data for stopping the controlled object on the fail-safe side is displayed. Select function to output 240
Control.

If the data comparison information 1922-192
i, the failure occurrence information comparison information 194 to 194i does not represent a match, the data comparison information that does not represent a match and the difference represented by the failure occurrence information comparison information are used to process each of the processing nodes that sent these information. -Restore the data, failure occurrence information, and category, estimate the reliability of the data 141 and each restored data, and report which data has the highest reliability to the selection function 240, as described above. . The selection function 240 makes a selection based on the report from the determination function 230.

As described above, when no failure occurs, the processing nodes 12 to 1i have the data comparison information 1922 to 192i and the failure occurrence information comparison information 194.
2 to 194i, that is, data 141, failure occurrence information 15
Since only 1 and a match with the category 161 are transmitted, the amount of information exchanged via the network 3 can be reduced.

Further, the processing nodes 12 to 1i
, If they represent a match, the data comparison information 1922-19
If the transmission of 2i and the failure occurrence information comparison information 1942 to 194i is omitted, the number of times of message exchange itself is significantly reduced. In this case, the output selection node 21 is
When the data comparison information and the failure occurrence information comparison information are not received from a certain processing node, the above operation is performed assuming that the data comparison information indicating the coincidence and the failure occurrence information comparison information are received from the processing node which is not received. To do so.

The sixth embodiment of the present invention will be described below.

The sixth embodiment relates to the high reliability of the node as shown in the previous embodiment which is an element of the fault tolerant system.

FIG. 13 shows a node 50 according to the sixth embodiment.
Shows the configuration of. As shown in the figure, in this embodiment, node 5 is used.
The output selection device 7 is added to 0.

The node 50 is a computer having a general configuration including an MPU and a memory as will be described later, and has processes 1 (61) to N having the same functions realized by executing software. (6N) is internally executed in a time-sharing manner. The results of this processing 1 (61) to processing N (6N) are the input port 1 (711) to input port N of the output selection device 7.
It is input to (71N), and one processing result which seems to be correct is selected by the output selection circuit 72 and output.

The output selection circuit 72 selects and outputs the processing result by a simple majority logic.
No. 15946, etc., the processing result is selected and output by a majority decision of a fixed number of processing results in which no failure is detected by a predetermined failure detection function and MV (Modified Voter). Good.

According to the sixth embodiment, the effect of transient faults, which accounts for most of the generated faults, can be masked by the redundancy of the processing in the node without increasing the hardware.

Next, a seventh embodiment of the present invention will be described.

FIG. 14 shows the configuration of the processing node 50 according to the seventh embodiment.

As shown in the figure, in the seventh embodiment, the processing nodes are multiplexed with the nodes 5a to 5x, and the processing executed by each node is also multiplexed. The nodes 5a to 5x have the same structure as the node 50 of the sixth embodiment.

The output selection device 7 includes the multiplexed node 5a.
As described above, one output result is selected by the output selection device 7 from the results of the further multiplexed processing of .about.5x.

Here, when the majority of the processing results multiplexed in the same node are the same, it is understood that no error due to a transient fault has occurred in the processing results. Further, when the processing result matches the processing result of at least one of the other nodes, it can be seen that the hardware that is executing the processing has not failed, that is, the error due to the fixed fault has not occurred.

Therefore, in the seventh embodiment, when the majority of the processing results multiplexed in the same node match and at least one processing result of another node matches, the output is output. The output selection device 7 makes a selection. As a result, it is possible to always obtain a processing result that is not affected by any of the transient fault and the fixed fault, that is, a correct processing result.

According to the seventh embodiment, the effects of transient faults, which account for most of the generated faults, are masked by the multiplexing of processing in the node, and fixed as represented by the failure of the node hardware. The effects of faults are masked by node multiplexing. Since the multiplicity of processing is higher than the multiplicity of hardware, it is a strong rational configuration due to the more frequent transient faults that occur.

By the way, the minimum configuration of the node according to the seventh embodiment is as shown in FIG.

FIG. 15 shows an embodiment in which the nodes are duplicated and the processing is duplicated in each node, that is, the hardware is duplicated and the processing is quadrupled. When a transient fault occurs, the result of any one of process 1a, process 2a, process 1b, and process 2b is different, so the result that matches the other may be selected. However, during a fixed fault, the results of processing 1a and processing 2a do not match the results of processing 1b and processing 2b, and there are cases where normal output cannot be specified.
Therefore, in an application where safety is important, in such a case, an object controlled by the 73 output outputs an output for stopping the operation to the safe side. That is, the fail-safe operation is performed. On the other hand, if the usage rate is important, the result that matches the others may be selected as the output.

Here, an example of selection of the output selection circuit 72 suitable for the use where safety is important is shown.

The example of selection shown by the logical expressions below is an output for selecting one processing result to be output when a transient fault occurs and continuing the processing, and for stopping the operation to the safe side when a fixed fault occurs. This is a file-safe operation for selecting.

[0120]

[Equation 5] SEL_1a = (1a = 2a) AND ((1a = 1b) OR (1a = 2b))

[0121]

SEL — 2a = (1a = 2a) AND
((2a = 1b) OR (2a = 2b))

[0122]

SEL_1b = (1b = 2b) AND
((1a = 1b) OR (2a = 1b))

[0123]

[Formula 8] SEL_2b = (1b = 2b) AND ((1a = 2b) OR (2a = 2b))

[0124]

[Formula 9] Fail-Safe = NOT (SEL_1a OR SEL_1b) However, when SEL_1a is true: the result of the process 1a is selected.

When SEL_2a is true: The result of the process 2a is selected.

When SEL_1b is true: The result of the process 1b is selected.

When SEL_2b is true: The result of the process 2b is selected.

When Fail-Safe is true: Select the safe side stop output.

1a: Result of process 1a 2a: Result of process 2a 1b: Result of process 1b 2b: Result of process 2b As described above, according to the example, transients with relatively high frequency of occurrence occur due to increase in hardware. When a fault occurs, the process can be continued, and when a fixed fault that occurs relatively infrequently occurs, a fail-safe operation can be performed, so that it is possible to improve the operating rate and ensure safety.

The eighth embodiment of the present invention will be described below.

In the eighth embodiment, the sixth embodiment is applied to the output selection nodes shown in the first to fifth embodiments.

FIG. 16 shows the output selection node 20 in this case.
Shows the configuration of.

Data selection function of selected node, that is, processing result collection function 210 data comparison function 230, determination function 2
30, processing 1 (61) for multiplexing the selection function 240
It is the process N (6N).

By doing so, it is possible to improve the reliability of the data selection itself and, in turn, the reliability of the entire system.

Similarly, the processing result collection function 210, the data comparison function 230, the judgment function 230, and the selection function 240.
As the processing 1 (61) to processing N (6N) for multiplexing, the output selection nodes shown in the first to fifth embodiments are
The embodiment can also be applied.

Similarly, the multiplexing techniques of the sixth and seventh embodiments can be applied to the processing nodes of the above-mentioned first to fifth embodiments. In this case, for example, the entire processing function of the processing nodes shown in the sixth and seventh embodiments may be multiplexed.

Next, a ninth embodiment of the present invention will be described.

The ninth embodiment relates to the configuration in the node 50 which realizes the multiplexing of the processes in the nodes shown in the sixth, seventh and eighth embodiments.

The structure of the node 50 according to the ninth embodiment is shown in FIG.

As shown, the present node 50 is an MPU.
(Micro-Processing Unit) 501, Memory (MEM) 5
03, an interface (I / F) 504 are connected to each other via a bus 500. MPU50
An ordinary computer 1 executes processing according to software while reading and writing various data from and in the memory (MEM) 503. Input port 1 of output selection device 72
(711) to output port 1 (51) of the interface (I / F) 504 connected to the input port N (71N)
Different addresses are assigned to 1) to port N (51N).

As shown in FIG. 18, the MPU 501 has processing 1 (61), processing 2 (62) ,. ． ． Process N (6N) is sequentially executed, and process 1 (61) is executed each time it is executed.
~ Interface (I / F) 5 for the result of processing N (6N)
No. 04 output port 1 (511) to port N (51N). Further, the memory (MEM) 503 is divided and used for each of processing 1 (61) to processing N (6N). That is, the output port 1 (511) to port N that is different for each process
(51N), a different area of the memory 503 is used.

As a result, processing 1 (61) to processing N (6)
The independence of N) can be increased. FIG. 19 shows the address map of the memory (MEM) 503 and output port of the ninth embodiment.

According to the ninth embodiment described above, when the malfunction of the processor 501 is due to a transient fault, a normal output can be obtained by selecting a processing result that can be normally executed. it can. Further, since most of the factors causing the malfunction of the processor 501 are due to the transient fault, according to the present embodiment, it is possible to mask the influence of most of the fault without increasing the hardware.

Next, a tenth embodiment of the present invention will be described.

The tenth embodiment is also similar to the ninth embodiment.
It relates to the configuration of the node 50 which realizes the multiplexing of the processes in the nodes shown in the sixth, seventh and eighth embodiments.

The structure of the node 50 according to the tenth embodiment is shown in FIG.

The tenth embodiment is different from the ninth embodiment in that only one output port of the node interface 504 is used and the output value of the counter 506 is used for the input port of the output selector 7. 1 (711) to input port N (71
The point is that N) is switched.

The timer 505 outputs a signal every time one processing ends, and the counter 506 outputs a count value each time, and the currently executed processing is processing 1 (61) to processing N.
It recognizes which process (6N), and outputs the process number 509. The port selection device 507 selects the corresponding input port according to the processing number 509.

According to the tenth embodiment, the processor 50
Since the input port corresponding to the processing number can be selected even when 1 malfunctions, the output selection device 7 can be reliably operated even if the processor 501 malfunctions.
Therefore, when the malfunction of the processor 501 is due to a transient fault, a normal output can be obtained by selecting a processing result that can be normally executed.

Furthermore, if the processor 501 is reset by a signal from the timer 505 at each time when one processing ends, even if a transient fault occurs in one processing, it is after reset in the next processing, the influence of the transient fault occurs. Can be removed. Therefore, processing 1 (61) to N (6N)
It is possible to further increase the independence between them.

Next, an eleventh embodiment of the present invention will be described.

The eleventh embodiment is also similar to the ninth embodiment.
The present invention relates to the configuration in the node 50 which realizes the multiplexing of the processes in the nodes shown in the sixth, seventh and eighth embodiments.

FIG. 21 shows the structure of the node 50 according to the eleventh embodiment.

The eleventh embodiment differs from the ninth embodiment in that the address from the MPU 501 is forcibly converted into a different address area by the address conversion function 508 so that the memory 501 which is different for each process is forcibly processed. The point is that the area is used.

In the figure, the timer 505 outputs a signal every time one processing ends, the counter 506 counts each time, and the currently executed processing is processed 1 (6).
1) to the processing N (6N) are recognized, and the processing number 509 is output. The address translation function 508 follows the processing number 509 from the counter 506 and follows the MPU 501.
Translate addresses from.

FIG. 22 shows an example of conversion of the address conversion function. As shown in FIG. 22, the output value of the counter 506 is
If it is inserted at the position of the higher-order second higher-order bit of the address output from the MPU 501, as shown in FIG.
Even if the U501 repeats exactly the same processing every time, the memory area and output port to be used are forcibly converted according to the processing number as shown in the address map of FIG. However, the most significant bit am output from the MPU 501
Does not change when the memory 501 is used for each process and the output port is addressed.

According to the tenth embodiment, the processor 50
Since the input port corresponding to the processing number can be selected even when 1 malfunctions, the output selection device 7 can be reliably operated even if the processor 501 malfunctions.
Moreover, since the processing 1 (61) to the processing N (6N) can be executed by the same processing program, transparency to software (transparency) can be realized.

The respective embodiments of the present invention have been described above.

As described above, according to each of the embodiments of the present invention, it is possible to realize a highly reliable method and a redundant resource management method suitable for a wide-area distributed system that takes advantage of the characteristics of the network. Furthermore, it is possible to provide a system with high operating rate and high safety by increasing the amount of hardware required.

[0160]

As described above, it is an object of the present invention to provide a fault tolerant system suitable for distributed arrangement of subsystems and output selection circuits.

Further, even when the coverage of the failure detection / failure recovery function is different for each subsystem, the processing result with the highest reliability can be selected from the processing results output by each subsystem. -A rult tolerant system can be provided.

Further, according to the present invention, when processing in one piece of hardware is multiplexed, it is possible to localize so that an error of each processing does not affect other processing.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a fault tolerant system according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a processing result transmission function according to the first embodiment of the present invention.

FIG. 3 is a timing chart showing the operation of the processing result transmission function according to the first embodiment of the present invention.

FIG. 4 is a block diagram showing a configuration of a fault tolerant system according to a second embodiment of the present invention.

FIG. 5 is a block diagram showing a configuration of a processing result transmission function according to a second embodiment of the present invention.

FIG. 6 is a diagram showing a first format example of a packet used in the second embodiment of the present invention.

FIG. 7 shows a first message used in the second embodiment of the present invention.
It is a figure which shows the example of a format of.

FIG. 8 is a diagram showing a second format example of a packet used in the second embodiment of the present invention.

FIG. 9 is a second message used in the second embodiment of the present invention.
It is a figure which shows the example of a format of.

FIG. 10 is a block diagram showing a configuration of a fault tolerant system according to a third embodiment of the present invention.

FIG. 11 is a block diagram showing the configuration of a fault tolerant system according to a fourth embodiment of the present invention.

FIG. 12 is a block diagram showing the configuration of a fault tolerant system according to a fifth embodiment of the present invention.

FIG. 13 is a block diagram showing a node configuration of a fault tolerant system according to a sixth embodiment of the present invention.

FIG. 14 is a block diagram showing a first structural example of a node of a fault tolerant system according to a seventh embodiment of the present invention.

FIG. 15 is a block diagram showing a second configuration example of the node of the fault tolerant system according to the seventh embodiment of the present invention.

FIG. 16 is a block diagram showing a configuration of an output selection node of a fault tolerant system according to an eighth embodiment of the present invention.

FIG. 17 is a block diagram showing an internal configuration of a fault tolerant system according to a ninth embodiment of the present invention.

FIG. 18 is a diagram showing a procedure of processing performed by a node of the fault tolerant system according to the ninth embodiment of the present invention.

FIG. 19 is a diagram showing an address map in the node of the fault tolerant system according to the ninth embodiment of the present invention.

FIG. 20 is a block diagram showing an internal configuration of a fault tolerant system according to a tenth embodiment of the present invention.

FIG. 21 is a block diagram showing an internal configuration of a fault tolerant system according to an eleventh embodiment of the present invention.

FIG. 22 is a diagram showing the configuration of a node address conversion function of the fault tolerant system according to the eleventh embodiment of the present invention.

FIG. 23 is a diagram showing a processing procedure performed by a node of the fault tolerant system according to the eleventh embodiment of the present invention.

FIG. 24 is a diagram showing an address map in the node of the fault tolerant system according to the eleventh embodiment of the present invention.

[Explanation of symbols]

11-1n …… Processing node, 20-2m ……
Output selection node, 3 ... Network, 111-11n
...... Data processing function, 121 to 12n ...... Self-diagnosis function, 111 to 11n ...... Execution task determination function, 131 to
13n ... Processing result transmitting function, 210-21m ... Processing result collecting function, 220-22m ... Data comparing function, 2
30 to 23 m ... Judgment function, 240 to 24 m ... Selection function, 7 ... Output selection device, 50 ... Node

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Michi Sato 7-1-1 Omika-cho, Hitachi-shi, Ibaraki Hitachi Ltd. Hitachi Research Laboratory (72) Inventor Shinya Otsuji 7-chome, Omika-cho, Hitachi-shi, Ibaraki 1-1 Hitachi Ltd. Hitachi Research Laboratory (72) Inventor Takashi Hotta 7-1 Omika-cho, Hitachi City, Ibaraki Prefecture Hitachi Ltd Hitachi Research Laboratory

Claims (14)

[Claims] 1. A plurality of processing nodes for executing the same processing, at least one output selection node, and a network connected to the plurality of processing nodes and the output selection nodes. Each processing node includes a data processing unit that executes the process, a self-diagnosis unit that detects or recovers from a failure that occurs in the process performed by the data processing unit, and data that is a processing result of the process executed by the data processing unit. And a transmission means for transmitting fault occurrence information indicating a situation of fault detection or recovery of the self-diagnosis means to the network, and the output selection node transmits from each processing node to the network. Data comparing means for detecting a match / mismatch between the detected data, and a match / mismatch between the detected data, and Based on the status of fault detection or recovery in each processing node represented by the fault occurrence information transmitted from the processing node to the network, each data transmitted from each processing node to the network is described. A fault-tolerant system comprising: a determination unit that determines the most reliable data among them, and a selection unit that outputs the data that is determined to have the highest reliability. 2. The fault tolerant system according to claim 1, wherein the processing node further comprises a configuration of the processing node, reliability of the processing node, and fault detection or recovery coverage of the self-diagnosis means. A category that is information that represents at least one of
The output selection node has a means for transmitting to the network together with the data and the failure occurrence information, and the determination means of the output selection node detects whether the detected data match each other or not, and each processing node.
In addition to the status of fault detection or recovery in each processing node represented by the fault occurrence information transmitted from the network to the network, the category transmitted from each processing node to the network is further represented. , Based on at least one of the configuration of each processing node, the reliability of each processing node, and the fault detection or recovery coverage of the self-diagnosis means of each processing node, transmitted from each processing node. A fault-tolerant system characterized by determining the most reliable data among the respective data. 3. The fault-tolerant system according to claim 1, further comprising a plurality of processing nodes that execute the respective processes and a plurality of output selection nodes for each of the plurality of processes. The processing node of at least a part of the nodes further comprises means for detecting a match / mismatch between the data transmitted from the other processing nodes to the network, and a means for detecting the data matching between the detected data. Each processing node is executed based on the match / mismatch of each other and the status of the detection or recovery of the failure in each of the other processing nodes represented by the failure occurrence information transmitted from each of the other processing nodes to the network. The processing with the lowest processing reliability is determined, and the processing with the lowest processing reliability is determined as the data. A fault-tolerant system, comprising: an execution process determining unit that causes a processing unit to execute. 4. The fault tolerant system according to claim 2, further comprising a plurality of processing nodes that execute the respective processes and a plurality of output selection nodes for each of the plurality of processes. At least a part of the processing nodes of the processing node further comprises means for detecting a match / mismatch between the data transmitted from the other processing nodes to the network, and a means for detecting the detected data mutual. Match / mismatch between the other processing nodes, and other processing nodes indicated by the failure occurrence information transmitted from the other processing nodes to the network.
Status of failure detection or recovery in each processing node, status of failure detection or recovery in each processing node represented by the category transmitted from each of the other processing nodes to the network, and the category Based on the configuration of the processing node represented, the reliability of the processing node, and at least one of the fault detection or recovery coverage of the self-diagnosis means, the most processing among the processings executed by each processing node. A fault-tolerant system, comprising: a process having a low reliability, and executing process determining means for causing the data processing unit to execute a process having the lowest reliability of the determined process. 5. A plurality of input ports that receive the processing results and the respective processing results received by the plurality of input ports are collected, and the comparison and collation results of the respective processing results or the respective execution results are collected. An output selection device that outputs the most probable processing result among the collected processing results according to the majority decision, and a processor that repeatedly executes the same processing a plurality of times,
A fault tolerant system comprising: a node having means for outputting the processing result of each processing executed by the processor to different input ports of the output selection device. 6. The fault tolerant system according to claim 5, wherein said node relays a processing result of said processor to a different input port of said output selecting device for each processing. A fault-tolerant system having a switching means. 7. The fault tolerant system according to claim 5, wherein said node has a plurality of output ports to which the processing results of each time of said processor are inputted, respectively. Is a fault tolerant system, wherein the input processing result is output to different input ports of the output selection device. 8. The fault tolerant system according to claim 5, wherein the node receives a plurality of output ports to which the processing results of the processor are input, and the processing results of the processor. And a means for forcibly switching the output port for each processing, wherein the plurality of output ports output the input processing results to different input ports of the output selection device. A fault tolerant system characterized by outputting. 9. A memory comprising: a memory; a processor for repeatedly executing the same processing a plurality of times while using the memory; and means for forcibly switching an area on the memory used by the processor for each processing. And the execution result of each processing executed by the processor, and the most probable processing among the collected execution results according to the comparison and collation result of each execution result or the majority decision of each execution result. A fault-tolerant system, comprising: an output selection device that outputs a result. 10. The fault tolerant system according to claim 6, 8 or 9, wherein the processor is initialized for each processing. 11. A plurality of nodes and an output selection device, each node comprising a processor for repeating the same processing a plurality of times, and the output device each time of each processor. The processing results of the process are collected, the processing results of the majority of the processing results of the processing performed a plurality of times on the same node match, and the matching processing results are A fault-tolerant system, which selects and outputs the matched processing result when at least one processing result among the processing results of the second processing matches. 12. The fault tolerant system according to claim 11, wherein the output device collects processing results of each processing of each processor, and outputs the processing results of a plurality of processings executed by the same node. If there is no processing result that matches the processing results of the majority of the processing results, and there is no processing result that matches at least one processing result among the processing results of the plurality of processing executed in other nodes, A fault-tolerant system that outputs a signal of the indicated value. 13. The fault tolerant system according to claim 11, wherein the output device collects a processing result of each processing of each processor, and outputs the processing results of a plurality of processing executed by the same node. If there are no processing results that match the processing results of the majority of the processing results, and there is no processing result that matches at least one processing result of the processing results of multiple times executed by other nodes, A fault tolerant system, which outputs a signal for operating an object controlled by the output of the device relatively safely. 14. A plurality of processing nodes performing the same processing, at least one output selection node, and a network connected to the plurality of processing nodes and the output selection nodes, Each processing node includes a data processing unit that executes the process, a self-diagnosis unit that detects or recovers from a failure that occurs in the process performed by the data processing unit, and data that is a processing result of the process executed by the data processing unit. And a transmission means for transmitting failure occurrence information indicating the status of failure detection or recovery of the self-diagnosis means to the network, and the output selection node transmits from each processing node to the network. Matches / mismatches between the detected data are detected, and the detected match / mismatch between the data and each processing Of the data transmitted from each processing node to the network based on the detection or recovery status of the fault in each processing node represented by the fault occurrence information transmitted from the network to the network. A processor that repeats the process of determining the data with the highest reliability and outputting the data determined to have the highest reliability as the processing result a plurality of times, and the execution of each processing executed by the processor It is equipped with an output selection device that collects results and outputs the most probable processing result of the collected execution results according to the comparison and collation result of each execution result or the majority decision of each execution result. Fault tolerant system.