GB2427366A

GB2427366A - Fault tolerant fail safe rebreather control device and method

Info

Publication number: GB2427366A
Application number: GB0512586A
Authority: GB
Inventors: Alex Deas
Original assignee: Individual
Current assignee: Individual
Priority date: 2005-06-21
Filing date: 2005-06-21
Publication date: 2006-12-27
Also published as: GB0512586D0

Abstract

A rebreather tolerant of multiple failures comprises blocks of functions connected in series by data verifiers 65 and selectors. The functions may comprise sensors 11-17, displays, actuators and alarms etc.. These functions are divided into blocks of small granularity such that failure detection in any number of single functions can fail and the system operate normally. Blocks that are prone to failure can have multiple instances such that the probability of all instances failing can be reduced to an acceptable level. Major subsystems of the rebreather may be connected by both optical links 79 and wired links 78. Data may be sent by infra red or optical means. Also disclosed is a method of rebreather control.

Description

Fault Tolerant, Fail Safe Rebreather Control Device and Method Background The present invention relates to the structure of electronics for critical systems, particularly rebreather controllers and rebreather information display systems. Rebreathers have been used for 150 years to allow a person to breath in an adverse environment, for example in a mine filled with lethal gasses, or to breath underwater. The electronics in an Electronic Closed Circuit Rebreather (ECCR) operates by measuring the value of oxygen and other sensors, which may sense pressure, moisture, humidity, ambient pressure, pressure differentials, scrubber life, scrubber health, carbon dioxide partial pressure, helium partial pressure and power levels, and from this data determines when to inject oxygen, and gives indications (instructions or signals) to the user. Various approaches to the failure tolerance and fail safety can be found in commercial rebreather products. Almost all products use multiple oxygen sensors, usually three, with voting logic to determine the actual reading. Whilst methods of fail safe, redundant and fault tolerant system design have been known for decades, rebreathers and similar portable equipment involving multiple modules, pose a particular challenge which has resulted in weak spots such that single failures can cause most rebreathers to cease functioning, and dual failures in the general case to cause all rebreathers to cease functioning. Examples of the extensive prior art in fail safe system design include N. G. Leveson described methods for software safety in multiple papers including "Expecting the worst and preparing for it". In Proc. IEEE Compcon '84, Washington D.C., pages 294{300, New York, September 1984. IEEE, "Software safety: Why, what, and how." Computing Surveys,18(2):125{163, June 1986, "Software safety in embedded computer systems." Communications of the ACM, 34(2):34{46, February 1991, "Safeware: System Safety and Computers," a book by Addison-Wesley Publishing Company, 1995. Despite this, basic principles such as hardware interlock and backup implemented using entirely different principles is not found in any rebreather, because the software tends to be all encompassing. When rebreathers use finite state machines they do not even have redundant codes, so any failure would likely cause the FPGA to jump to another valid condition. Cables linking handsets to controllers fail under high levels of EMI, again because of space, no alternative system for communication was possible. Methods described by T. Anderson and P. A. Lee in "Fault Tolerance: Principles and Practice." Prentice Hall International, 1981 are limited in practical application in rebreathers to the dual controller redundancy and voting logic for 02 sensors. There are many failure points in contemporary rebreathers. Almost all rebreathers follow the structure shown in Figure 2, where there are sensors in the scrubber housing 1, driving two identical control and display units 2, 3, and off handset annunciation 4. These subsystems are linked by cables 6, containing wires 5 with low level signals from sensors, plus a common collector type wired ORed connection to the injector solenoid 10, and a bidirectional wire 24 to advise the other handset which one is master, the other being the slave. The sensors commonly include temperature sensors 11, three oxygen sensors 12, 13, 14, and may include C02 sensors 15 including the system required for that sensor often differential pressure, flow rates, Helium sensing.The handset 2, 3 normally provides the control system, and comprises a pressure sensor 21, water temperature sensor 20, which with other signals is multiplexed by a MUX 22 to an Analogue to Digital Converter (ADC) 23, rarely are multiple ADCs used. The signals are then processed by a microcontroller 26, by a program stored in non-volatile memory plus RAM 25 for execution; the result drives a display 27. The off-handset annunciation 4 head up display 4 may be driven by the controller or may replace one handset if the display can signal enough information. This structure is expensive, amounting to the sale of two complete control systems for every rebreather, yet is still liable to expose the user to a risk of death from single component failure. There are three items of the utmost criticality in a rebreather 02 sensing C02 or scrubber sensing 02 injection Displays If the systems providing any of these four functions fail, the user is at risk of death. The market leading products use two completely independent electronics systems 2 and 3 in Figure 2 operating in parallel, and the user must compare the two identical displays, which should show the same data. One system acts as a master and can drive the oxygen solenoid 10 and buzzer 43; the other acts as a slave but can take over as a master if the master unit is simply switched off. In general, this system can tolerate only one fault. For example, if one battery 29 fails then one control system and handset is down, so a second fault will cripple the system. The APD Inspiration and Evolution rebreathers are examples of this architecture.The CIS-Lunar is also an example of this architecture where the controller display is primary but a secondary display run from the power generated by the 02 sensors provides information to allow the unit to be controlled manually should be main unit fail. Again the system can survive only one failure. Many aspects of a rebreather have no fault tolerance. In most rebreathers, the 02 injector is a binary on-off device controlled by a pulse with modulated data stream operating on either a predictive or level based control system. Any failure in that system would be detected only when the 02 level is incorrect. Users have experienced single failures in the rebreathers that could be expected to cause the death of a user. For example, one well known rebreather uses a microcontroller which has been demonstrated to hang, such that the displays do not show a change in oxygen levels, no oxygen is injected yet the master does not pass control to the slave. This is an example of non-fault tolerant, non-fail safe behaviour. It is apparent from Figure 1, that single faults can cause the failure of any of these systems. For example, if the microcontroller 26 hangs, then it will not update the display, nor drive the solenoid to inject 02. It will not pass control to the second handset. It will not drive the off handset annunciation 4. This failure mode is a real mode on some commercially available products. Four examples of how this particular mode occurs can be illustrated: 1. Some commercially available products these do not even take the precaution of spreading the code in memory and filling unused locations with a JMP 00 instruction, where code at 00 is a failure recovery sequence. This is a basic method uses in life critical systems so that wherever the program counter jumps, if it lands on a sequence that is not valid, then it will either jump immediately to the recovery code, or it will execute one or more NOP instructions then jump. This example assumes the NOP instruction is 00. If the NOP is hex 23, then the jump would be JMP 23 and recovery code would be placed at address 23.A random jump in the program counter can be caused by any memory failure such as from alpha particles or electrical fault, the clock not providing the correct signal due to any clock component failing, a failure in the processor, a failure of decoupling components allowing power glitches into the processor and for a host of other reasons. 2. Power supply brown out circuits are commonly inadequate: they can be demonstrated not to work in every case. This causes very large power glitches to pass to the processor, which then hangs. Watchdog circuits are not implemented on all commercial rebreathers to catch the hang. 3. The cable 6 is often not even shielded, so a high current injection can cause corrupted data to be received. High currents can come from induction or coupling from dive lamp power circuits, undersea cables, or when on the surface, the radar on a ship. 4. If one handset is flooded, then the handset can put a signal of 10mV out onto the 02 sensors, due to galvanic action in the handset in the presence of sea water, giving a high 02 reading on the working handset. The controller that is not flooded, then depending on the output of the sensors when they were last calibrated, the system will not inject 02 because it believes 02 is already high. 5. Widely used rebreathers appear to have power coupling to the sensors, that is a leakage path, such that at particular internal resistance values of the batteries, the system pulls down the 02 sensors as power is drained on activating the solenoid. All of these four failure cases in this one failure syndrome have been reported to occur in the field. There are very many of these different failure syndromes. It is clear from these examples that single failures can bring down a rebreather. For example the single failure of interference on the cable affects both handsets, because they are often and literally wired in parallel. Another single point failure is a flood in the sensor area of the scrubber, moisture in the battery compartment when the two batteries are co-located in the scrubber. There are various single points of electro-mechanical failure, for example pulse type gas solenoids are used. These are designed for operation at 6 bar, but the interstage pressure from diving regulators is 10 to 12 bar. A regulator is adjusted specially for the rebreather injector Overpressure of the solenoid gas supply will cause it to stick on, and there is no override, or sensing that it is working. The system can only report rising 02 levels. None of the contemporary rebreathers is tolerate of two failures. For example, if the multiplexer fails on one handset, and the power fails on another handset, then the entire system fails. Power is often not in the handset but in the main unit and both batteries are located side by side. This is another example of a single failure (water in the battery compartment), causing the entire system to fail. Software for rebreather control systems is often written by one person, without safeguards that are common practice in the design of dependable or life critical systems, and without the use of formal verification and other tools. As both handsets are running the same code on the same processor, a software error can have serious consequences. There are tens of thousands of CCRs being used. With two handsets each, failures are highly likely in the population of users. Users are aware of the dangers of system failure, but lack the expertise to address this. Frequently a user will add a fourth oxygen sensor, as the oxygen sensors have life of only a year and are subject to moisture or socket failure. The user may interpret the display from the fourth sensor, and use that data to override the rest of the system: that is, the user has just reduced a system which uses triple redundancy and a voting system, to one which has no redundancy and is driven by a single sensor which is known to fail frequently! Object It is a primary object of the present invention to provide an electronic control and information system for a rebreather than can withstand more than one failure and still operate safely. In the ultimate case, the present invention can withstand large numbers of failures of individual components and circuits and still operate safely.It is a further object of the present invention that when failure occurs, the failure is detected. It is a further object of the present invention that when gross failure occurs, the system enters a safe state: that is, the system is fail-safe. Statement of Invention The present invention divides the chain of functions from sensors to displays, to actuators and to alarms, into redundant blocks of small granularity; with failure detection such that any number of single functions can fail and the system operate normally. Blocks that are prone to failure can within the present invention have multiple instances such that the probability of all instances failing can be reduced to any level determined to be acceptable by the designer. Brief Description of the Invention and Figures The invention will now be described by way of example, without limitation to the generality of the invention, and with reference to the following figures: Figure 1 is a block diagram of an example embodiment of a triple redundant, multiple fault tolerant and fail-safe rebreather control and display system according to the present invention. In this example, there are three subsystems, a base unit 1 performing sensing and gas injection functions in the scrubber housing, a handset 2 and an off handset annunciation system 4 comprising a head up display with two lights 41 and 42 driven from LEDs mounted in the base unit via optical fibres 80 and 81 along with voice annunciation driving data verifier 83 using a PCM sequence and a speaker 84. The signals that drive the speaker 84 for voice annunciation of status and alarms are carried on both optical 79 and wired links 78 or may be a data stream using an asynchronous serial protocol such as that used in RS432 or a PCM stream. The wired link 78 is data over a power and ground wire, that is a high frequency modulation of a power signal, the power signal powering the decoder 83 that is a circuit that selects which of the two signal sources it uses. The optical link 79 in this example is simply a LED and Photodiode, broadcasting data down the breathing hoses to the mouthpiece, without any optical cable, the inside of the hose being treated with a partially reflective material. The data for the alarms comes from a data driver 66, using data from a data verification and selection block 65.Different data driver circuits are used for the head up display, speaker wired and speaker optical channels. Tracing the path from sensor to actuator, four 02 sensors 12, 13, 14 and 15, pressure sensor 17, scrubber monitoring sensors 11, moisture, humidity sensors and C02 sensors 16 are sampled by three signal conditioning and multiplexer circuits 22 and three Analogue to Digital Conversion (ADC) circuits 64, with the output of the ADCs 64 driving three data verifiers 65. Each data verifier takes the data stream from the multiple sources, and determines which source is valid, or combination of sources is valid, and then provides a vote on the data and a copy of the voted data and downgraded data (i.e. data marked not preferred) to the next block. For example in the case of the oxygen sensors, the same data for four sensors should come from each of three ADCs 64.The data verifier determines which of the ADC 64 outputs are invalid due any of ADC 64, MUX 22 or sensor failures, and using voting logic provides the correct level as well as tagged raw data to the next block. Where there is no data within valid limits, the data verifier sends the data from the trend it forecasts from the changes of the data that were previously valid along with all the incorrect data. Data Verifier 65 operate in triplicate in this example, preferably capable of using different power supplies 62, 63 and power from 29 chosen by the power conditioning and supervisor function in block 61. The data verifiers are checked in the control system FPGA 31 and microcontroller 26 to ensure they have operated correctly and corrective action taken if any particular unit fails, such as making another instance the master.All data is sent with CRC codes to determine if any bit or bit combination is corrupt up to a designed bit error run length. One of the drivers following the data verifier drives the head up display and annunciation driver, and two sets of handset drivers, together forming 66. The data verifier also drives the oxygen data to two other data verifiers 60 which drive two oxygen injectors 10. The handset 2 has two sections, each independently calculating values to display from the data, which is presented in multiple copes. One section uses a Finite State Machine (FSM) 31 to take the data from the incoming data verifier and selector 91, of which there are multiple copies, along with data from private sensors 21 such as for pressure and temperature. The FSM 31 determines whether the unit is within set levels, and drives a display, such as a two line text display. It also drives a channel to send commands to a combined solenoid driver and data verifier 60, which drives one of the two solenoids 10.The optical and wired data should be identical, but if incorrect, the data verifier removes the data with an incorrect CRC code, or that is outside set ranges, and prefers the other source. If the data is contrary to limits set from the source where 02 sensor data is acquired directly from the data verifier 65, then the first verifier 60 checks if data is available from the second solenoid driver 60 and uses that. If the verifier 60 cannot get valid data, it indicates a warning and takes over control to keep 02 set to safe level, for example, fixing control at a PP02 of 1.0. The solenoid driver operates in a closed loop, such as a needle valve controlled by DC motor with position sensing, so instructions to the motor to move by a desired amount can be checked in closed loop. The second control system comprises data buffers and verification 90, microcontroller 26, display 27, ROM and RAM 25, private sensors 21, and drives a separate solenoid 10 via a second verifier 60. Power is provided in the example in Figure 1 in triple redundancy, both in source and in distribution. Batteries 62, 62 in a hermetic compartment with a bleed screw for removal of access caps are regulated and controlled by a suitable controller 61, and power from the handset battery 29 is used as a backup should both main batteries 62, 63 fail. In the case of power failure, the system can if set by the user, reduce energy by reducing the degree of control over the solenoids to allow a wider PP02 tolerance, and by other measures. It is common practice to power off circuitry not in use. Power to the handset in the example in Figure 1 is provided from the base unit 1 by the two sets of wires that carry one copy of the data to the handset, using data over power technology, and are used to drive the handset with each of the two sections of the handset, the FSM system and the microcontroller system, having independent power management. Disconnection of the handset 2 from the base unit 1 would still result in a useable rebreather, in that the handset 2 would continue operating but displaying the fact of the disconnect, and the base unit, deprived of instructions on solenoid 10 would fall back to safe operation of each solenoid to maintain a PP02 of 1.0 in this example. Should the user desire, then rebreather could be operated manually based on the status information from the voice annunciation 83, 84 and head up display lights 41, 42, with buzzer and vibrator 43 to alert the diver of lapses in manual control. Figure 2 (Prior Art) has been described already, but for clarity, shows the most commonly used controller and display structure and method used in contemporary rebreathers. In those systems the sensors in the scrubber housing 1, drive two identical control and display units 2, 3, and a off handset annunciation system 4 comprising head up display LEDs 41, 42, and sound devices 43 such as buzzers, vibrators and a voice annunciation system. These four subsystems are linked by cables 6, containing wires 5 with low level signals from sensors, plus a common collector type wired ORed connection 28 to the injector solenoid 10, and a bidirectional wire 24 to advise the other handset which one is master, the other being the slave.The sensors in the scrubber unit commonly include temperature sensors 11, three oxygen sensors 12,13,14, and may include C02 sensors 15 including the sensory subsystem required for that sensor: often differential pressure, flow rates and Helium sensing. The handsets 2, 3 provides the control system, and comprises a pressure sensor 21, water temperature sensor 20, which with the other signals is multiplexed by a MUX 22 to an Analogue to Digital Converter (ADC) 23. The signals are then processed by a microcontroller 26, by a program stored in non-volatile memory plus RAM 25 for execution; the result drives a display 27. The off-handset annunciation and head up display 4 may be driven by the controller or may replace one handset if the display can signal enough information.Batteries 29 provide power and may be located in the handsets as shown or in a batter compartment in the scrubber. The scrubber is monitored using a temperature sensor 11 distributed in the scrubber and by monitoring of C02 levels using dedicated sensors. Figure 3 is a block diagram of the method of the present invention, comprising a series of function blocks 51, 51, 54, 55, 58,59, 600, 620, interleaved with checking and voting logic 52, 53, 56, 57, 610 between each function block. Each function block uses different technology or methods where this is possible, including for communication of data. Blocks such as display 58, 59 can be located in the same handset using separate compartments to prevent a flood causing failure of both, but are driven by blocks 54 and 55 which have fundamentally different implementations, such as one using a FPGA and the other a microcontroller.This block diagram showing the method of the present invention applied to rebreathers is at the highest level of granularity. It will be apparent to a person skilled in the art to apply the same method also at finer granularity to the subsystem design within the rebreather. Operation of the Present Invention The operation of the invention will be described, by reference to example embodiments without limit to the generality of the invention. The application of Figure 3 to the example in Figure 1 will be described to provide an example of the operation of the present invention, taking key functions. The first function is 02 injection. In the example in Figure 1, the entire signal path provides triple redundancy, is tolerant of three simultaneous failures of the same component, tolerate of large numbers of failures of random components, and is fail safe. This assumes the two 02 injectors 10 are used in conjunction with an auto-bail out, auto-shutoff valve, as in GB 2,404,593 and US 6,817,359 that has the obvious by-product of auto-flush. The 02 injectors 10 the base can withstand either control system failing in any state, disconnection of the control system and even total loss of power, preferably by using a geared DC motor, using a worm gearing of greater than 1:2, which will hold its position when not powered such that 02 is injected at a constant rate equal to that at the last measured rate when the electronics was operational.The system is thus fail-safe, as well as failure tolerant. There is a high observability of all failures, which can be communicated by a serial bus between function units. The system is highly testable due to the presence of this bus and the serial breakdown of functions. Full self test can therefore be performed very quickly. The next function taken as an example, is the information system to the user. This is a key system so is quad redundant, with two independent controllers being the microcontroller 26 and FSM 31, driving two independent displays 27, 30. Each of these use different technology, for example a text display and a direct dive display, the microcontroller and a FSM implemented in FPGAs respectively. The key information is also sent to the user by the voice annunciation system driving 83, 84 and the head up display 4 with LEDs driving light pipes or optical fibre 41, 42. The next function taken as an example is the wiring between the handset and base unit. This is in quad redundancy in each direction, with two sets of data over power and two optical fibres to the handset, and two optical fibres and two power channels from the unit. The power management system is in quad redundancy, with each section of the handset having its own power controller, as well as power being provided in regulated form from two power sources (batteries 62 63) in the base unit 1. The temperature sensor is poly redundant, with a scrubber temperature measurement stick 11, providing multiple temperature readings, as well as temperature sensors in the handsets private to each controller. A severe electrical discharge into the cable, or magnetic coupling, would not affect the optical path of data as it has physical immunity to this mode of failure, and during the discharge the handset would run on its own power. The power supplies and power management functions 61, 33 are preferably switching type to minimise power loss and for the maximum resilience to high frequency power line disturbance. Under these circumstances, the entire rebreather would operate normally. When there are multiple instances of a function, they should preferably be implemented using different technology. A key point to note in the method and example implementation is that any function can fail, and the rest of the system takes over. This conforms in broad principle to the layering of data verifiers as shown in Figure 3. Both the microcontroller and the FPGA incorporate watchdog circuits to detect if the clock or program has hung and forces a controlled restart. All power supply and power control systems preferably have brown out circuits, to detect a drop in power levels, voltages or currents, that risk malfunction of the electronics using those supplies, and forces a controller restart of the affected electronics. It will be appreciated by a person skilled in the art that some elements may be simplified. For example, the head up display can be driven using light guides from LEDs in the Base Unit 1, and the speaker 84 could be driven by an audio signal, on the basis the redundancy of the information to the user is already sufficient. Likewise, instead of two data over power copper connections between the Base Unit 1 and Handset 2, one connection could be used and reliance made on a single optical connection.

Claims

1. A rebreather tolerant of multiple failures using function blocks connected in series by data verifiers and selectors.

2. A rebreather according to claim 1 where major subsystems are connected by both optical and wired links sending the same or similar data, with data verification of the received data.

3. A device according to any of claims 1 to 2, where data is sent using infra red or optical means between units without an optical fibre.

4. A device according to any of claims 1 to 3 with a plurality of 02 injectors that have closed loop control.

5. A device according to any of claims 1 to 4 with a plurality of 02 injectors that have closed loop control, comprising a geared motor that holds a needle valve in the last known good position on a failure occurring.

6. A device according to any of claims 1 to 5 with data verifies comprising circuitry to confirm the validity of each data source, with voting logic to select between multiple sources, and trend tracking such that if all the data is invalid, the data can be indicated that would result from previous trends being continued, with onwards transmission of the data choice and tagged or otherwise distinguished erroneous data.

7. A device according to any of claims 1 to 6 where the power systems are local to the sensors or injectors, and also provided in the handset, such that overall multiple redundancy is achieved.

8. A device according to any of claims 1 to 7 where one display is driven by a state machine, and the other driven by a microcontroller with brown out and watchdog protection, and unused memory locations filled with NOPs and a jump to a controlled restart sequence.

9. A method for rebreather control tolerant of multiple failures using function blocks connected in series by data verifiers and selectors.

10. A method for rebreather control according to claim 9 where major subsystems are connected by both optical and wired links sending the same or similar data, with data verification of the received data.

11. A method for rebreather control according to any of claims 9 to 10, where data is sent using infra red or optical means between units without an optical fibre.

12. A method for rebreather control according to any of claims 9 to 11 with a plurality of 02 injectors that have closed loop control.

13. A method for rebreather control according to any of claims 9 to 12 with a plurality of 02 injectors that have closed loop control, comprising a geared motor that holds a needle valve in the last known good position on a failure occurring.

14. A method for rebreather control according to any of claims 9 to 13 with data verifies comprising circuitry to confirm the validity of each data source, with voting logic to select between multiple sources, and trend tracking such that if all the data is invalid, the data can be indicated that would result from previous trends being continued, with onwards transmission of the data choice and tagged or otherwise distinguished erroneous data.

15. A method for rebreather control according to any of claims 9 to 14 where the power systems are local to the sensors or injectors, and also provided in the handset, such that overall multiple redundancy is achieved.

16. A method for rebreather control according to any of claims 9 to 15 where one display is driven by a state machine, and the other driven by a microcontroller with brown out and watchdog protection, and unused memory locations filled with NOPs and a jump to a controlled restart sequence.