WO2015174963A1 - Système et procédé de surveillance de santé et de fatigue de conducteur - Google Patents

Système et procédé de surveillance de santé et de fatigue de conducteur Download PDF

Info

Publication number
WO2015174963A1
WO2015174963A1 PCT/US2014/037814 US2014037814W WO2015174963A1 WO 2015174963 A1 WO2015174963 A1 WO 2015174963A1 US 2014037814 W US2014037814 W US 2014037814W WO 2015174963 A1 WO2015174963 A1 WO 2015174963A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
vehicle
antennas
occupant
monitoring
Prior art date
Application number
PCT/US2014/037814
Other languages
English (en)
Inventor
David S. Breed
Original Assignee
American Vehicular Sciences, LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by American Vehicular Sciences, LLC filed Critical American Vehicular Sciences, LLC
Priority to PCT/US2014/037814 priority Critical patent/WO2015174963A1/fr
Publication of WO2015174963A1 publication Critical patent/WO2015174963A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024—Detecting, measuring or recording pulse rate or heart rate
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/08—Detecting, measuring or recording devices for evaluating the respiratory organs
    • A61B5/0816—Measuring devices for examining respiratory frequency
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6887—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient mounted on external non-worn devices, e.g. non-medical devices
    • A61B5/6893—Cars
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K28/00—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions
    • B60K28/02—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver
    • B60K28/06—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver responsive to incapacity of driver
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B60W50/16—Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal

Definitions

  • the present invention relates generally to systems and methods for monitoring a driver of a vehicle to determine at least one characteristic, condition, property and/or state of the driver, for example, whether the driver is falling asleep or otherwise unable to operate the vehicle.
  • At least one embodiment of the present invention is primarily concerned with detecting a drowsy or sleepy driver but a number of the other causes of the distracted driver also affect the parameters that are the focus of embodiments of the invention and thus will also be addressed, and are encompassed within the scope of the invention in its entirety.
  • a primary focus of at least one embodiment of the present invention is to measure the heartbeat rate and respiration rate of the driver using electric fields or optics, and through analysis of these rates, predict whether the driver is losing his/her ability to safely operate the vehicle.
  • Driver inattention is a larger problem than driving sleepiness and fatigue.
  • a vehicle including a seat in which an occupant, such as a driver, sits during use of the vehicle, and a monitoring system for monitoring the occupant in said seat.
  • the monitoring system includes a plurality of sets of electric field antennas, each set including at least one antenna and more likely a plurality of antennas, and a control unit connected to the antenna sets and that includes selectors coupled to said antennas and that controls the selectors to obtain signals from one or more of said antennas serving as receiving antennas and one or more of said antennas serving as sending antennas.
  • the control unit After obtaining signals from multiple combinations of the antennas, and maybe even all of the possible combinations of antennas, the control unit determines which combination of one or more of said antennas serving as the sending antennas and one or more of said antennas serving as the receiving antennas provides a strongest signal in an expected heartbeat range and/or expected respiration range of the occupant. Thereafter, the control unit monitors this determined combination for changes and/or deviations from a normal range of heartbeats and/or respiration rate, and preferably only this determined combination. Changes and/or deviations from the normal range of heartbeats and respiration rate may be used to assess the drowsiness or sleepiness of the occupant, especially when the driver of the vehicle, and activate a notification system that requires a response to enable continued operation of the vehicle by the driver.
  • a method for monitoring an occupant of a vehicle, especially the driver of a vehicle includes arranging a plurality of antennas on a seat, each separated from one another; selecting from a plurality of different possible combinations of the antennas, two or more of the antennas to form a sending and receiving arrangement of antennas, generating a weak electric field via the selected antenna arrangement, detecting information related to a current flow in the selected antenna arrangement during generation of the weak electric field, and determining, using a processor, a health condition of the occupant based on the detected information.
  • the health condition includes, but is not limited to, the drowsiness or sleepiness of the driver.
  • a method for optically monitoring a driver of a vehicle in accordance with the invention includes illuminating a portion of the driver with electromagnetic radiation in an infrared portion of the electromagnetic spectrum using at least one illumination device, obtaining images of the illuminated driver using at least one image obtaining device, and analyzing the images to derive a measure of flow of blood in at least one blood vessel, capillary and vein in the face of the occupant.
  • the blood flow may be analyzed over time to determine whether the driver has lost the ability to continue to control the vehicle.
  • the loss of ability to continue to control the vehicle arises from, for example, the driver becoming drowsy, falling asleep or otherwise being incapable of controlling the vehicle after initially having been awake or otherwise capable of controlling the vehicle.
  • action may be taken automatically to effect a change in the operation of the vehicle, e.g., brake control, throttle control, steering control or a combination of these. Additionally or alternatively, action by the driver may be required to indicate regaining of the ability to control movement of the vehicle.
  • the driver may be notified of the determination that they have lost the ability to continue to control the vehicle, required to generate a response to the notification which response may be monitored and then vehicular operation controlled based on the monitoring of the driver for the response to the notification.
  • Analysis of the images may entail determining a heartbeat of the driver, in which case, analysis of the blood flow over time to determine whether the driver has lost the ability to continue to control the vehicle entails analyzing variability of the heartbeat of the driver.
  • Analysis of the images may additionally or alternatively entail determining a size of each blood vessel, capillary and vein, in which case, analysis of the blood flow over time to determine whether the driver has lost the ability to continue to control the vehicle entails analyzing variability of the size of the blood vessel, capillary and vein driver.
  • the frequency of the electromagnetic radiation may be selected to ensure reflection from blood in the driver.
  • the frequency may be between 500 nm to about 600 nm or more specifically, between about 540 nm to about 580 nm.
  • the illumination may also be modulated.
  • a pattern of the blood vessel, capillary and vein in the face of the occupant may be monitored to provide a biometric identifier of the driver, and then adjustment of a vehicular system, such as a mirror, seat, seatbelt, radio, etc., controlled based on the biometric identifier of the driver.
  • Pattern recognition may be used to determine a location of the face of the driver from images being obtained by the imaging device.
  • the images may also be analyzed to detect presence of alcohol or drugs in blood in each blood vessel, capillary and vein in the face of the occupant.
  • the method involves obtaining at least one initial image, analyzing the initial image to determine location of the face of the driver and based thereon, selecting a portion of the face of the driver for infrared illumination. Only the selected portion of the face of the driver may then illuminated. The amount of illumination may be controlled to harm to the driver. It may also be focused to avoid eyes of the driver. Movement of the selected portion of the face of the driver may be monitored.
  • An arrangement for optically monitoring a driver of a vehicle in accordance with the invention includes at least one illumination device that illuminate a portion of the driver with electromagnetic radiation in an infrared portion of the electromagnetic spectrum, at least one image obtaining device that obtains images of the illuminated driver, and a processor coupled to the image obtaining device(s) and that analyzes the images to derive a measure of flow of blood in at least one blood vessel, capillary and vein in the face of the occupant and then analyzes the blood flow over time to determine whether the driver has lost the ability to continue to control the vehicle.
  • FIG. 1 is a side view with parts cutaway and removed of a vehicle showing the passenger compartment containing a driver sitting in a vehicle seat which contains a plurality of electric field antennas in accordance with a first embodiment of the invention.
  • FIG. 2 is a block circuit diagram of the electric field sensing system of FIG. 1.
  • FIG. 3 is a perspective view with parts cutaway and removed of a vehicle showing the passenger compartment containing an electric field occupant sensing systems on both the driver and passenger seats in accordance with a second embodiment of the invention.
  • FIG. 4 is a block diagram of a variety of occupant sensing and health monitoring systems.
  • FIG. 5 is a side view, with certain portions removed or cut away, of a portion of the passenger compartment of a vehicle showing several preferred mounting locations of a driver health and fatigue monitoring system in accordance with a third embodiment of the invention.
  • FIGS. 6A, 6B and 6C illustrate the principles of operation of a single electric field monitoring element in the presence of a body element and a modified body element.
  • FIGS. 7 A, 7B and 7C illustrate the principles of operation of a pair of interacting electric field monitoring elements in the presence of a body element and a modified body element.
  • FIG. 8 is a block diagram illustrating the relationship between the occupant sensing unit and the vehicle safety system in accordance with the invention.
  • FIG. 9 is a flow chart showing the operation of the driver health and fatigue monitoring routine in accordance with the invention.
  • FIG. 10 is a flow chart showing the operation of the warning and feedback system routine in accordance with the invention.
  • FIG. 11A is a view of an optical monitoring system in accordance with the invention that monitors a portion of the face of the driver from an imager mounted near the top of the windshield with an alternate mounting location on the rear view mirror.
  • FIG. 1 IB is a view of another embodiment of an optical monitoring system in accordance with the invention that monitors a portion of the face of the driver from an imager mounted in the instrument panel facing so that it reflects off a portion of the windshield.
  • FIG. 12A shows a seated-state detecting unit along with an optical sensor, a weight sensor, a reclining angle detecting sensor, a seat track position detecting sensor, with a heartbeat, respiration rate and motion sensor as part of an bladder weight sensor, for use in any of the embodiments of the invention.
  • FIG. 12B is an illustration as in FIG. 12A with the replacement of the bladder weight sensor of
  • FIG. 12A by a strain gage or displacement weight sensor within a cavity in the seat cushion and on the seat support structure, for use in any of the embodiments of the invention.
  • the invention is not limited to any particular vehicle and is thus applicable to all relevant vehicles including all compartments of a vehicle including, for example, the passenger or other compartment of an automobile, truck, farm tractor, construction machine, train, airplane and boat.
  • Distraction can take many forms as described in the background section and each has a particular suit of solutions.
  • a texting driver for example, can be detected with a camera and appropriate pattern recognition software and/or by the electromagnetic emissions from his or her cellphone or other connected device being used to text.
  • a driver that is talking on a hand-held cellphone can be detected optically and/or by the electromagnetic emissions of his/her cell phone arising from the area of the driver seat.
  • a particular sensor can be placed so as to monitor the driver seat or a pair of sensors that can determine the angle of arrival of the EM signal can be mounted on the dashboard, for example.
  • An inebriated driver can be detected, for example, by an alcohol monitor that samples air in the passenger compartment or by driver behavior monitoring or by monitoring his heartbeat and/or breathing (i.e., respiration rate).
  • Driver behavior indicative of inebriation can be detected optically with pattern recognition software and/or by monitoring the driving behavior of the driver with respect to a map database and accurate vehicle location system or optically using a camera that monitors the vehicle position on the roadway.
  • pattern recognition software applied or executed by a computer or other processing unit can be trained to recognize abnormal behavior.
  • Such pattern recognition software is typically embodied on non-transitory computer-readable media available to a processor that is coupled to any system that provides information about the driver or operator.
  • the pattern recognition software may also be in the form of a computer program stored on non-transitory media and that may be installed on the vehicle prior to sale to the consumer.
  • the software and computer program may be updated via a wired link or a wireless link, e.g., via the Internet or another communications network.
  • Detecting the drowsy or health impaired driver is a different issue generally requiring different hardware and software. Monitoring the eyes of the driver is one approach when the eyes are visible. However, it may be difficult to diagnose a driver having a heart attack by his eye motion, or lack thereof, in time to take remedial action. Also, a driver with sunglasses or even ordinary glasses or a hat with a shade or veil may present difficulties for eye tracking software and then the driver may not be looking at the camera at the time that an event occurs. A system that can work with eye trackers or independently is thus needed to sense cases missed by eye trackers or to confirm the eye tracker results in order to minimize false alarms.
  • the described methods are optical, radar and driving behavior related, for example, to maps.
  • Three methods that have not been discussed and will be discussed here are measuring: (1) the variations in pressure in a bladder or PDVF sheet caused by heartbeats and/or respiration rate, (2) the variations in the appearance of a patch of skin on the face of a driver due to heartbeat, respiration rate, blood pressure and temperature, and (3) the variations in capacitance or electric field signals also due to heartbeat and/or respiration rate.
  • a fourth method has recently become available using Plessey PS 2501 ECG sensors mounted in the seatback.
  • an electric field monitoring system is particularly useful especially since it is now in production as an occupant presence and position sensor.
  • driver heartbeat rates and/or respiration rates a determination can be made as to whether the driver is drowsy and at risk of falling asleep and/or whether he or she is experiencing a health event that could interfere with his or her ability to operate a vehicle.
  • the respiration rate is indicative of the driver's breathing and other parameters indicative of breathing may alternatively be used, i.e., not necessarily a rate at which the driver breathes or respires.
  • the facial monitoring approach which is being developed at the MIT Media Lab and which has been considered for patient monitoring, uses a facial image to determine the flow of blood in the face vessels and then can derive an accurate pulse reading.
  • the respiration rate and blood pressure readings can also be determined from the facial image. While these techniques are in the experimental stage, their incorporation into a vehicle for the purpose of driver health monitoring has not been considered by others and is believed to be unique to inventions disclosed herein. With this system, there is no need for other apparatus, only a camera that can view the driver's face with green and, in some cases, IR illumination. Neither green nor IR illumination has been disclosed in the MIT study, although green frequencies reflected from the face from natural or broad spectrum illumination is discussed. Thus, a single camera can determine eye tracking, closure state, blinking rate and pulse and respiration measurements, as well as blood pressure.
  • Apparatus have also been developed in Japan to remotely measure heart rate and respiration rate using low power microwaves. This is similar to use of radar as described in U.S. Pat. No. 7,196,629. Also, a smart cap can be used to measure certain EEG signals and can be used to determine operator fatigue.
  • the driver emits electromagnetic signals that are used in some Electroencephalogram (EEG) and electrocardiogram (EKG or ECG) devices to monitor brainwaves and electro cardio functions of the body and use of sensors placed in the seat, seatbelt, seatback and headrest to monitor these signals is also contemplated herein and some are discussed below.
  • EEG Electroencephalogram
  • EKG electrocardiogram
  • Other observables include IR radiation from the driver and particularly his or her face indicative of the driver's face or other location temperature. All of these monitoring techniques are included herein as driver health and condition monitoring systems.
  • driver monitoring systems are imprecise and thus subject to false alarms and consequently, before any action is taken to take control of any vehicle systems, the diagnosis should ideally be verified through driver feedback.
  • This feedback can take many forms such as employing a heads-up display on the windshield or elsewhere in the view of the driver that display a query asking the driver to respond, such as orally or by depressing the horn or other switch or button, or making a gesture, for example.
  • Other feedback systems include an oral alarm or verbal statement requesting a response from the driver or any other visual or audible alarm or message requiring the driver's attention and subsequent action indicating that the sensed event was a false alarm.
  • the driver monitoring system can repeat the request a number of times, wherein the specific number may be dependent on or determined by the urgency of the situation (i.e., a more urgent situation might require more requests than a less urgent situation).
  • the driver monitoring system can ask the driver "are you alright?" and if the driver responds with something like "yes", then the system can be reset.
  • the request can be changed to a more complex request such as "if you are alright, depress the horn button or raise your hand.” If the driver fails to respond appropriately to this more complex request, one or more of a variety of actions can be taken by the vehicle system depending on an assessment on the vehicle as to what is most appropriate.
  • the driver monitoring system in accordance with the invention includes an algorithm embodied on storage media that is accessed by the processor to select requests, i.e., a request selection algorithm, based in part on a history of requests that may be maintained in a memory unit also accessed by the processor.
  • the actions that may be taken can include a gradual slowing of the vehicle and flashing the emergency lights, sounding the horn for example in a pulsed alarm mode, taking control of the steering to maintain the vehicle in its lane using lane monitoring cameras, if available, or GPS with an accurate map database if available, and avoiding a collision, notifying a remote site, allowing a remote site to assess the situation and take partial control of the vehicle, etc.
  • the action or actions are determined by an action selection algorithm embodied on storage media that is accessed by the processor that can factor in various factors, including, for example, the severity of the situation.
  • a preferred system will first be disclosed based on an electric field occupant monitoring sensor system whereby the variations in the electric field strength at particular frequency ranges corresponding the human heartbeats (about 30 Hz to about 200 Hz) and respiration rates (about 0.05 to about 2 Hz). Since it is unlikely that steady signals from other sources will interfere with these frequency ranges, filtering and locking onto these signals and tracking them while searching for anomalies, using electronic hardware and software that is known to those skilled in the art, requires only straightforward signal processing that could be easily developed by one skilled in the art without undue or unreasonable experimentation. Gross occupant movement or even the movement of his or her extremities can also be monitored and, even without knowing the exact motion detected, this information can be used to collaborate the analysis of the respiration and heartbeat signals.
  • the accuracy and reliability of the entire system can be dramatically improved.
  • the passenger can tell if the driver is having a heart attack or falling asleep. Therefore, an optical or other monitoring system can also do so. Adding in a heartbeat and/or respiration monitor should make the accuracy of such a system close to 100%.
  • FIG. 1 is a side view, with parts cutaway and removed of a vehicle showing the passenger compartment, or passenger container, containing a driver 2 operating a vehicle.
  • Embedded in the seat or as part of the seat cover material are two sets of electric field antennas 4 and 6.
  • the manner in which each set of electric field antennas 4, 6 can be embedded in the seat may be by placing the electric field antennas in the cushion material of the seat.
  • electrically conductive material may be woven into the material of the seat cover or arranged on the material of the seat cover.
  • Other alternatives include placing a conductive mat in the seat just below the seat cover material or, in some cases using the seat heating wires as an antenna. There are thus many ways that an area of a seat can be made conductive and to function as an antenna.
  • Each set of electric field antennas 4, 6 can have any appropriate number of individual antennas such as from 2 to about 20.
  • Antenna set 4 is illustrated with 7 antennas and antenna set 6 with 5 antennas.
  • Antennas can additionally be placed at other convenient locations within the vehicle such as antenna 8 located on the ceiling or headliner above the seat or seating location in which the driver is expected to be present, and antenna 10 located on the steering wheel 12, and others not shown in FIG. 1 can be on the vehicle door and floor etc.
  • Antenna 8 may be arranged on the exposed surface of the material forming the ceiling or headliner, or behind this surface.
  • Antenna 10 may be arranged within the housing defining the steering wheel or on the outer surface thereof.
  • ECU 9 Each of the antennas is connected to an electronic control unit (ECU) 9 which performs the health monitoring calculations as explained below.
  • ECU 9 includes software and hardware known to those skilled in the art, such as a processor and signal processing circuitry and software embodied on non-transitory computer-readable media to implement the processing stages described herein.
  • the connections between the antennas and the ECU 9 may be a wired or wireless connection.
  • a "control unit” will generally mean any component or combination of components that is capable of processing input data and/or signals for an identified purpose, and generating from the data and/or signal processing, commands or signals to be directed to over one or more other electronic components to be controlled.
  • a control unit in accordance with the invention may have other functions, e.g., to generate signals to be transmitted via a communications system.
  • a representative circuit block diagram of the ECU 9 is illustrated generally at 40 in FIG. 2 for the sets of antennas illustrated in FIG. 1.
  • One skilled in the art would understand how to implement this circuit block diagram in a commercial electronic package.
  • an oscillator 21 operating in a frequency range of, for example, about 100 kHz to about 10 MHz is attached to a resistor 22, a pair of selection switches 23 and 24, a voltage buffer 25, a detector 26, an A/D converter 27 and a processor 28.
  • the antennas in the seatback or seat back portion (antenna set 4) are labeled 4-1 through 4-7 and the antennas in the seat bottom or seat bottom portion (antenna set 6) are labeled 5-1 through 5-5.
  • the seat bottom portion is that portion of the seat on which the driver 2 sits while the seat back portion is that portion of the seat against which the driver's back rests. Not all antennas are individually labeled in FIG. 2, e.g., antenna 8.
  • selection switches 23, 24 or selectors are to choose that combination of sending and receiving antennas that provide the best heartbeat, respiration and occupant movement signals.
  • any type of selection device or selection means that performs the same function as selection switches 23, 24 may be used in the invention.
  • Any of the antennas can operate in a sending mode or receiving mode and a single antenna can be both considered as a sender and receiver, that is, the occupancy of a seat can affect the antenna current.
  • An excellent background for the functioning of electric field occupant sensors is provided in U.S. Pat. Nos. 6,684,973 and 6,960,841 as well as others assigned to Elesys and Hyundai covering electric field occupant sensing and others assigned to Automotive Technologies International, Inc, (ATI) mentioned herein. Whereas Elesys and ATI primarily use this technology for identifying the occupancy of the passenger seat for the purposes of airbag suppression or depowered deployment, the purpose here is to use the technology in an altogether different manner to achieve health and fatigue monitoring of the driver.
  • the variation in the electric field signal caused by the beating heart and respiration needs to be separated from the overall signal.
  • the antennas that provide the most information for occupant identification are not necessarily the same antennas that provide the best heartbeat or respiration signals.
  • the antennas that provide the best signal for heartbeat monitoring are not necessarily the same antennas that provide the best signal for respiration monitoring, however, the respiration rate can generally be derived from the heartbeat signal.
  • Antennas that are best for motion monitoring are not necessarily the same ones that are best for identification or for heartbeat or respiration monitoring.
  • the driver In current airbag systems, the driver often does not need to be classified for the purpose of airbag suppression so there is no need to install an electric field occupant monitor in the driver seat. It is typically assumed that the driver has the characteristics of an adult and that the airbag does not need to be suppressed unless the driver's seat is positioned far forward indicating that the driver is a small adult (a position-based determination of airbag suppression or control). This can be determined by a seat track sensor. Thus, previously, electric field sensors have not been installed in the driver seat for driver monitoring. Of course, if they have been installed for health monitoring, they can also be used for driver out-of-position sensing at little, if any, additional cost.
  • Selection switches 23, 24 in FIG. 2 can, in general, cycle through the various combinations of antennas while the processor 28 searches for the strongest signals in the expected heartbeat range and in the expected respiration range and once the strongest signal in each range is found, the processor 28 locks onto that signal and monitors it for changes and/or abnormalities.
  • Such abnormalities can include slow or fast heartbeats or breathing, skipped beats, tachycardia, other irregular heart beats, irregular breathing, etc.
  • the best choice of antennas may change so the processor 28, while monitoring the chosen signals (selected at a preceding time), can continue to search for better signals (at the current time).
  • a single antenna may provide the best signal for the property to be monitored while, in others, a pair of antennas will yield a superior signal.
  • the searching may be confined to single antennas or pairs of antennas but, in some cases, the best signal may come from one transmitting antenna and two or more receiving antennas.
  • the best signals may come from antennas in the seatback whereas motion sensing antennas can involve any of the antennas.
  • the steering wheel-mounted antenna can provide a superior signal.
  • the ceiling-mounted antenna also can provide occupant motion signals.
  • a door mounted antenna can also provide valuable information.
  • FIG. 3 A perspective view with parts cutaway and removed of a vehicle showing the passenger compartment containing electric field occupant sensing systems associated with both the driver and passenger seats is illustrated in FIG. 3. Although the systems are shown to be the same on the driver and passenger seats, in fact, their function and mode of operation is different and this may dictate a different antenna arrangement for the various seats.
  • Antenna sets or groups 65 and 66 associated with the passenger seat 62 are occupant classification sensors for the purpose of determining, for example, whether there is a child or a rear-facing child seat present or if the seat is unoccupied, in which case, the airbag will be suppressed or perhaps depowered.
  • Antenna groups 65, 66 may be integrated into the passenger seat 62 in the same manner as antennas of the antenna sets 4, 6 are integrated into the seat.
  • the electric field system associated with the driver seat 61 appears to be the same, but the function is to monitor the heartbeat and respiration rate to determine whether the driver is falling asleep, having a heart attack or is otherwise unable to operate the vehicle.
  • the passenger seat 62 has electric field antenna sets or groups 65 and 66 and the driver seat 61 has electric field antenna sets or groups 63 and 64.
  • Antenna groups 63, 64 may be integrated into the driver seat 61 in the same manner as antennas of the antenna sets 4, 6 are integrated into the seat.
  • Each seat 61, 62 can have its own ECU labeled 67 in the driver seat 61 and 68 in the passenger seat 62.
  • a single ECU could service both seats 61, 62 and the ECU(s) need not be located in the seats.
  • ECU's 67, 68 may be arranged on the respective seat, in the respective seat or even separated from the seat, in all cases, coupled through wires or wireless to the respective antenna groups.
  • the heartbeat sensor, respiration sensor and motion sensors can be used with other occupant sensor systems.
  • the health and fatigue monitoring system can be only used on the driver seat and weight sensors based on strain gages or bladders, optical sensors, ultrasonic sensors, seat track and seatback sensors etc. can be used for passenger classification or, in the case of the seat track sensor, for driver classification.
  • FIG. 4 is a block diagram of a variety of occupant sensing and health monitoring systems illustrating that various systems can be used together.
  • a respiration sensor 71 is used together with a heartbeat sensor 72, an optical sensor 73, a motion sensor 74, one or more weight sensors 75, a seatback angle sensor 76, one or more ultrasonic sensors 77 and a seat track sensor 78 (although only a subset of these sensors may be provided and/or used at any one time).
  • These types of sensors are known sensors and any such sensors may be used in the invention.
  • Occupant motion can be determined by any of these technologies and need not be part of the electric field sensing system.
  • Each of these sensors can feed into an ECU 79 which controls or interfaces with the airbag system 80 and the health and fatigue monitoring system 81.
  • Alternative or additional sensors that provide information about the occupant may also be used in the invention.
  • the airbag system 80 is controlled based on data obtained from the sensors.
  • Various airbag system control techniques are envisioned.
  • FIG. 5 is a side view, with certain portions removed or cut away, of a portion of the passenger compartment of a vehicle showing several preferred mounting locations of a driver health and fatigue monitoring system and its various components. Several of the components are the same as illustrated and described above with regard to FIG. 1 and are numbered the same and will not be described again.
  • Additional electric field antennas each of which is coupled to the processor of the health and fatigue monitoring system with wires or wirelessly (ECU 9), include antenna 82 which is located in the side door alongside the driver' s seating location and is useful when the driver is leaning more against the door than the seatback.
  • Antenna 82 may represent one or more antennas. Each antenna 82 may be arranged in the door, i.e., between the panels defining the door, or on an inner side of the innermost panel.
  • Antenna 81 which can also be a set or group of one or more antennas, is located in the floor of the vehicle and is useful for monitoring motion of the driver.
  • Each antenna 81 may be arranged on top of the floor pan in front of the driver's seat.
  • the system may use this information along with other information to conclude that the driver is incapacitated, and possibly assume partial or total control of the vehicle or take other action (of the type described herein).
  • each antenna set or group 83, 84 including one or more antennas.
  • An antenna on or in the seatbelt if properly worn, can be the closest antenna to the lungs and heart and thus can provide the best heartbeat and respiration signals.
  • each antenna group 83, 84 may be associated with the seatbelt so that one or more antennas thereof is (are) closer to the lungs and/or heart of the occupant.
  • Antenna groups 83, 84 may be arranged on the seatbelt or woven into the material of the seatbelt and, in all cases, coupled through wires or wireless to the ECU 9.
  • Communications antenna 92 is illustrated for that purpose. It is coupled to, e.g., via a wired connection, and can be controlled by the health monitoring system ECU 91. Communications antenna 92 is positioned anywhere on the vehicle in which it can receive wireless signals.
  • the health monitoring system ECU 91 can also receive information from the electric field monitoring system ECU 9, via a coupling or electrical connection therebetween (not shown), and notify the driver that the electric field monitoring system detects that he or she may no longer be capable of operating the vehicle and ask for feedback.
  • a message to the driver can be in any form, such as a display, sound, light or any other system that can get the attention of the driver.
  • the health monitoring system ECU 91 can also receive information from an occupant information providing sensor 90 mounted in the A-pillar, which sensor 90 may use waves (as represented) to enable a determination of information about the occupant (e.g., in the manner of ultrasonic sensors 77 described herein).
  • a recorded or synthesized voice can speak a message using speaker-microphone 87 to the driver and ask him or her to perform some task such as respond by saying a word or by depressing the horn or button 85 or performing a gesture such as raising his/her hand or pointing.
  • the ECU 91 includes appropriate command circuitry (hardware/software) to enable the speaker-microphone 87 to generate the sound message, e.g., a message recorded in storage media at the speaker-microphone 87 or at the ECU 91.
  • a time delay between the message and the action by the driver can be a good indication of the driver's condition.
  • This time delay is computed by a processor in, for example, the health monitoring system ECU 91 that is provided with the time at which the message is conveyed to the driver via the speaker-microphone 87 or other device, and the time at which the driver provides the requested response thereto.
  • the health monitoring system can ask one or more times for a response, as above, from the driver in order to determine the alert response time of the driver and to train the driver as to what to expect so that he/she is not surprised at a later time.
  • this initial test can determine that the driver is not fit to drive the vehicle and prevent him/her from doing so (via any vehicle operation suppression system known to those skilled in the art including any of these described herein). If the driver is the vehicle owner, then he/she should be familiar with the operation of this system and if he/she responds differently than at some previous time, then again vehicle operation can be suppressed.
  • FIGS. 6A-6C illustrate the principles of operation of a single electric field monitoring element in the presence of a body element and a modified body element, which principles may be used in any of the embodiments of the invention herein.
  • FIG. 6A illustrates the case where there is no body element to affect the electric field and thus current il is the resulting current in the circuit.
  • Oscillator 101 provides the electric field to the antenna 100.
  • a body element 103 is placed in the electric field, a different current i2 results as shown in FIG. 6B.
  • a driver When a driver is breathing, his effect on the electric field changes as he breathes and thus there will be a cyclic variation in ⁇ 2- ⁇ 3 which corresponds or can be related to his respiration rate. If he/she suddenly stops breathing, as might happen during a heart attack, the cyclic variation will stop and this can be determined by monitoring the variation in the current in the antenna circuit, i.e., by a current monitor that is known to those skilled in, for example, the electrical art. In a similar manner, the heartbeat can be monitored. Finally, in the case where only the heartbeat is monitored, the respiration signal can be derived from the heartbeat signal. This is further made possible since the heartbeat frequency is substantially higher than the respiration frequency and thus the two signals can be readily separated by signal processing techniques that are well known to those skilled in the art.
  • FIGS. 7A-7C illustrate principles of operation of a pair of interacting electric field monitoring elements in the presence of a body element and a modified body element.
  • two antennas are selected which are those that are best able to monitor the heartbeat or respiration rates.
  • the effect of breathing or the beating heart is to change dielectric properties of the body element and thus the current in the antenna circuit, i.e., from i4 wherein there is no body element (FIG. 7A), to i5 wherein the body element 103 is placed into the electric field between the two antennas (FIG. 7B), to i6 wherein the body element 103 changes shape to 104 (FIG. 7C).
  • FIG. 8 A block diagram illustrating one possible relationship between an occupant sensing unit 120 and a vehicle safety system 130 is illustrated in FIG. 8.
  • An electric field monitoring system 121 is connected to a diagnostics and data logging module 123 which verifies that the system 121 is operational and logs any error codes.
  • Electric field monitoring system 121 can also log particular characteristics of a particular driver if a set of individual preferences are identified and associated with the normal drivers of the vehicle. Logging of error codes and particular characteristics of an identified driver may be performed in a manner known to those skilled in the art, with the logged data being stored in storage medium accessible by a processor.
  • the electric field monitoring system 121 would know what the normal heartbeat rate and respiration rate of driver 1 is and can then better identify anomalies as they occur, once the driver is identified by accessing the logged data previously entered into the storage medium.
  • a driver for example, may have a history of skipping beats and it must be known that this should not be a concern of the health monitoring system, unless the rate of skipped beats increases which can be attributed to a potential medical problem, alcohol or drug abuse.
  • Such a system may also be used as a biometric measure to identify the driver along with normal heartbeat rate and respiration rate, for example.
  • Electric field monitoring system 121 also provides data for the passenger occupant identification and location system 122, the conventional use of electric field sensors, and the driver health and fatigue monitoring system 124 (see FIG. 8).
  • Occupant identification and location system 122 connects with the vehicle airbag system 131 and controls the vehicle airbag system 131, such control being variable and including causing inflation or deployment of one or more airbags in the airbag system 131 to be suppressed for a rear-facing child seat, for example.
  • Driver health and fatigue monitoring system 124 similarly connects to a vehicle warning and verification system 132 which tests the driver to see whether he is capable of operating the vehicle and if not, control of the vehicle can be taken over by a vehicle control system 133.
  • FIG. 9 An exemplifying, non-limiting flow chart showing the operation of the driver health and fatigue monitoring routine, for any of the embodiments described herein, is shown in FIG. 9.
  • the routine is illustrated generally at 140 and when invoked, starts the select antenna routine 141 which surveys all, or a subset, of the various antenna scenarios, including each one individually and then in pairs. Select antenna routine 141 will look for the strongest antenna signals for heartbeat, respiration rate and occupant motion. Once selected, control passes to the modules that acquire respiration 142, acquire heartbeat 143, and acquire motion 144 and establish or record normal respiration 145, normal heartbeat 146, and normal motion frequency 147, what will be termed the established respiration rate, heartbeat rate and typical occupant motion.
  • the time periods for each of these functions will often differ significantly. Less than one minute is generally sufficient to establish a representative heartbeat rate, several minutes is generally sufficient to establish a representative respiration rate and a longer time may be needed to establish how the particular driver typically moves while operating the vehicle.
  • Establishment of a normal motion is the least important of the three since it will mainly become important when an anomaly has been detected in the heartbeat or respiration rates. If the driver has stopped his normal motion when there is a significant heartbeat event, then this may confirm that there may be a problem with the driver, rather than a failure of the electric field monitoring system 121. Also, significant motion may be a clue that the select antenna routine 141 requires reactivation. Thereafter, the respiration, heartbeat and motion are acquired at 148, 149, 150, respectively.
  • Steps 151, 152 and 153 determine whether any of the measured heartbeat, respiration and motion rates are normal and if not, i.e., if one is not normal, a warning and feedback system is invoked or activated at 154. If the driver fails a feedback test generated by the warning and feedback system, step 155, then a routine to control the vehicle is invoked or activated at 156.
  • the vehicle control routine that is invoked at step 156 may involve, for example, signaling to a remote site or facility for attempted driver communication and then for exercising vehicle control provided there is sufficient equipment on the vehicle to allow the remote operator to observe the driving conditions, traffic, etc. and thus permit a remote operator to slow the vehicle and guide it off onto a shoulder.
  • the vehicle control routine 156 may also be performed partly or entirely by equipment on the vehicle upon command by a software program that is executed when needed.
  • the equipment necessary for the control unit to invoke control of the vehicle includes, but is not limited to, devices that are coupled to the steering wheel or steering system, the brakes or braking system, the throttle system, transmission and/or engine. These devices may be controlled by software to reduce the speed of the vehicle to a stop and direct the vehicle to a safe area, e.g., the side or shoulder of a road.
  • the vehicle control routine 156 may therefore correspond to or encompass a processor and control equipment that is coupled to vehicular parts that affect operation of the vehicle, e.g., the brakes, steering wheel and throttle, and possibly servos or actuators therefor. The manner in which this structure operates to effect vehicular control would be easily understood by those skilled in the automotive art in view of the disclosure herein.
  • the presence and characteristics of the heartbeats of vehicle occupants can be valuable information that can and should be communicated by the communications unit to EMS personnel as part of the automatic collision notification message, if it is available. Therefore, the EMS personnel will know whether the occupants are alive and some indication as to their health state.
  • This may be effected using a communication system or a transmission system on the vehicle (not shown) that is coupled to the vehicle safety system 130, or a part thereof, and obtains and transmits the information about the occupant(s) of the vehicle.
  • the transmission may be accompanied by a location of the vehicle, obtained using a location determining system on the vehicle or apart from the vehicle, and may be effected using any known communications protocol, including using the Internet.
  • FIG. 10 An exemplifying, non-limiting flow chart showing the operation of an exemplifying, non- limiting warning and feedback system routine, for any of the embodiments described herein, is shown in FIG. 10.
  • the routine is illustrated generally at 180.
  • a first step in the routine 180 is to initiate a warning set counter.
  • the second step 182 is to generate and send a message in some form to get the attention of the driver.
  • This message can be in any of several forms, e.g., audible, visual, tactile, such as a warning buzzer or a flashing light which can suffice, but this may confuse a driver that is unfamiliar with the system.
  • a message can be displayed on one or more of such displays.
  • a preferred message is to have a synthesized or recorded voice announce a message to the driver.
  • a combination of the above warnings can be better still.
  • the message can instruct the driver to take some deliberate action which will not interfere with his ability to operate the vehicle but will nevertheless indicate that he has received the message and responded in a timely manner.
  • Another display that can be used when sufficient data is available is a light or other visual display that shows or otherwise depicts the calculated awake state of the driver.
  • a driver can monitor this display and get an indication as to whether the system thinks that he/she is falling asleep or otherwise experiencing a decrease in his/her ability to operate the vehicle.
  • a driver that sees that he/she is gradually becoming drowsy can, and will ideally, plan when to stop and rest or get a cup of coffee.
  • the driver is sure that the system is in error, he/she can provide feedback to the system that can be taken into account to improve the system accuracy.
  • One candidate device is the Ambient Orb from Ambient Devices, Cambridge, MA which is programmed to change color based upon the output of a state detection system.
  • the counter is incremented at 183 and the system waits for a response at 184. If the response is not the expected response or if it is tardy (tardiness being determined relative to a predetermined time threshold that may be a function of, for example, the message or the current situation), then the response is judged to not be acceptable at step 185. If the response is judged acceptable, then control is passed back to the health monitoring routine at step 188. The count of the counter is checked at step 186 after an unacceptable driver response and if the counter exceeds some predetermined limit, such as three, then the vehicle control routine is invoked at step 187.
  • step 182 If the counter is below its limit, then control is passed to step 182 and the driver is given another chance to respond to the message. As shown in FIG. 10, the routine returns to step 182 to send a message to the driver. However, if the message is a continuous message, such as one on a display, then the routine could be programmed to return to the increment counter step 183 (see the dotted line in FIG. 10).
  • the system can also provide some advice, especially after a number of detected abnormalities that perhaps the driver should take a break. If the driver does not take the advice and there is a likelihood that the driver will in fact fall asleep, then the authorities can be notified.
  • Such notification may be additional or alternative to the invoke vehicle control routine 187, see step 189 in FIG. 10.
  • This notification may be provided by a communications system or transmission system (not shown in FIG. 10) that is coupled to the electric field monitoring system 121 and generates and sends an appropriate message relating to the lack of the driver providing a response for the predetermined amount of time to the sent message.
  • Another feedback system that has been suggested is to provide some change in vehicle operation that requires a driver response and then measure the response time.
  • One such idea is to perturb the steering wheel either with a pulse or with a gradual drift of the vehicle to one side of the road and then measure how quickly and even if the driver makes a response.
  • FIG. 11A is a view of an optical monitoring system that monitors the face of the driver from a camera and illumination system 200 mounted on or near a ceiling 204.
  • One preferred location for example, is on or adjacent to the rear view mirror 202 (schematically illustrated) since that location usually has a good view of the driver' s face and is not obstructed by the visor.
  • the optical monitoring system comprises a camera and a source of illumination, represented by the camera and illumination system 200, which is usually in the non-visible part of the electromagnetic spectrum and in particular, in the infrared (IR) portion of the electromagnetic spectrum.
  • the optical monitoring system also comprises a processor or processing unit coupled to the camera, and optionally the illumination source (the processor may be resident or part of the ECU 210 described below with reference to FIG. 11B). This processor is configured to analyze the images to obtain information or data therefrom, and specifically, from images of blood vessels, capillaries and/or veins in a face of the occupant (driver) being imaged.
  • the face of the driver is being illuminated and images thereof are obtained after illumination, this is partly as a result of the face being the part of the driver that is most likely to be exposed, i.e., uncovered by a material that would block infrared radiation. Nevertheless, other parts of the driver may be illuminated and images thereof obtained. These parts should preferably be exposed parts of the driver, where reflections from blood vessels, capillaries and/or veins may be obtained.
  • Blood in particular, reflects in the IR portion of the spectrum and thus permits a clearer image of the blood in the blood vessels, capillaries and veins in the driver' s face to be obtained and visible in the images being obtained by the camera of camera and illumination system 200. Pulsations of the blood flow can then be used to measure the heartbeat rate of the driver. In particular, the heartbeat variability can be analyzed by the processor as this permits a determination of the drowsiness and/or other attributes of the driver, or more generally, the loss of ability of the driver to continue to control the vehicle. This is discussed below in the analysis section.
  • Another use of this technology is to use the pattern of blood vessels, capillaries and veins in the driver's face as a biometric identifier of the driver.
  • the processor would perform this analysis to identify the driver and then, control access to the vehicle, adjust seating conditions, adjust environmental conditions, etc., based on pre-set information about the biometrically-identified driver.
  • the temperature of the driver's face or portions thereof can also be measured by a temperature measurement device or algorithm and compared to ambient temperature as measured by a temperature sensor to determine whether the driver's temperature is normal or indicative of a medical problem.
  • the overall motion of the driver's face can also be tracked by the processor, e.g., using a tracking algorithm or under control or command of software available to the processor, to determine the respiration rate of the driver.
  • the face location can be determined in the images using, for example, pattern recognition techniques operated by the processor.
  • the degree to which the blood capillaries, vessels or veins change in size during a beat or their absolute size, as determined by the processor can provide a measure of the driver' s blood pressure.
  • the pulse rate has been found to be best determined from the green part of the optical spectrum. Since blood is particularly absorbent of frequencies in the 500 nm to 600 nm range, the illumination can contain frequencies in this range. Since green light can be distracting to the driver and vehicle occupants, care must be exercised as to how it is used.
  • a patch on the forehead, cheek, neck or other convenient place can be chosen as the spot to be illuminated. Then, the amount of illumination can be adjusted, via control of the source of illumination by the processor or control unit more generally, so that it is barely above, typically 1% to 10% above, that present in the ambient light level.
  • the green light can also be modulated, via a modulation device or technique, so that it can be separated from ambient light reflections. This may require that the receiving device for the green light be a photodiode or equivalent to permit the modulations to be sensed and separated from the ambient light.
  • Camera and illumination system 200 can also be used to track the eye and eyelid motions of the driver as is discussed in prior art eye tracking systems for determining driver alertness. As mentioned above, such systems can be used to augment the monitoring system discussed herein but, when relied on alone, suffer from problems associated with sunglasses and even ordinary glasses which frequently block IR.
  • FIG. 11B illustrates an alternate arrangement for monitoring the face of a driver.
  • a camera 206 is mounted below the top surface of the instrument panel in such a position as to view the driver's face through a reflection off of a surface 208 of the windshield in a manner similar to heads-up displays.
  • This implementation has the advantage that a larger camera can be used with greater resolution and can contain focusing hardware and even pointing hardware under control of an ECU 210 to obtain a better view of the face of the driver and to more exactly project the artificial illumination and particularly the patch of green light discussed above.
  • This patch projection will move as the driver' s head moves, under control of a processor and appropriate control algorithm or program in the ECU 210.
  • FIG. 1 IB is similar to that of FIG. 11 A.
  • the same ECU 210 can be used to analyze the driver for signs of fatigue and initiate a warning such as a light 214 which requires a feedback response from the driver, such as depressing button 212, speaking a word, pointing etc. as described above.
  • Temperature variations can be monitored to yield the respiratory cycle. Thermal face measurements can also provide evidence of inebriation or drug use that can impair the driver' s ability to operate the vehicle safely.
  • the monitoring of sudden head motion can indicate an attempt by the driver to wake from a micro-sleep.
  • the area adjacent the eyes can exhibit pulsation corresponding to the heartbeat.
  • Saccade frequency rapid eye movements
  • respiration wave form was accurately determined for several people from a distance of 6-8 feet indicating that video monitoring with the appropriate illumination can measure respiration.
  • monitoring of the driver's chest can augment facial monitoring for determination respiration.
  • Major arteries in the neck can be used to measure heartbeats through thermal imaging if these are visible to the thermal imaging system. This becomes problematic when the driver is wearing a winter coat or sweater which covers these arteries.
  • Another approach is to simultaneously monitor the gaze direction of the driver and the events occurring in the space surrounding the vehicle. If such an event catches the attention of the driver, then the speed of his/her reaction can be measured as an indicator of inattention. If the driver misses a series of events that he/she should not have missed, then this can be a dramatic indication of inattention and the driver should be tested.
  • the volume of air coming from the nostrils is measured in accordance with apparatus and techniques disclosed in U.S. Pat. No. 5,689,241 to determine whether the person is getting enough oxygen. If not, then drowsiness can quickly result. This is known as hypoventilation. This can be done using far IR illumination in the 6-15 ⁇ wavelength range.
  • FIGS. 12A and 12B A variety of different sensors that also have some capability of measuring the heartbeat and/or respiration rate, or contributing the determination of the heartbeat and/or respiration rates, as well as gross movement of the driver are shown in FIGS. 12A and 12B. These sensors can be used in combination with the systems disclosed elsewhere herein or independent therefrom.
  • strain gage weight sensor 224 can be positioned within the seat bottom cushion (see FIG. 12B) in such a manner so that it also can measure the heartbeat and respiration rates of the driver although not as accurately as the electric field or optical sensors discussed elsewhere herein.
  • Fluid-filled pads have been used for detecting heartbeat and respiratory motions in patients and infants but not to vehicle operators or passengers as reported in U.S. Pat. Nos. 5,515,865, 5,684,460 and 5,853,005.
  • Use of strain gages for heartbeat and respiration measurements to determine the presence and classification of an occupant is disclosed in U.S. Pat. No. 7,019,641; however, there is no disclosure as to how to use the invention to detect drowsiness.
  • Another method of monitoring the heartbeat of an occupant of a vehicle is to make use of a local relative displacement measuring seat mat.
  • This device can be implemented in several versions with the preferred version comprising a grid of small cells which can be from a few millimeters to a centimeter is size and where for each cell the displacement of the top surface relative to the bottom surface can be monitored using, for example, capacitance.
  • the heartbeat is detectable in several large veins or arteries in the butt of the occupant and this causes relative displacement of the tops of several cells toward the bottoms. If both the top and bottom of each cell is conductive then the capacitance changes which can be measured.
  • each cell can be individually interrogated to determine its capacitance.
  • the tops and bottoms of the cells can be held separate from each other through metal or other springs or through compressible fluid as, for example, in BubblePak®.
  • This technique can be used to determine the heartbeat rate independently of the heartbeat rate as determined by other systems or techniques disclosed herein, or to verify the heartbeat rate as determined by one of the other systems techniques disclosed herein, or vice versa.
  • Ultra wide band (UWB) radar is particularly appropriate for this application. This is also reported more recently for use in medicine in "UWB radars in medicine", Staderini, E.M., Aerospace and Electronic Systems Magazine, IEEE, Jan 2002 Volume: 17 Issue: 1, On pages 13-18.
  • the radar vital signs monitor disclosed therein is able to measure the subject's heartbeat and respiration rate at a distance up to 30 feet.
  • "Non-Contact Measurement of Heart And Respiration Rates With A Single-Chip Microwave Doppler Radar” Amy Diane Droitcour, PhD Dissertation Stanford University, June 2006, discloses the availability of such radars on a chip.
  • Sensitive accelerometers are capable of yielding signals from which the heartbeats and respiration can be measured. This is partially disclosed in U.S. Pat. No. 6,011,477 for the monitoring of infants. It is even possible to place accelerometers in the seat bottom for the same purpose with significant noise problems caused by vehicle motion.
  • Direct measurement of EEG emissions from the head of the driver can, in principle, be done in a non-contact manner as disclosed in U.S. Pat. Appln. Publ. No. 20080015801 ; however, the equipment disclosed in very bulky and expensive and thus not suitable for in-vehicle monitoring. EEG signals are probably the most reliable in determining the onset of sleep so, as this equipment is improved, it may become available for in-vehicle use.
  • the SmartCap discloses a cap that can monitor brain emissions as a sort of EEG as disclosed above.
  • Heartbeat rate An important part of the determination of driver drowsiness is in the analysis of the heartbeats and respiration. Looking at the heartbeat rate alone, one can say that if the rate is too high, the driver may be having a seizure or other perhaps critical health problem. If it is too low, this too can indicate a serious medical problem or that the driver is under the influence of drugs or alcohol. An irregular rate may indicate a heart attack, stroke or tachycardia event. Although a low heart rate may indicate that the driver is asleep, this is not always the case and by the time that it is detected, it may be too late.
  • respiration rate when the respiration rate is too high, a seizure or other serious medical problem may be in progress. Too low of a respiration rate may indicate sleep, alcohol or drugs and an irregular rate may indicate suffocation, a heart attack, stroke or other medical problem. Again, by the time that it is detected, it is usually too late to prevent an accident.
  • HRV heart rate variability
  • HRV is a measure of the oscillation of the interval between consecutive heartbeats.
  • the literature is replete with references to this measure that are expressed in terms of consecutive cardiac cycles. Such terms include: cycle length variability, heart period variability, R-R interval variability (referring to the beat-to- beat interval formed by consecutive R-waves of the cardiac signal), and the R-R interval tachogram.
  • the spectral components of the cardiac interval signal for short-term changes in HRV fall into three frequency ranges: the Very Low Frequency (VLF) range, the Low Frequency (LF) range, and the High Frequency (HF) range.
  • VLF Very Low Frequency
  • LF Low Frequency
  • HF High Frequency
  • the LF component is of chief interest as an indicator of cognitive workload.
  • This component commonly referred to as the 0.10 Hz component, reflects short-term changes in blood pressure and spans a range from 0.04-0.15 Hz. That is, there are approximately 6 fluctuations of the heart per minute due to changes in blood pressure. A peak in this component is indicative of lower cognitive workload conditions. A flattening of this component reflects conditions of greater mental workload.
  • Other influencers of HRV include smoking and the acute ingestion of alcohol both of which reduce HRV.
  • CHF chronic heart failure
  • SDNN Standard deviation of the time between beats
  • an irregular pulse rate can indicate cardiac abnormality which may or may not be serious. If this occurs suddenly and does not go away, the driver should be warned and the system should be prepared for the need to assume control of the vehicle.
  • the heart rate will fluctuate with the breathing cycle; inspiration is accompanied by heart rate elevation and expiration is accompanied by heart rate depression.
  • the respiration cycle can be determined from the heart rate, although it is more accurately determined by direct measurements. In some of the systems disclosed herein, only the heartbeats are measured and thus the respiration cycles must be determined from the heart beat cycles.
  • Heart Rate Variability in the VLF range decreases consistently and significantly minutes before falling asleep events.
  • the sympatho-vagal balance is very low compared to baseline wake values for about 5 minutes before the events.
  • the behavior of the LF/HF represents a low sympatho-vagal balance before FA as a sign of the relaxation taking over. After the FA, there is a surge in this balance signifying an increased stress aimed to overcome the drowsiness.
  • the very significant RRI changes with decreased variability and lower heart rate during FA events confirm the autonomic changes: during a microsleep (MS), there is a decrease in sympathetic drive with increased vagal activity, (ibid G Dorfman Furman et al.)
  • the first microsleep (MS) during MWT and driving simulator (DS) represented a point for significant changes in all HRV measures.
  • the first accident on DS occurred 2-7 minutes after the first MS.
  • HRV markers that indicate sleepiness in sleep deprived subjects. Provided some of these variables show the same trends in sleepy non sleep deprived subjects, a threshold should be defined as to imminent danger of a driver falling asleep at the wheel, (ibid. Gabriela Dorfman Furman et al.)
  • U.S. Pat. Nos. 6,669,632 and 7,397,382 disclose that lung cavity pressure (pleural) can be determined by HRV and is a measure of sleepiness giving another HRV property that can be used to predict the level of drowsiness that the driver is experiencing. Other studies have indicated that even changes in blood pressure can be estimated from HRV.
  • HRV HRV
  • Most of these methods deal with the heartbeats as normal sine wave cycles.
  • the shape of the heartbeat and of the respiration wave contain important information to further refine the falling asleep prediction.
  • the particular analysis method will depend on the fidelity of the waves obtained and thus the exact analysis method used will vary depending on the data acquisition method used.
  • One preferred technique that is generally applicable is to use neural networks that have been trained on the system with the chosen data acquisition method.
  • the input to the input layer nodes will be the raw heartbeat data with sufficient resolution as to capture the shape of each pulse when this differs from a sine wave, or the periods and amplitudes of the waves when a sine wave is a good approximation to the waves.
  • a neural network can then be trained using a driving simulator using a sufficient number of subjects as to capture the population at large. This can form the basic analysis program which can be modified in use based on measurements of a particular driver.
  • the system can recognize that this is the case, by face or other biometric recognition, and learn the normal heartbeat and respiration properties of that particular driver.
  • the program can revert to the generic population program.
  • the analysis program will take other information into account when it is available such as the temperature, blood pressure and oxygen level of the blood. This will depend on the particular suite of sensors that are used and such information will not be available, for example, if electric field sensors are the only ones being used.
  • Other information that can also be incorporated includes lane departure, steering wheel motion, vehicle speed and acceleration, time of day, head and body motion, state of the eyes and blinking rates and any other measured properties of the occupant, vehicle and how it is being driven and/or the environment.
  • Pulse recovery can be aided by removing signals with frequency lower than about 0.67 Hz
  • the heart rate decreases during sleep by about 5-40 %.
  • Blood pressure decreases during sleep to about 77-80 % of its value when the driver is awake.
  • the HRV is much larger during sleep. That is, one beat interval may be about 0.9 seconds and the next one about 1.3 seconds in the sleeping person while in the awake person, about 0.7 to about 0.8 seconds.
  • Gesture detection systems have been reported by the ⁇ Media Lab, but these concepts have not been applied to the sensing of gestures in a vehicle. See, for example, “Musical Applications of Electric Field Sensing", Joseph A. Paradiso and Neil Gershenfeld, Draft 1.9, April 1996; to be published in Computer Music Journal and “Talking to the Wall”, Kate Greene, MIT Technology Review, May 3, 2011, and U.S. Pat. Nos. 5,844,415 and 5,936,412.
  • Gestures that are recognized by electric field systems can also be used to control other vehicle components. If a heads up display is present, then the driver can change the display contents by pointing at a selection of the display in a manner that can be recognized by an optical or electric field gesture recognition system. It has been demonstrated that electric field gesture recognition systems can have a high resolution and are capable of measuring displacements on the millimeter scale.
  • wires and connectors are not only expensive components to make and install in a vehicle, they are also the largest source of warranty repairs. Thus, there is a dire need to eliminate wires and connectors wherever possible. Much of this can be accomplished through use of electric field systems such as the Hyundai Elesys electric field occupant sensor. This invention can be implemented in many ways, only one of which will be discussed here.
  • a vehicle-wide power door lock switch is present in the driver' s door, for example, is replaced by a grounded metal plate having approximately the same area and the plate is connected to ground, then when the driver, of a vehicle with an electric field antenna in the driver seat touches the plate, the electric field is shorted to ground and the current in the seat antenna will change in a detectable manner.
  • a door operated switch can be replaced by a metal plate. The electric field system would quickly get confused if there were more than one such switch having a different function.
  • This problem can be solved by connecting the switch to ground through an impedance. This can be in the form of a resistance, but this might require that every driver have the same conductance and thus another solution is preferred. If the electrical field creating circuit can vary its frequency and if the plate switch is connected to ground through a resonating circuit, then the electric field system would show a maximum or minimum current as a function of frequency and if each switch was tuned to a different frequency, then they could each respond differently resonating, for example, at a different frequency.
  • One skilled in the art can now find or develop many other circuit designs that would allow one switch to be distinguishable from another so this invention is not limited to the tuned circuit design. By this manner, much of the wiring, many of the switches and many of the connectors can be eliminated from the vehicle, reducing its cost and reducing its maintenance costs.
  • the switch need not be a metal plate and can be anything that is conductive and connected to ground through an impedance or other electrical circuit or element. In some cases, it might be desirable to retain the switch but instead of connecting it to the vehicle electrical system, it is connected to ground as in the plate example above. In this case, the vehicle occupant would connect the impedance to ground by depressing the switch. This can be desirable when the duration of switch closure is used to control the degree of window opening, for example.
  • Electric field sensors and wave sensors are sometimes similar from the point of view of sensing the presence of an occupant in a vehicle.
  • a time varying electric field a form of a time-varying signal, is generated and directed into the volume of the passenger compartment and then disturbed or modified by the presence of the occupant.
  • Different occupants or occupying items will cause different disturbances or modifications.
  • the sensor is based on its capability to sense a change of wave characteristics of the electromagnetic field, such as amplitude, phase or frequency. As the frequency drops, other characteristics of the field are measured.
  • the occupant's dielectric properties modify parameters of the reactive electric field in the occupied space between or near the plates of a capacitor.
  • the sensor senses the change in charge distribution on the capacitor plates by measuring, for example, the current wave magnitude or phase in the electric circuit that drives the capacitor. These measured parameters are directly connected with parameters of the displacement current in the occupied space.
  • the presence of the occupant reflects, absorbs or modifies the waves or variations in the electric field in the space occupied by the occupant.
  • the images from imagers are different but there are also similarities between all images of rear facing child seats, for example, regardless of where on the vehicle seat it is placed and regardless of what company manufactured the child seat. Alternately, there will be similarities between all images of people sitting on the seat regardless of what they are wearing, their age or size.
  • the problem is to find the "rules" which differentiate the images of one type of object from the images of other types of objects, e.g., which differentiate the occupant images from the rear facing child seat images.
  • the similarities of these images for various child seats are frequently not obvious to a person looking at plots of the time series and thus computer algorithms are developed to sort out the various patterns. For a more detailed discussion of pattern recognition, see, for example, U.S. Pat. No. RE 37,260.
  • Determination of these rules is important to the pattern recognition techniques used in at least one of the inventions disclosed herein such as the optical monitoring of the face of the driver.
  • three approaches have been useful, artificial intelligence, fuzzy logic and artificial neural networks (including cellular and modular or combination neural networks and support vector machines - although additional types of pattern recognition techniques may also be used, such as sensor fusion).
  • the rules are sufficiently obvious that a trained researcher can sometimes look at the returned signals and devise a simple algorithm to make the required determinations.
  • artificial neural networks can be used to determine the rules.
  • Electromagnetic energy based occupant sensors exist that use many portions of the electromagnetic spectrum.
  • a system based on the ultraviolet, visible or infrared portions of the spectrum generally operate with a transmitter and a receiver of reflected radiation.
  • the receiver may be a camera or a photo detector such as a pin or avalanche diode.
  • the absorption of the electromagnetic energy is primarily used and at still other frequencies, the capacitance or electric field influencing effects are used.
  • the human body will reflect, scatter, absorb or transmit electromagnetic energy in various degrees depending on the frequency of the electromagnetic waves.
  • any portion of the electromagnetic signals that impinges upon, surrounds or involves a body portion of the occupant is at least partially absorbed by the body portion.
  • the human body is composed primarily of water, and that electromagnetic energy of certain frequencies is readily absorbed by water.
  • the amount of electromagnetic signal absorption is related to the frequency of the signal, and size or bulk of the body portion that the signal impinges upon. For example, a torso of a human body tends to absorb a greater percentage of electromagnetic energy than a hand of a human body.
  • electromagnetic waves or energy signals are transmitted by a transmitter, the returning waves received by a receiver provide an indication of the absorption of the electromagnetic energy.
  • absorption of electromagnetic energy will vary depending on the presence or absence of a human occupant, the occupant's size, bulk, surface reflectivity, etc. depending on the frequency, so that different signals will be received relating to the degree or extent of absorption by the occupying item on the seat.
  • the receiver will produce a signal representative of or corresponding to the returned waves or energy signals which will thus constitute an absorption signal as it corresponds to the absorption of electromagnetic energy by the occupying item in the seat.
  • One or more of the transducers can also be image-receiving devices, such as cameras, which take images of the interior of the passenger compartment including the face of the driver. These images can be transmitted to a remote facility to monitor the passenger compartment or can be stored in a memory device for use in the event of an accident, i.e., to determine the status of the occupant(s) of the vehicle prior to the accident. In this manner, it can be ascertained whether the driver was falling asleep, talking on the phone, etc. Alternately, the images can be forwarded to a remote monitoring site when the driver fails to respond to the determination of the systems described herein that he/she is falling asleep, having a heart attack, under the influence of drugs or otherwise unable to effectively operate the vehicle. The remote site can observe and monitor the driver and issue verbal instructions as to what action the driver should take. If the driver fails to heed the directions then the vehicle can be gradually immobilized using technology available today.
  • image-receiving devices such as cameras, which take images of the interior of the passenger compartment
  • a memory device for storing images of the passenger compartment, and also for receiving and storing any other information, parameters and variables relating to the vehicle or occupancy of the vehicle may be in the form of a standardized "black box" (instead of or in addition to a memory part in a processor).
  • the IEEE Standards Association is currently beginning to develop an international standard for motor vehicle event data recorders.
  • the information stored in the black box and/or memory unit in the processor can include the images of the interior of the passenger compartment including the face of the driver as well as the number of occupants and the health state of the occupant(s).
  • the black box would preferably be tamper-proof and crash-proof and enable retrieval of the information after a crash.
  • radar- based sensors can also be used. Radar also can be used to determine the presence and location of occupants in a vehicle.
  • a heartbeat sensor based on micropower impulse radar is described in McEwan, U.S. Pat. Nos. 5,573,012, 5,361,070 and 5,766,208.
  • MIR micropower impulse radar
  • This heartbeat sensor can be positioned at any convenient position relative to the seats where occupancy is being monitored. A preferred location is within the vehicle seatback.
  • This MIR sensor as disclosed in these patents is to determine the presence of a heartbeat and not used to disclosed the health and fatigue state of the driver as disclosed herein.
  • Another method which can be applicable when a camera is used is to allow for a gesture response. Perhaps the driver has a speaking impediment and the system requires a spoken response to the interrogation. In such cases, an alternative can be provided whereby the driver indicates his response through a hand and/or head motion. As long as it is the expected motion, this response can be acceptable.
  • a sophisticated algorithm can interpret a gesture, for example, that may be in response to a question from the computer system (conveyed on a display or vocalized using a speaker). The question may be one of a plurality of different questions, so that the same gesture can be used in response to multiple questions and interpreted only in consideration of the posed question.
  • the driver may indicate by a gesture, within a reasonable time after the posing of the question, that he or she is OK and the system can then interpret a "thumbs up" gesture for such an OK response.
  • a gesture within a reasonable time after the posing of the question, that he or she is OK and the system can then interpret a "thumbs up" gesture for such an OK response.
  • a living object such as an animal or human has a fairly high electrical permittivity (Dielectric Constant) and relatively lossy dielectric properties (Loss Tangent) absorbs a lot of energy when placed in an appropriate varying electric field. This effect varies with the frequency. If a human, which is a lossy dielectric, is present in the detection field, then the dielectric absorption causes the value of the capacitance of the object to change with frequency. For a human (poor dielectric) with high dielectric losses (loss tangent), the decay with frequency will be more pronounced than for objects that do not present this high loss tangency. Exploiting this phenomena it is possible to detect the presence of an adult, child, baby or pet that is in the field of the detection circuit.
  • Dielectric Constant electrical permittivity
  • Loss Tangent relatively lossy dielectric properties
  • the processor may be programmed to vary the acceptable response time indicative of alertness of a driver for different drivers, so that one response time for one driver may be acceptable while the same response time for another driver may be indicative of that driver being incapacitated.
  • the direction of the road ahead can be determined soon after driving begins as the driver will most likely be looking at the road at least at the start of his trip and for most of his driving time. Tracking the head from that point forward, based even on a camera overhead, is expected to be relatively easy and the direction of gaze determined.
  • the driver can fall asleep with his gaze unchanged so in addition to motion of the driver' s head, motion of the steering wheel, the position of the vehicle relative to the lane (as determined from an outside looking camera) may be monitored and analyzed in combination therewith to enable a determination of the driver's ability to operate the vehicle.
  • monitoring of the steering wheel which may be effected by a sensor that monitors rotation of the steering wheel relative to the steering column, lack of turning of the steering wheel or erratic, uncontrolled turning of the steering wheel would be an excellent confirmatory indication of the driver falling asleep.
  • Other training of the processor or pattern recognition technique used thereby can involve motion statistics that lead to an expectation as to what a particular driver does when he is alert. If the driver passes the test, then the thresholds can be modified. In particular, as a person begins to fall asleep, he can execute some jerking motions or other telltale motions that will be different from his normal alert behavior. Any such out-of-the -ordinary movements can evoke the test of his response time.
  • Such unordinary movements can be programmed into the trained pattern recognition technique in the processor, i.e., embodied on computer-readable medium accessible by the processor, so that once one of these movements is detected by the pattern recognition technique in the processor, the processor would control the reactive component(s) accordingly, or otherwise monitor detection of a response to cue or other required response by the driver indicating alertness.
  • optical monitoring of the driver's head or part thereof is most likely to enable an adequate determination of whether the driver is falling asleep or otherwise unable to operate the vehicle
  • other types of systems for monitoring the driver may be used alone or in combination with the optical monitoring system, including ultrasonic sensors, electric field sensors, bladder sensors, heartbeat sensors, respiration sensors and/or strain gage weight sensors to aid in this determination.
  • the processor would be coupled to, for example, the strain gage weight sensors which provide an indication of the weight distribution of the occupant, and would analyze both the images obtained by the optical imaging system and the data provided by the strain gage weight sensors in order to make the determination of the driver's ability to operate the vehicle.
  • a liquid-filled bladder could be placed in contact with the driver on the seat and the heart rate and breathing can be monitored thereby (see, e.g., U.S. Pat. No. 3,727,606), and/or an ECG can be picked up with conductors put into the seat surface (see, e.g., U.S. Pat. No. 3,954,100) or by a Pessey ECG sensor such as PS2501. These would provide additional data for use by the processor when making the determination as to whether the driver is falling asleep or otherwise unable to operate the vehicle. Any such sensors may be coupled by wiring or wirelessly to the processor.
  • Additional data for use by the processor may include data from an accelerometer arranged in seatbelt and coupled to the processor.
  • Such an accelerometer is designed to measure the heartbeat and/or respiration of the driver which could be correlated to and used in a determination by the processor that the driver is falling asleep or otherwise unable to operate the vehicle.
  • the precise motion of the chest of the occupant may be determined using a laser, for example, arranged in front of the driver, e.g., in the dashboard.
  • the laser would also be coupled to the processor which would use the data from the laser in its determination of the attentiveness or alertness of the driver. This could be a low cost way of monitoring respiration of the driver, which is correlated to the driver's ability to operate the vehicle. If the laser operates in the terahertz frequency range, it can pass through the driver's clothes and reflect off of the driver's chest providing a more accurate motion of the chest.
  • the processor which makes the determination of the driver's ability to operate the vehicle may also be arranged to control a system which generates a cue such as an audio request for the driver to "look at the road”, generates a rumble strip sound over the speakers in the vehicle and/or generates a vibration of the seat and/or steering wheel.
  • a cue such as an audio request for the driver to "look at the road”
  • the processor When people run over a rumble strip, the natural reaction would be to look at the road. In that case, the driver would not have to be trained to, for example, depress the horn button. If he or she shifted his or her gaze in response to the rumble strip noise, which would be a sufficient response and the reaction time can be measured.
  • the continued monitoring of the driver's head by the processor after the determination of the driver's inability to operate the vehicle is used to determine the driver having regained the ability to operate the vehicle.
  • the reactive component may be coupled to the processor and perform cognitive tests including requiring an oral response, visual response or gesture response, for example, to the cue.
  • the reactive component may be one which requires feedback from the driver, i.e., a detection of speech. In other words, if the driver is already looking at the road, then he can orally tell the system that he is awake and attentive.
  • Some of the applications above have been based on the idea of training a neural network or modular neural network (MNN) and then installing it in the vehicle where it does not change.
  • MNN modular neural network
  • Such a training session for each driver of the vehicle is within the scope of the invention and may be useful when a particular driver is the sole or one of the few drivers of a vehicle, such as for municipal transportation systems.
  • some of the parameters of the neural network can be made adaptive, that is they change over time as the system adapts to a particular individual.
  • the trained pattern recognition technique applied by the processor may be trained on all measures of occupant behavior that might indicate driver attentiveness or lack thereof while driving, i.e., whether the vehicle is moving or not.
  • vehicle parameters may also be analyzed such as acceleration, steering wheel angle, angular motion of the vehicle, etc. This would be especially useful if accurate maps were available so that it is known where the vehicle is relative to the lane boundaries.
  • the attempted running of stop lights and stop signs would be a clear indication of lack of attentiveness that can be used if maps or an external viewing camera were present, and indicate the presence of stop lights and stop signs, or other objects, to which the driver has not respected.
  • Probe vehicles may be used to map roads for attentiveness purposes, so that once a map is obtained and installed on the vehicle, the processor may access the map and determine whether a change in the driver's head position and/or use of the steering wheel is the result of the driver falling asleep or simply following a curvature in the road. If a map is not available, then the processor could be designed to include the motions of drivers on the same stretch of road and then measure the driving behavior of the driver compared with the other vehicle drivers on the same stretch of road to determine inattentiveness. The expected behavior of a driver on a stretch of road may even be included in the map. Map updating techniques may be used to ensure that the processor on the vehicle that is analyzing the driver's action relative to data from the map is up-to-date.
  • Another factor to be used by the processor is the amount of time when the driver is not looking at the road ahead. If this exceeds, e.g., 2 seconds, then the processor could use this factor in its determination of inattentiveness.
  • head orientation which may be determined by analysis of the images of the driver. Most of the time the driver is looking forward, so the position in which the driver's head is in most of the time may be considered the normal position and variations from this position for an extended period of time may be indicative of the driver' s inattentiveness.
  • the foregoing structure enables significant advantages to be obtained.
  • the foregoing structure and methods for using the structure provide new and improved methods and systems for monitoring a driver of a vehicle to determine whether the driver is falling asleep or otherwise unable to operate the vehicle, new uses of electric field sensors to determine the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle, to interrupt vehicle occupant gestures for the control of one or more vehicle components and/or to replace vehicle mechanical or electrical switches, connectors and/or wires, new monitors that monitor the health state of one or more vehicle occupants based on measurements of their heartbeat and respiration rates, and new uses of optical sensors to determine the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle.
  • the invention provides for new uses of weight sensors, such as bladders, strain gages, load sensors and/or displacement sensors, to determine the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle, new uses of radar-based sensors or accelerometers to determine the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle, and new analyzers that analyze the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle to determine the state of drowsiness or health state of the driver or other occupant of a vehicle.
  • weight sensors such as bladders, strain gages, load sensors and/or displacement sensors
  • radar-based sensors or accelerometers to determine the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle
  • new analyzers that analyze the heartbeat and/or respiration rates of the driver and/or other occupants of a vehicle to determine the state of drowsiness or health state of the driver or other occupant of
  • a vehicle including a system for monitoring a driver of the vehicle in accordance with the invention includes a frame, an electric field occupant sensing and monitoring system arranged to monitor periodic changes in an electric field caused by a driver' s heartbeat and/or respiration as well as possibly motion of the driver and parts of the driver.
  • Signal processing algorithms are provided to separate the signals from the heartbeat, respiration and occupant motion to be analyzed separately.
  • the vehicle can have, in addition to or as a substitution for, the electric field monitoring system, an optical imaging system arranged on the frame to obtain images of the driver, and a processor coupled to the optical imaging system and arranged to analyze the images obtained by the optical imaging system to locate a head of the driver in the images and monitor the driver's head or a part thereof over time, e.g., for movement of the driver's head or part thereof.
  • the imaging system is particularly arranged to monitor the face of the driver to obtain information related to the heart rate, respiration rate, temperature, alcohol content, glucose level, oxygenation and/or blood pressure of the driver.
  • the processor also determines, based on the monitoring of the driver' s head or face or a part thereof, whether the driver has lost the ability to operate the vehicle.
  • the loss of ability to operate the vehicle can arise from the driver falling asleep or otherwise being incapacitated due to the effects of alcohol or drugs or of a medical condition such as a diabetic event, heart attack, stroke or seizure.
  • a reactive component on the vehicle is affected by the determination by the processor that the driver has lost or is in danger of losing the ability to operate the vehicle, and can require a responsive action by the driver to indicate regaining the ability to operate the vehicle or, in lieu of an appropriate responsive action, can exert control over the vehicle to slow the vehicle and bring it to a stop.
  • the processor may use a trained pattern recognition technique when monitoring the driver' s head, face or part thereof, e.g., when locating the head or face of the driver or part of the driver's head or face in the obtained images.
  • the optical imaging system may include a plurality of optical imagers spaced apart from one another and each positioned to obtain images of a face of the driver.
  • the processor monitors the driver' s head in one embodiment by determining a position of the head of the driver in images obtained at different times and analyzing the determined position of the head of the driver in the different images.
  • the processor monitors the driver's eyes by determining a position of the eyes of the driver in images obtained at different times and analyzing the determined position of the eyes of the driver in the different images.
  • the processor monitors the driver's eye lids by determining a position of the eye lids of the driver in images obtained at different times and analyzing the determined position of the driver's eye lids in the different images.
  • the processor monitors the face of the driver or a part thereof and determines one or more of the heartbeat, respiration rate, blood pressure, and temperature.
  • the reactive component may be a warning light, an alarm light, a warning sound generator, a tactile sensation generator, and a voice generating device. More generally, the reactive component may be warning system or alarm system, which provides one or more audible, tactile, visual warnings.
  • the reactive component may be a communication transmission to a remote facility informing authorities that the vehicle has been stopped due to an incapacitated driver. Each reactive component could be controlled differently depending on the monitoring of the driver's head, face or part thereof or of the heartbeat, respiration rate, blood pressure, glucose level, alcohol level, drug presence, temperature or motion.
  • One reactive component may be a button which, when pressed, indicates that the driver has regained the ability to operate the vehicle.
  • Another may be a microphone and associated voice recognition software.
  • the processor may also operate the horn and lights of the vehicle, or other externally apparent warning or alarm system, after determining that the driver is falling asleep or otherwise incapacitated to warn other vehicles of the inability of the driver to operate the vehicle.
  • the driver may also be required to react by providing an audio signal such as by enunciating one or more spoken words.
  • Also disclosed above is a method for monitoring operation of a vehicle by a driver that includes obtaining images of the driver from at least one vehicle-mounted optical imager, analyzing the obtained images to locate a head of the driver in the images, monitoring the driver's head or a part thereof over time, determining, based on the monitoring of the driver's head or part thereof, whether the driver has lost the ability to operate the vehicle, the loss of ability to operate the vehicle arising from the driver falling asleep or otherwise being incapacitated, and upon a determination that the driver has lost the ability to operate the vehicle, affecting a reactive component which requires action by the driver to indicate regaining of the ability to operate the vehicle or exerts control over the vehicle to slow the vehicle and bring it to a stop.
  • Monitoring the driver' s head or a part thereof over time may entail training a pattern recognition technique to locate the head of the driver or other part of the driver.
  • Monitoring the driver's head or a part thereof over time may entail determining a position of the head of the driver in images obtained at different times and analyzing the determined position of the head of the driver in the different images.
  • this monitoring may include determining a position of the eyes of the driver in images obtained at different times and analyzing the determined position of the eyes of the driver in the different images.
  • this monitoring may include determining a position of the eye lids of the driver in images obtained at different times and analyzing the determined position of the driver's eye lids in the different images.
  • this monitoring may include selecting an appropriate skin portion, illuminating that portion, perhaps with non-visible electromagnetic radiation, and tracking that portion in subsequent images in order to determine the variation in one or more properties of the skin portion in order to determine at least one of the temperature, glucose level, alcohol level, drug presence, heartbeat, respiration rate and blood pressure as expressed in the skin patch or the capillaries thereof.
  • one or more biological properties of the driver is/are obtained by filtering the signals from one or more electric field sensors and monitoring one or more frequency ranges indicative of one or more of the heartbeat rate, respiration rate and bodily motion of the driver in order to determine that the driver may be: falling asleep; experiencing a heart attack, stroke or seizure; or is under the influence of alcohol and/or drugs.
  • Other systems and methods which are capable of monitoring the heart rate, respiration rate, blood temperature or other vital signs of the driver include fluid-filled bladders, pressure mats, Plessey ECG sensors, radar -based and other occupant sensors.
  • the driver can be alerted and the system may then require a timely response from the driver before any action is taken by a vehicle component.
  • a primary method disclosed above involves expanding in what is considered to be an unobvious manner, use of electric field (EF) sensors such as being marketed by Nissan Elesys for detection and classification of passenger occupants in a vehicle for the purpose of airbag suppression.
  • EF electric field
  • other non-obvious uses have been disclosed, such as for communication to a vehicle system using gestures and the elimination of wires associated with vehicle switches.
  • Methods described herein for using the heartbeat and respiration rates for determining drowsiness can be combined with each other as well as with other methods, disclosed herein or elsewhere, or otherwise known or applied in the field to which this invention pertains, such as optical eye closure monitoring methods, in order to improve the forecasting accuracy and reduce false alarms.
  • These other measures can include a physiological diagnostic based on the eyelid motion observation and a behavioral diagnostic based on the observation of the vehicle lateral control behavior. They can be combined with additional in-vehicle and contextual information to provide a final decision about the driver's vigilance state, as reported in the literature.
  • a driver should be considered to include an operator or a vehicle, a conductor of a vehicle, a pilot of a vehicle, etc.

Abstract

L'invention concerne un véhicule comprenant un siège sur lequel un occupant (2) est assis lors de l'utilisation du véhicule et concerne un système de surveillance servant à surveiller l'occupant assis sur le siège. Le système de surveillance comprend des ensembles d'antennes de champ électrique (4, 6), chacun comprenant au moins une antenne, une unité de commande (9) connectée aux ensembles d'antennes (4, 6) et comprenant des sélecteurs (23, 24) couplés aux antennes. Les sélecteurs (23, 24) sont commandés par l'unité de commande (9) pour obtenir des signaux en provenance d'une ou de plusieurs antennes servant d'antennes de réception et en provenance d'une ou de plusieurs antennes servant d'antennes d'émission. L'unité de commande (9) détermine quelle combinaison d'antenne(s) d'émission et d'antenne(s) de réception fournit un signal le plus fort dans une plage de rythme cardiaque prévu et/ou une plage de respiration prévue de l'occupant (2) et surveille ensuite cette combinaison pour des modifications et/ou des écarts par rapport à une plage normale du rythme cardiaque et/ou de la respiration.
PCT/US2014/037814 2014-05-13 2014-05-13 Système et procédé de surveillance de santé et de fatigue de conducteur WO2015174963A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2014/037814 WO2015174963A1 (fr) 2014-05-13 2014-05-13 Système et procédé de surveillance de santé et de fatigue de conducteur

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2014/037814 WO2015174963A1 (fr) 2014-05-13 2014-05-13 Système et procédé de surveillance de santé et de fatigue de conducteur

Publications (1)

Publication Number Publication Date
WO2015174963A1 true WO2015174963A1 (fr) 2015-11-19

Family

ID=54480341

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/037814 WO2015174963A1 (fr) 2014-05-13 2014-05-13 Système et procédé de surveillance de santé et de fatigue de conducteur

Country Status (1)

Country Link
WO (1) WO2015174963A1 (fr)

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106875632A (zh) * 2017-03-10 2017-06-20 湖北文理学院 防止疲劳驾驶的监控系统及方法
DE102016200048A1 (de) * 2016-01-06 2017-07-06 Volkswagen Aktiengesellschaft Warneinrichtung und Verfahren zum Warnen von Kraftfahrzeuginsassen
WO2017140122A1 (fr) * 2016-02-18 2017-08-24 广东百事泰电子商务股份有限公司 Enregistreur de surveillance intelligent
WO2017177221A1 (fr) * 2016-04-08 2017-10-12 Magna Seating Inc. Détection de variabilité de fréquence cardiaque et de somnolence
US9925872B1 (en) 2016-11-14 2018-03-27 Denso International America, Inc. System for monitoring driver alertness and adapting vehicle settings thereto
DE102017201405A1 (de) 2017-01-30 2018-08-02 Audi Ag Verfahren zum Betreiben eines Kraftfahrzeugs
EP3438948A4 (fr) * 2016-05-11 2019-05-01 Sony Corporation Appareil de traitement d'image, procédé de traitement d'image et corps mobile
TWI663581B (zh) * 2018-06-04 2019-06-21 亞東技術學院 車輛駕駛員防瞌睡整合系統與車輛駕駛員防瞌睡方法
US10328946B2 (en) 2017-01-03 2019-06-25 Ge Global Sourcing Llc Alerter augmentation system
US10370012B2 (en) 2017-03-09 2019-08-06 Ge Global Sourcing Llc Adaptive vehicle control system
CN110301921A (zh) * 2018-03-27 2019-10-08 香港生产力促进局 用于进行困倦检测和干预的系统和方法
IT201900000133A1 (it) * 2019-01-07 2020-07-07 St Microelectronics Srl "Procedimento di elaborazione di immagini, sistema, veicolo e prodotto informatico corrispondenti"
CN112137601A (zh) * 2020-09-23 2020-12-29 中国第一汽车股份有限公司 信号处理方法、装置、车辆及存储介质
JP2021010657A (ja) * 2019-07-08 2021-02-04 株式会社Ｓｏｋｅｎ 人のｒｒｉを算出するための測定システム
WO2021095067A1 (fr) * 2019-11-11 2021-05-20 Phoenix Service S.R.L.S. Dispositif de sécurité dans un véhicule configuré pour surveiller la santé cardiaque des conducteurs de véhicule
CN113056228A (zh) * 2018-10-18 2021-06-29 深度科学有限责任公司 使用多模态传感器检测生理信息的系统和方法
US11065958B2 (en) 2017-01-03 2021-07-20 Transportation Ip Holdings, Llc Control system and method
CN113370787A (zh) * 2021-07-02 2021-09-10 江苏芯灵智能科技有限公司 基于指静脉特征判断的酒驾限制系统
CN113425299A (zh) * 2020-03-23 2021-09-24 安波福技术有限公司 包括预测性睡眠风险因素的驾驶员警觉性监测
WO2022236204A1 (fr) * 2021-05-03 2022-11-10 Qualcomm Incorporated Système à capteurs multiples pour suivi cardiovasculaire et respiratoire
US11498518B2 (en) * 2018-11-29 2022-11-15 Littelfuse, Inc. Radar-based occupancy detector for automobiles

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090261979A1 (en) * 1992-05-05 2009-10-22 Breed David S Driver Fatigue Monitoring System and Method
RU104438U1 (ru) * 2010-11-01 2011-05-20 Общество с ограниченной ответственностью "ТОЧНАЯ ЭЛЕКТРОНИКА" Устройство для контроля состояния сердечно-сосудистой системы водителя
US20110224875A1 (en) * 2010-03-10 2011-09-15 Cuddihy Mark A Biometric Application of a Polymer-based Pressure Sensor
US20120259181A1 (en) * 2009-10-14 2012-10-11 Delta Tooling Co., Ltd. Biological state estimation device, biological state estimation system, and computer program
US20140097957A1 (en) * 1995-06-07 2014-04-10 American Vehicular Sciences Llc Driver fatigue monitoring system and method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090261979A1 (en) * 1992-05-05 2009-10-22 Breed David S Driver Fatigue Monitoring System and Method
US20140097957A1 (en) * 1995-06-07 2014-04-10 American Vehicular Sciences Llc Driver fatigue monitoring system and method
US20120259181A1 (en) * 2009-10-14 2012-10-11 Delta Tooling Co., Ltd. Biological state estimation device, biological state estimation system, and computer program
US20110224875A1 (en) * 2010-03-10 2011-09-15 Cuddihy Mark A Biometric Application of a Polymer-based Pressure Sensor
RU104438U1 (ru) * 2010-11-01 2011-05-20 Общество с ограниченной ответственностью "ТОЧНАЯ ЭЛЕКТРОНИКА" Устройство для контроля состояния сердечно-сосудистой системы водителя

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KRYZHANOVSKIY E.V. ET AL.: "Novaya mnogokanalnaya sistema blizhney infrakrasnoy spektroskopii dlya funktsionalnogo issledovaniya golovnogo mozga.", BIOTEKHNICHESKIE SISTEMY I TEKHNOLOGII, pages 44 - 46 *

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102016200048A1 (de) * 2016-01-06 2017-07-06 Volkswagen Aktiengesellschaft Warneinrichtung und Verfahren zum Warnen von Kraftfahrzeuginsassen
DE102016200048B4 (de) * 2016-01-06 2020-08-13 Volkswagen Aktiengesellschaft Warneinrichtung und Verfahren zum Warnen von Kraftfahrzeuginsassen
WO2017140122A1 (fr) * 2016-02-18 2017-08-24 广东百事泰电子商务股份有限公司 Enregistreur de surveillance intelligent
WO2017177221A1 (fr) * 2016-04-08 2017-10-12 Magna Seating Inc. Détection de variabilité de fréquence cardiaque et de somnolence
EP3438948A4 (fr) * 2016-05-11 2019-05-01 Sony Corporation Appareil de traitement d'image, procédé de traitement d'image et corps mobile
US9925872B1 (en) 2016-11-14 2018-03-27 Denso International America, Inc. System for monitoring driver alertness and adapting vehicle settings thereto
US10328946B2 (en) 2017-01-03 2019-06-25 Ge Global Sourcing Llc Alerter augmentation system
US11065958B2 (en) 2017-01-03 2021-07-20 Transportation Ip Holdings, Llc Control system and method
US10845802B2 (en) 2017-01-30 2020-11-24 Audi Ag Method for operating a motor vehicle
DE102017201405B4 (de) * 2017-01-30 2018-12-13 Audi Ag Verfahren zum Betreiben eines Kraftfahrzeugs
DE102017201405A1 (de) 2017-01-30 2018-08-02 Audi Ag Verfahren zum Betreiben eines Kraftfahrzeugs
US10370012B2 (en) 2017-03-09 2019-08-06 Ge Global Sourcing Llc Adaptive vehicle control system
CN106875632A (zh) * 2017-03-10 2017-06-20 湖北文理学院 防止疲劳驾驶的监控系统及方法
CN110301921A (zh) * 2018-03-27 2019-10-08 香港生产力促进局 用于进行困倦检测和干预的系统和方法
TWI663581B (zh) * 2018-06-04 2019-06-21 亞東技術學院 車輛駕駛員防瞌睡整合系統與車輛駕駛員防瞌睡方法
CN113056228A (zh) * 2018-10-18 2021-06-29 深度科学有限责任公司 使用多模态传感器检测生理信息的系统和方法
US11498518B2 (en) * 2018-11-29 2022-11-15 Littelfuse, Inc. Radar-based occupancy detector for automobiles
US11471084B2 (en) 2019-01-07 2022-10-18 Stmicroelectronics S.R.L. Image processing method and corresponding system
US11793406B2 (en) 2019-01-07 2023-10-24 Stmicroelectronics S.R.L. Image processing method and corresponding system
IT201900000133A1 (it) * 2019-01-07 2020-07-07 St Microelectronics Srl "Procedimento di elaborazione di immagini, sistema, veicolo e prodotto informatico corrispondenti"
JP2021010657A (ja) * 2019-07-08 2021-02-04 株式会社Ｓｏｋｅｎ 人のｒｒｉを算出するための測定システム
JP7226152B2 (ja) 2019-07-08 2023-02-21 株式会社Soken 人のｒｒｉを算出するための測定システム
WO2021095067A1 (fr) * 2019-11-11 2021-05-20 Phoenix Service S.R.L.S. Dispositif de sécurité dans un véhicule configuré pour surveiller la santé cardiaque des conducteurs de véhicule
CN113425299A (zh) * 2020-03-23 2021-09-24 安波福技术有限公司 包括预测性睡眠风险因素的驾驶员警觉性监测
CN112137601A (zh) * 2020-09-23 2020-12-29 中国第一汽车股份有限公司 信号处理方法、装置、车辆及存储介质
CN112137601B (zh) * 2020-09-23 2024-03-15 中国第一汽车股份有限公司 信号处理方法、装置、车辆及存储介质
WO2022236204A1 (fr) * 2021-05-03 2022-11-10 Qualcomm Incorporated Système à capteurs multiples pour suivi cardiovasculaire et respiratoire
US11844589B2 (en) 2021-05-03 2023-12-19 Qualcomm Incorporated Multi-sensor system for cardiovascular and respiratory tracking
CN113370787A (zh) * 2021-07-02 2021-09-10 江苏芯灵智能科技有限公司 基于指静脉特征判断的酒驾限制系统

Similar Documents

Publication Publication Date Title
US8725311B1 (en) Driver health and fatigue monitoring system and method
US20140276090A1 (en) Driver health and fatigue monitoring system and method using optics
WO2015174963A1 (fr) Système et procédé de surveillance de santé et de fatigue de conducteur
US11383721B2 (en) System and method for responding to driver state
US10867218B2 (en) Biometric sensor fusion to classify vehicle passenger state
US10210409B1 (en) Seating system with occupant stimulation and sensing
WO2015175435A1 (fr) Système et procédé de suivi de la santé et de la fatigue d'un conducteur
US10486590B2 (en) System and method for vehicle control integrating health priority alerts of vehicle occupants
EP3683623B1 (fr) Système et procédé de réponse à l'état d'un conducteur
US6575902B1 (en) Vigilance monitoring system
US20150379362A1 (en) Imaging device based occupant monitoring system supporting multiple functions
US6337629B1 (en) Method and a system for monitoring a person
CN107234968A (zh) 一种基于多模式生物感应的汽车控制系统
US11751784B2 (en) Systems and methods for detecting drowsiness in a driver of a vehicle
US11447140B2 (en) Cognitive tunneling mitigation device for driving
KR20110105411A (ko) 무선 측정 생체 신호를 이용한 안전 운전 시스템
Kumari et al. A survey on drowsy driver detection system
Bhaskar EyeAwake: A cost effective drowsy driver alert and vehicle correction system
GB2556410A (en) System for managing alertness of a driver
WO2001019249A1 (fr) Procede et systeme de surveillance d'une personne
Bhavya et al. Intel-Eye: An Innovative System for Accident Detection, Warning and Prevention Using Image Processing (A Two-Way Approach in Eye Gaze Analysis)

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14891953

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14891953

Country of ref document: EP

Kind code of ref document: A1