WO2015152536A1 - Procédé et dispositif pour détecter la somnolence au moyen d'un signal physiologique basé sur image animée - Google Patents

Procédé et dispositif pour détecter la somnolence au moyen d'un signal physiologique basé sur image animée Download PDF

Info

Publication number
WO2015152536A1
WO2015152536A1 PCT/KR2015/002558 KR2015002558W WO2015152536A1 WO 2015152536 A1 WO2015152536 A1 WO 2015152536A1 KR 2015002558 W KR2015002558 W KR 2015002558W WO 2015152536 A1 WO2015152536 A1 WO 2015152536A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
drowsiness
vibration
image
video
Prior art date
Application number
PCT/KR2015/002558
Other languages
English (en)
Korean (ko)
Inventor
최진관
황성택
Original Assignee
주식회사 바이브라시스템
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 바이브라시스템 filed Critical 주식회사 바이브라시스템
Publication of WO2015152536A1 publication Critical patent/WO2015152536A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/372—Analysis of electroencephalograms
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02—Alarms for ensuring the safety of persons
    • G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
    • A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/48—Other medical applications
    • A61B5/4806—Sleep evaluation
    • A61B5/4809—Sleep detection, i.e. determining whether a subject is asleep or not
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6887—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient mounted on external non-worn devices, e.g. non-medical devices
    • A61B5/6893—Cars
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2503/00—Evaluating a particular growth phase or type of persons or animals
    • A61B2503/20—Workers
    • A61B2503/22—Motor vehicles operators, e.g. drivers, pilots, captains
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/162—Testing reaction times
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K28/00—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions
    • B60K28/02—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver
    • B60K28/06—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver responsive to incapacity of driver

Definitions

  • the present invention relates to a method and apparatus for predicting or detecting a driver's drowsiness and taking safety measures according to the result to promote the safety of the driver and the occupant. Specifically, the present invention relates to obtaining a physiological signal using a video obtained from a living body. The present invention relates to a method of predicting or detecting a driver's drowsiness and an apparatus applying the same.
  • a technology of determining a sleepiness by detecting a driver's eye state through a camera, imaging a driver's face using a CCD camera, and determining a sleepiness by detecting a change in the eye area from the image signal thus captured Technology to take driver's face image using camera detect eye line from the face image and analyze minute movement of eye line to determine awakening state such as drowsiness and low attention, frequency of eye blink, pulse frequency of eyelid Technology to determine drowsiness based on physiological measurement parameters such as facial muscle loosening degree, eyes blink from driver's face image, eye opening degree, eyes closed time, etc. to determine drowsiness Technology to compare head orientation, exact position of eyes and nose from facial images captured by the camera
  • a technique etc. which determine a sleepiness degree by this.
  • the conventional EEG during driver's driving is memorized as reference EEG, after detecting EEG in real driving, this EEG is classified according to frequency, and the frequency of occurrence of each EEG classified in this way is calculated It is a technique of determining whether or not drowsiness by comparing with the reference EEG stored in advance.
  • One conventional patent document provides a method of installing a lamp for transmitting infrared rays toward a driver's face for night photographing, and imaging a driver's face image through a face-oriented camera to detect a drowsiness from the driver's face.
  • edge-processing the image from the acquired face image the driver's face left and right end positions and face upper and lower end positions are detected, and the amount of movement thereof is calculated to determine whether the driver is asleep.
  • the position of the driver's eyes or face or the movement state of the driver changes according to the driving state of the vehicle, so that the movement amount is compensated based on the driving state of the vehicle.
  • the present invention relates to a method and apparatus for predicting or detecting a driver's drowsiness and taking safety measures according to the result to promote the safety of the driver and the occupant. Specifically, the present invention relates to obtaining a physiological signal using a video obtained from a living body. The present invention relates to a method of predicting or detecting a driver's drowsiness and an apparatus applying the same.
  • Predicting or detecting a drowsiness of the driver from the physiological signal Predicting or detecting a drowsiness of the driver from the physiological signal.
  • the method may further include generating and visualizing a corresponding image of the physiological signal of the subject using the physiological signal.
  • An image acquisition unit photographing the driver to obtain a continuous image
  • a light detector for continuously imaging the driver's image, an A / D converter for converting the captured image into image data, and analyzing the converted continuous image data to measure a vibration parameter, and to measure the measured vibration parameter.
  • a driver drowsiness prediction or detection apparatus incorporating a camera, comprising: a processor configured to generate a biosignal image based on the display; and a display unit configured to display the generated biosignal image.
  • the present invention can be mounted on the vehicle to protect the driver and passengers from accidents caused by drowsiness.
  • FIG. 1 is a flowchart illustrating an example of a method of obtaining a biosignal to detect a drowsiness state of a driver.
  • FIG. 2A is a schematic block diagram functionally categorizing an apparatus implementing the method shown in FIG.
  • Figure 2b schematically illustrates the configuration and flow of a driver drowsiness method according to the present invention and a system applying the same.
  • FIG. 3A is a block diagram of an electronic device implementing the method illustrated in FIG. 1.
  • FIG. 3B is a block diagram showing correlations between components in the electronic device shown in FIG. 3.
  • FIG. 4 illustrates the concept of matching the EEG signal and vibration parameters in the driver drowsiness prediction method according to the present invention
  • FIG. 5 is a diagram illustrating a specific flow.
  • FIG. 6 is a flowchart of a method for obtaining a biosignal of a driver in the driver drowsiness prediction method according to the present invention.
  • FIG. 7A illustrates the emission of bioenergy (aura) around a subject (driver) human body image formed by the amplitude component of the vibration image.
  • FIG. 8A and 8B show the bio-image radiation according to the state of the subject, and FIG. 8A shows a stable state and FIG. 8B shows an unstable stress state.
  • Fig. 9A is a distribution graph of frequency components (biosignal images) of a human body vibration image in a stable state.
  • 9B is a distribution graph of the frequency components (biosignal images) of the human body vibration image under stress.
  • FIG. 10 is a flowchart illustrating an algorithm extraction method in an R & D stage in implementing the driver drowsiness prediction method according to the present invention.
  • 11 is a flowchart of a driver's drowsiness prediction method using the extracted algorithm.
  • FIG. 12 shows a change of a parameter related to sleepiness among vibration parameters in a driver drowsiness prediction system according to the present invention.
  • Figure 13 illustrates a user interface appearing on a display in a system that actually performs the drowsiness prediction method according to the present invention.
  • FIG. 14 illustrates a relationship between a camera photographing a driver and LED lighting.
  • Fig. 15 is a view for explaining an angle of view relationship with a driver when a camera is installed in a room mirror of a vehicle.
  • FIG. 16 illustrates an example of a photographing mechanism for minimizing the influence of the vibration of the camera during photographing.
  • Figure 17 shows the result of the prediction of the present invention and the comparison of the drowsy state wearing the wireless EEG headset.
  • the present invention obtains a real-time video from the driver as shown in FIG. 1 (11), processes (analyzes) the video to extract or generate vibration parameters (parameters), and uses the vibration parameters.
  • the physiological signal is extracted (13).
  • the driver's sleep state is evaluated (predicted or detected) 14 using the physiological signal.
  • Fig. 2A is a functional block diagram of the drowsy state detection device.
  • the drowsiness prediction or detection detecting device includes a camera 21 for photographing a driver, an image processor 22 for analyzing an image obtained from the camera 21, and an image processor 22.
  • the analysis unit 23 uses the signal to extract vibration parameters (parameters), generates a physiological signal, and processes and analyzes the physiological signal to evaluate an operator's drowsiness
  • the application unit 24 includes:
  • the controller may be connected to a safety guard 25 including a display for generating a character or an image for arousing a driver and displaying the same or an audio device for reproducing sound or music.
  • the image processing unit 22, the determination unit 23, and the application unit 24 may be implemented by an application program based on a central processing unit (CPU) or an application processor (AP). It can be implemented as a device or built-in to a system in an automobile.
  • the signal component corresponding to the EEG signal is extracted from the above parameters, thereby predicting or detecting the drowsiness of the driver.
  • the signal corresponding to the EEG signal is extracted from the video, which may have properties similar to the EEG signal.
  • the vibration parameter component appearing in synchronization with the EEG signal in the drowsy state is analyzed and extracted. Or the corresponding component signal of the EEG signal for determination.
  • the process of acquiring video from the driver, extracting vibration parameters, and extracting EEG-corresponding signals from the parameters will be described later in detail, and a system applying this method may be embedded in a vehicle, which is shown in FIG. 2B. It may have a structure as shown.
  • the analysis system the extraction of vibration parameters and the like, and the analysis of the drowsiness state using the same are performed, and according to the result, the sound for converting the driver to the awake state by transmitting to various systems operating in conjunction with the vehicle system, Drive a device such as vibration (C).
  • the vibration parameters are extracted from the black box by a terminal device such as a smartphone to analyze the driver's physiology, psychological state, for example, drowsiness, fatigue, emotional state, etc. Through the visual or auditory perception, the driver's physiological and emotional state can be controlled through various contents as well as prevention of drowsiness (E).
  • the camera can be mounted on the driver's front contrast board, or on the other hand in a room mirror.
  • the camera may have a structure for removing vibrations transmitted from the vehicle, which will be described later in detail.
  • the frequency components of the biosignal images obtained have the most information on the bioenergy, or psychophysiological characteristics, of the observed organisms.
  • Analysis of the obtained biosignal image may be performed by a person or mathematically by processing at least one of the obtained digital biosignal image and its components by a program. In order to prepare and analyze algorithms for mathematical processing, it is good to make a biosignal image which is convenient for visual analysis such as pseudo-color image of monitor screen.
  • the frequency component of the biosignal image to be obtained allows to continuously and clearly specify the levels of psychophysiological and emotional states of the human body and to distinguish the changes in the human state when various stimuli occur in humans. do.
  • an image showing the human body's bioenergy field represented by an aura located around the human body can be used to evaluate the psychophysiological state of the human body faster and more accurately than other methods.
  • aura refers to an integral characteristic of the psychophysiological state of the human body. These auras appear around the human body and have specific relationships with the bioenergy components of the human body. The image of the human aura provides a lot of information when studying the psychophysiological parameters of the human body, and the following factors are considered. Human emotional state can literally change every second. The average person can not stay in one emotional state for a long time.
  • the geometrical correlation of the elements of the biosignal image to the elements of the real image is less effective than the frequency components of the vibration image represented by the aura located around the real image.
  • the elements of the biosignal image are topologically related to the elements of the actual image, the elements with the maximum vibration frequency are not visible in the entire background when the image is subjected to color-frequency adjustment.
  • the biosignal image to be obtained must be visually controlled in advance.
  • the proposed image of the frequency component of the biosignal image, in the form of an aura is consistent with the physical concept of bioenergy radiation and enables visual control and analysis of the device-generated image.
  • amplitude components is more effective in topological relationships.
  • Acquiring information about the level of aggression of a creature consists of constructing a frequency distribution histogram and measuring the head vibration image parameters of the creature.
  • Aggregation of Aggression Levels consists of:
  • the biological head vibration image parameter is measured to obtain information about the stress level of the creature.
  • the stress level St is calculated by the following Equation 2.
  • Anxiety level (Tn) is measured by the following Eq.
  • the compatibility level (C) is counted as the following ⁇ Equation 4>.
  • vibration image parameters of the head of the organism are measured to obtain information about the integrated level of change in the psychophysiological state.
  • cybernetics and information theory examines the applicability of operational methods and techniques to organisms and living systems.
  • Modern concepts of cognitive biology are usually related to the concepts and definitions of signal information and transfer theory, and enable the psychophysiological information of mathematical parameters established in information theory.
  • the author's long study and observation of the study of human head micromovement with the help of statistical parameters used in information theory shows that there is a statistically reliable dependence between the state of human psychophysiology and the head micromovement information statistics parameter.
  • the present inventors can present their own interpretations of these phenomena and vestibular emotion reflections.
  • psychophysiological energy coordination metabolic energy coordination
  • All typical emotional states can be characterized by a correlation between specific energy consumption and individual physiologically necessary energy and emotional energy.
  • the physiological energy is formed to realize the physiological process, and the emotional energy is formed as a result of the conscious or unconscious process.
  • the attack state if it is the same attack condition, should be expressed differently in various people, and natural adjustment process such as age, gender and education level should be considered.
  • these differences should not have a fundamental meaning in the relative amounts and locations of energy release within the body.
  • the human head in a vertical, semi-balanced state can be seen as an overly sensitive mechanical indicator of all the energy processes in the body. From a biomechanical point of view, maintaining the vertical balance and equilibrium of the heads far above the center of gravity requires tremendous continuous effort and reduction of the neck-head bone muscles. Moreover, this movement is realized reflexively under vestibular system. All meaningful phenomena in the organs lead to changes in the ongoing physiological process. This is similar to other physiological process changes traditionally used for psychophysiological analysis, such as galvanic skin response (GSR), arterial pressure, and heart rate.
  • GSR galvanic skin response
  • the parameters of head movement vary with the amount of energy expression and the location of energy expression.
  • the spatial three-dimensional trajectory of head movements is very complicated because the shape of the head resembles a sphere.
  • the movement trajectory of each point can vary significantly in the movement of hundreds of neck muscles.
  • Statistical analysis of informative motion parameters enables reliable quantitative parameter differentiation of head movements. In other words, it is possible to measure and confirm the emotional state through the measurement of energy and vestibular response.
  • the laws of mechanics appear to be consistent, and behavior is always reactionary to maintain equality. Energy measurements in the body organs that naturally target a wide variety of people will result in consistent corresponding changes in head movement parameters through vestibular activity.
  • the overall emotional classification according to the informational / statistical parameters of the presented head movements confirms all emotional states.
  • Modern psychology mainly uses qualitative criteria in the evaluation of emotional state, which essentially makes it impossible to measure quantitatively, and the objective evaluation of human state is difficult.
  • the suggested method allows us to measure all emotional states.
  • head movement parameters are a general characteristic psychophysiological state of man.
  • the accuracy of agreement of the proposed formulas for counting emotional states according to existing assessment criteria is low compared to the emotional state assessment method through head micromovement.
  • the proposed method is characteristic in that an integrated approach is possible for all emotion measurements. All previous methods were also used to assess various emotional states. Adopting the proposed concept for measuring emotional state allows the inclusion of psychology in precision science and enables the same emotional measurement.
  • the movement speed of the head of the creature is measured as the average frequency of marker movement, determined in units of 10 seconds, which yields the maximum frequency of TV camera work.
  • the vibration image simultaneously represents the spatial and temporal distribution of the target motion energy
  • the number of factors having the same vibration frequency for a specific time is aggregated to obtain a frequency histogram. Histograms therefore exclude information about the spatial distribution of vibration frequencies. This apparent loss of spatial information actually increases the motion information, because in terms of physiological energy, it is not very important in which part of the head the movement is performed unlike the fine movement of the face.
  • the configuration of the frequency histogram is determined according to the following.
  • the next step is to statistically identify meaningful vibration image information parameters that determine vibration image acquisition and subsequent aggression levels. This determines, among other things, vibration symmetry parameters for amplitude and frequency vibration images.
  • the formula presented allows us to measure the stress level (St) from 0 to 1, and above all, the minimum stress level corresponds to the minimum measurement, In people with stress levels close to one.
  • the following is a statistical analysis of meaningful vibration image information parameters that determine vibration image acquisition and subsequent anxiety levels. This relates, among other things, to the fast activity signal frequency spectrum construction of amplitude and frequency vibration images.
  • the presented formula allows us to measure the level of anxiety from 0 to 1.
  • the minimum level of anxiety meets the minimum measure, and those with high levels of anxiety have stress levels close to one.
  • the fast signal frequency spread spectrum of the vibration image appears for the operator's or system operator's control.
  • Another example is to find statistically meaningful informative parameters of the vibration image that determine the vibration image acquisition and then the level of compatibility between the people. Best of all, this consists of a vibration image histogram configuration at each individual frequency.
  • the proposed formula measures the level of compatibility from 0 to 1.
  • the minimum measure corresponds to the minimum compatibility (compatibility), and the high level of compatibility measure on both sides appears close to one.
  • Pc-Vibration image parameter changes when determining false level
  • n-number of measurement parameters may vary from the number of visual parameters
  • the formula presented allows us to measure false levels from 0 to 1.
  • the minimum level of false matches the minimum measure, while the highest level of false has a value close to one.
  • the present invention is utilized solely for the measurement of emotional and psychophysiological states of humans presented above.
  • the present invention allows us to describe all human conditions through the head micromovement parameters and / or the head vibration image parameters.
  • psychology it is an unclear principle to translate the traditional concept of motion into reflex micromovement of the human head using reliable statistical parameters.
  • the head fine motion frequency distribution histogram is constructed as the basis for the informational entropy calculation.
  • the informational entropy (H) calculation is based on the following formula.
  • thermodynamic entropy (S) calculation follows the following formula.
  • thermodynamic entropy is a state of anxiety in humans. It was found that there is a big connection with.
  • human energy (E) was able to be based on the difference between the mean square error and the frequency peak, based on a frequency histogram representing the highest frequency of the vibration image recording.
  • the camera 21 installed in a dashboard or a room mirror in an automobile is used to digitize an imaging device such as a CCD or CMOS and an analog signal therefrom.
  • the image processor 22 includes an encoder for generating a video of a specific format.
  • the signal analyzer 23 measures the vibration parameter by the method as described above using the image, and generates or extracts psychophysiological information (signal or physiological signal) therefrom.
  • the vibration parameters include vibration frequency, amplitude, and phase according to the change of position of each part of the subject.
  • the psychophysiological information includes psychological / emotional / emotional states such as a stable state, an excited state, and a stress state, which predict or detect drowsiness of the driver.
  • the physiological signal application unit 24 may include a physiological signal processing algorithm for evaluating the mental and emotional state of the subject using the physiological signal and a display for displaying the result.
  • the physiological signal processing algorithm may classify the subject's states into nine emotional states according to, for example, James Russell's two-dimensional emotional model.
  • the display provided in the application unit 24 displays the final result in the form of a text or an image as described above.
  • the application unit 24 may include various safety measures systems for transitioning from the drowsy state of the driver to the awake state. This may include a horn-like sound, or a music or video device that can wake you up. In addition, it may include a vibration device that can be installed in the driver's seat or the like to physically stimulate the driver.
  • Such a device may be based on various types of systems, not only general OS for PC but also portable OS such as Windows mobile, Android, iOS, Symbian, BlackBerry, Bada.
  • FIG. 3 shows a driver drowsiness prediction or detection device 30 that forms part of a system implementing the device or system shown in FIGS. 2A and 2B.
  • the driver's drowsiness prediction or detection device 30 includes an A / D converter that digitizes an analog image signal of an image photographing unit having an imaging device, that is, the camera 31 and the camera 31. 32), a processor 34 for performing image analysis and parameter extraction and physiological signal generation, drowsiness prediction or detection1, and a display unit 35 for displaying the result.
  • the apparatus includes an input device for inputting information from the outside, for example, a key input unit 36 such as a keypad and the like, and a storage unit 33 including a memory used in the above-described image signal processing.
  • EEG equipment 40 measures brain waves 41, and measures and analyzes brain waves in a drowsy state through measured brain waves (42) to identify brain waves in a drowsy state (43).
  • the vibration parameters are extracted from the moving image 45 obtained through the imaging device 44 (46), and the vibration parameters synchronized with the brain waves in the drowsy state are matched (47).
  • the parameters correlated to the sleepiness obtained through the matching are applied to the sleepiness prediction and detection program (48).
  • the delta component of C4 / F4 the alpha component of F3, and theta component of F3 are related to drowsiness and match three vibration image parameters ( F3, P17, P8F) were extracted.
  • FIG. 6 is a flowchart illustrating a process of acquiring a biological signal through a camera based on a video according to the present invention.
  • an image of a driver is acquired by a camera 31 and converted into an analog electric signal (S10 and S20).
  • the electrical signal obtained from the driver's image is an analog signal and is therefore converted into digital image data by the A / D converter 32 (S30).
  • the processor 34 calculates a vibration parameter by analyzing the change over time of each image data (S40).
  • the vibration parameter includes at least one of a vibration frequency, an amplitude, and a phase according to a change in position of each part of the subject. That is, the processor 34 analyzes the position change of each part of the driver to calculate the vibration frequency of each part, the magnitude of the position change (the magnitude of the vibration), the phase, and the like. Subsequently, the processor 34 may analyze the difference between the images using a vibration image analysis program, measure a position change with respect to the center of gravity, or calculate (calculate) a vibration parameter (parameter) using a Fourier transform.
  • the vibration parameter calculation is described in more detail as follows.
  • the processor 34 grasps the movement or vibration of the contour according to the driver's movement from a plurality of consecutive images and separates the contour into two equal parts (left and right). Then, determine the point indicated by the maximum vibration frequency in two parts of the row divided in half. This frequency determines the color of the corresponding horizontal row of the biosignal image.
  • the average amplitude of the positional variation in each of the two sections of the row divided in half located in the separate contour section determines the size (length) of the biosignal image.
  • Vibration images obtained at each point have certain positive and static characteristics, but integrated biosignal images are associated with psychophysiological parameters of the human body. This is the case in which the portable device with the camera is fixed to a non-moving support and is not affected by vibration from the outside.
  • Such filtering may include some or all of the various noise components that act as noise as well as vibration of the auto.
  • the processor 34 generates a biosignal image based on the calculated vibration parameter (S50).
  • the biosignal image may include an amplitude component and a frequency component.
  • the amplitude component is referred to as “internal biosignal image” and the frequency component is referred to as “external biosignal image”.
  • the concept of this term definition will be understood in the description of FIG. 5 below.
  • the processor 34 obtains psychophysiological information of the subject 1 from the calculated vibration parameter (S60). That is, the processor 34 may know the psychological state of the object 20 by analyzing the vibration parameter, and in particular, may predict or detect the drowsiness of the driver by the algorithm as described above.
  • FIG. 7A illustrates the emission of an aura of bioenergetic energy around an image of a human body formed from the amplitude component of a vibrating image.
  • the internal biosignal image expresses the magnitude of the change in position of each part in color. Through this it is possible to visualize the magnitude of the position change of each part of the subject (1).
  • the external biosignal image appears around the internal biosignal image and modulates the average peak vibration frequency into color.
  • FIG. 7B illustrates that a biosignal image, which is bioenergy, is radiated around an actual image of the human body.
  • the internal biosignal image is not represented and only the biosignal image is displayed around the actual image.
  • FIG. 8A and 8B show biosignal images in a stable state and an unstable state, respectively.
  • FIG. 8A shows a biosignal image of a subject in a stable or finished state and FIG. 8B in a stress state.
  • the biosignal image is sufficiently symmetrical in shape and color, and the color of the biosignal image is about halfway between the selected color scale (overall color-green).
  • the bio signal image shows that the driver is in a stable state.
  • the aura contains a lot of red components in the biosignal image.
  • the driver in this state is in an unstable state.
  • the subject becomes stressed or aggressive and the color of the biosignal image changes to a reddish color.
  • FIG. 9A is a distribution graph of frequency components (biological signal images) of a human body vibration image in a stable state
  • FIG. 9B is a distribution graph of frequency components (biological signal images) of a human body vibration image in a stress state.
  • the graph shown in FIG. 9A shows a typical frequency distribution of a person in normal working condition.
  • the results show that the majority of people in a calm state generally have a distribution of distributions similar to a single-mode distribution rule.
  • the state of the subject changes as shown in FIG. 8B.
  • the mean (medium) value of the frequency distribution (M) shifts towards increasing.
  • the mean (middle) value of the frequency distribution value (M) is shifted toward decreasing.
  • the frequency axis (X) can be expressed not only in relative units, but also in real units or time (Hz or sec.). The distance between the display values is determined by the actual parameters of the camera's rapid processing and the software's settings (time to accumulate images and the number of images in the processing sequence).
  • FIG. 10 is an operation flowchart of a drowsiness prediction program according to the present invention, which is an experimental flowchart of a process of extracting a drowsiness prediction algorithm by matching EEG and vibration parameters in a Research & Development (R & D) step.
  • the experiment is started, and first, the EEG sensor is worn by the driver or the subject to be measured (S10a), and the camera for observing the subject's head or the upper body including the same is prepared (S10b). In this state, while inducing drowsiness to the driver, while recording a video (S10d) and at the same time measures the EEG (S10c). From the video, the vibration parameter is extracted by the method as described above, and the driver's drowsiness is measured (S10f). Then, by comparing and analyzing the EEG component and vibration parameters (S10f), and extracts the vibration parameters related to sleepiness (S10h). When the vibration parameter is extracted, an algorithm for predicting drowsiness is prepared or extracted using the extracted parameter (S10i). The drowsiness prediction algorithm created or extracted here is applied to the actual drowsiness prediction apparatus.
  • FIG. 11 illustrates a process of predicting a driver's drowsiness using the above-described drowsiness prediction algorithm and transmitting and stimulating a stimulus when the drowsiness is predicted.
  • the sleepiness prediction algorithm is driven in a state in which a system according to the present invention is mounted on a real vehicle (S11a) (S11b).
  • the driver's drowsiness is predicted in real time based on the vibration parameter extracted from the moving image obtained from the driver using this drowsiness prediction algorithm (S11c).
  • a mild attention state S11d
  • a warning state S11e
  • S11f a threshold value
  • the infinite loop Circulation is carried out, and if it is above a threshold, an appropriate stimulus is generated (S11g).
  • Appropriate stimulation is to allow the driver to escape from the drowsiness, and may include anything that can stimulate at least one of the five senses of the human being, such as vibration, sound, and aroma.
  • an arbitrary number 20 represents the level of the variable, where 20 is a threshold indicating the start of sleepiness.
  • 20 is a threshold indicating the start of sleepiness.
  • the driver initially maintained the awakening state of 20 or more, and after a certain time, the drowsiness parameter began to fall below 20, that is, the drowsiness precursor appeared, and about 14 seconds had elapsed. At the time of complete drowsiness.
  • Equation 14 mathematically expresses an algorithm for predicting sleepiness.
  • Drowsiness variable 1 is a value between 0 and 1, representing a numerical value according to the sum of the values of Ii (intensity for pixels). As the drowsiness increases, the number decreases
  • the sleepiness variable 1 (SDP1) threshold for predicting sleepiness uses 0.2.
  • the change rate constant K for drowsiness variable 2 is a change rate constant obtained by dividing the standard deviation of the drowsiness variable 2 by the mean value of the drowsiness variable 2 (Mean of SDP 2).
  • Figure 13 illustrates a user interface appearing on a display in a system that actually performs the drowsiness prediction method according to the present invention.
  • (a) is a menu screen for determining performance of the sleepiness prediction method
  • (b) is a sleepiness prediction initialization screen
  • (c) shows a photographing screen of the driver in a state where the sleepiness prediction method is performed. As shown in (c), masking is applied around the periphery so that only the driver part appears in the video. In this state, when the driver's drowsiness is predicted, a warning sound can be heard along with the screen as shown in (d).
  • FIG. 14 illustrates a relationship between a camera photographing a driver and LED lighting.
  • the LED light source irradiates light to the driver's face, and the camera receives the light reflected from the driver's face.
  • the vibration or movement of the driver may perform frequency filtering with a synchronization structure based on a difference between perspective and reflection of the LED light.
  • Fig. 15 is a view for explaining an angle of view relationship with a driver when a camera is installed in a room mirror of a vehicle.
  • the position of the camera is optimal for the room mirror, and therefore, it is preferable to mount the camera and the lighting device on the room mirror itself.
  • the camera installed in the room mirror needs to be positioned so that the driver's head-neck can be photographed.
  • the photographing angle of the camera is about 90 at the top, bottom, left and right, and the size of the image is preferably 30% or more of the image frame.
  • FIG. 16 illustrates an example of a photographing mechanism for minimizing the influence of the vibration of the camera during photographing.
  • the LED light source emits light to the driver's head-face portion in the form of a pulse, and the camera captures an image only at the pulse time in synchronization with the pulse.
  • the vibration of the driver's head is analogous and continuous, but the vibration of the body is monotonous. Since the vibration of the human head is less than 10 Hz, it is possible to apply multi-sided filtering with the pulse of the LED light source being about 1 micro second. On the other hand, the vibration still present in the image can be attenuated by applying a software filter.
  • the vibration of the camera can be extremely suppressed by applying an optical image stabilizer (OIS) structure that has a built-in gyro sensor.
  • OIS optical image stabilizer
  • Figure 17 shows the result of the prediction of the present invention and the comparison of the drowsy state wearing the wireless EEG headset. As shown, it can be seen that a collision (drowsiness occurrence) occurred after about 12 from the time point when the drowsiness prediction value was the highest (drowsiness precursor time point). According to the experiment, the eyelid closure time was predicted as a drowsiness precursor about 11.6 seconds before the occurrence of drowsiness according to the present invention, and the drowsiness precursor was confirmed 11-12 seconds before the drowsiness prediction result using the wireless brain wave on the US ABM.
  • a camera mounted in an interior of a vehicle may store information obtained by a program for photographing a driver and predicting and detecting a drowsiness state with an IVN (In-Vehicle Network) of the vehicle. It is possible to be connected.
  • IVN In-Vehicle Network
  • various arousal systems may be applied according to the predicted drowsiness state, and for example, sound, image, or vibration may be applied.
  • the software contained in the main body detects the driver's drowsiness and enters the vehicle driving information from the IVN through a program that executes an algorithm according to the present invention with the real-time image information of the driver obtained from the camera in the room mirror while driving the vehicle.
  • it can be interlocked to take measures to prevent safety accidents caused by drowsy driving by generating alarm sound, fastening seat belt and steering wheel vibration, which causes awakening state according to vehicle driving state.
  • the present invention is a technology that has not existed until now, since it has been proved to have a very high reliability since a drowsiness prediction detection program was developed through mapping of EEG and video of the driver in real time and mapping with psychophysiological response parameters. . It is a supplementary service that detects drowsiness status in real time by taking a video of the driver and detects the driver's mental function state and emotional state (driver's fatigue, stress, concentration and depression level) and provides the driver with information about it. Very scalable. The present invention provides convenience without limiting the driver because the driver can detect the drowsiness even when the driver wears sunglasses, glasses, a mask, a hat, or the like.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Psychiatry (AREA)
  • Psychology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Physiology (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Radiology & Medical Imaging (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Anesthesiology (AREA)
  • Child & Adolescent Psychology (AREA)
  • Developmental Disabilities (AREA)
  • Educational Technology (AREA)
  • Hospice & Palliative Care (AREA)
  • Social Psychology (AREA)
  • Business, Economics & Management (AREA)
  • Emergency Management (AREA)
  • General Physics & Mathematics (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

Cette invention concerne un procédé et un dispositif de mesure fiable et précise d'un paramètre psychophysiologique sur un sujet. Ledit procédé de mesure comprend les étapes consistant à : acquérir une image animée d'un sujet d'essai psychophysiologique au moyen d'une image animée ; mesurer un paramètre de vibration du sujet d'essai à partir de l'image animée ; générer une image de signal biologique sur la base du paramètre de vibration ; et générer un paramètre de réaction psychophysiologique du sujet d'essai par traitement de l'image de signal biologique.
PCT/KR2015/002558 2014-03-31 2015-03-17 Procédé et dispositif pour détecter la somnolence au moyen d'un signal physiologique basé sur image animée WO2015152536A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2014-0037957 2014-03-31
KR1020140037957A KR101536348B1 (ko) 2014-03-31 2014-03-31 동영상 기반 생리 신호를 이용한 졸음 검출 방법 및 장치

Publications (1)

Publication Number Publication Date
WO2015152536A1 true WO2015152536A1 (fr) 2015-10-08

Family

ID=53793192

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2015/002558 WO2015152536A1 (fr) 2014-03-31 2015-03-17 Procédé et dispositif pour détecter la somnolence au moyen d'un signal physiologique basé sur image animée

Country Status (2)

Country Link
KR (1) KR101536348B1 (fr)
WO (1) WO2015152536A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109902574A (zh) * 2019-01-24 2019-06-18 北京元和新通科技有限公司 人体表象变化测定人体情绪的高危人员检测装置及方法
WO2020044249A1 (fr) * 2018-08-28 2020-03-05 (주)마인드아이 Procédé et dispositif d'évaluation de la fatigue cérébrale au moyen d'un système d'imagerie sans contact

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106618608A (zh) * 2016-09-29 2017-05-10 金湘范 基于视频精神生理参数监测危险人物装置及方法
KR101862696B1 (ko) * 2016-10-24 2018-06-01 (주)락싸 실사와 컴퓨터 그래픽스 영상을 이용한 생체 정보 표시 시스템 및 그의 표시 방법
US10150351B2 (en) 2017-02-08 2018-12-11 Lp-Research Inc. Machine learning for olfactory mood alteration
WO2018147838A1 (fr) * 2017-02-08 2018-08-16 Lin Zhuohua Apprentissage automatique pour modification olfactive de l'humeur
JP6650118B2 (ja) * 2018-03-15 2020-02-19 オムロン株式会社 乗員監視装置
KR102608299B1 (ko) * 2019-05-02 2023-12-01 에이치엘만도 주식회사 운전자 보조 시스템 및 그 제어 방법

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20050115788A (ko) * 2004-06-05 2005-12-08 최진관 대상 정보 획득장치 및 그 방법
KR20060009062A (ko) * 2004-07-20 2006-01-27 최진관 생체측정 정보 확인장치 및 그 방법
WO2012057646A1 (fr) * 2010-10-28 2012-05-03 Общество С Ограниченной Ответственностью "Многопрофильное Предприятие "Элсис" Procédé d'obtention d'informations sur l'état psychophysiologique d'un sujet vivant

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20050115788A (ko) * 2004-06-05 2005-12-08 최진관 대상 정보 획득장치 및 그 방법
KR20060009062A (ko) * 2004-07-20 2006-01-27 최진관 생체측정 정보 확인장치 및 그 방법
WO2012057646A1 (fr) * 2010-10-28 2012-05-03 Общество С Ограниченной Ответственностью "Многопрофильное Предприятие "Элсис" Procédé d'obtention d'informations sur l'état psychophysiologique d'un sujet vivant

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HWANG, SUNG TEAC ET AL.: "Extraction of Significant Vibraimage Parameter for Evaluating Drowsiness", HCI 2013 CONFERENCE, January 2013 (2013-01-01), pages 82 - 85 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020044249A1 (fr) * 2018-08-28 2020-03-05 (주)마인드아이 Procédé et dispositif d'évaluation de la fatigue cérébrale au moyen d'un système d'imagerie sans contact
CN112449579A (zh) * 2018-08-28 2021-03-05 心灵之眼株式会社 用非触摸视频系统的评估脑疲劳的方法和装置
CN109902574A (zh) * 2019-01-24 2019-06-18 北京元和新通科技有限公司 人体表象变化测定人体情绪的高危人员检测装置及方法

Also Published As

Publication number Publication date
KR101536348B1 (ko) 2015-07-14

Similar Documents

Publication Publication Date Title
WO2015152536A1 (fr) Procédé et dispositif pour détecter la somnolence au moyen d'un signal physiologique basé sur image animée
Guo et al. Detection of driver vigilance level using EEG signals and driving contexts
KR101739058B1 (ko) 동영상 기반 생리 신호 검출을 이용한 왜곡에 대한 정신생리적 탐지 (거짓말 탐지) 방법 및 장치
US20190239795A1 (en) Emotion estimation apparatus, method, and program
JP6590411B2 (ja) 感性評価方法
Solaz et al. Drowsiness detection based on the analysis of breathing rate obtained from real-time image recognition
JP5326521B2 (ja) 覚醒状態判断装置及び覚醒状態判断方法
KR102045569B1 (ko) 조종사 상태의 통합 감시 제어 장치 및 이를 이용한 조종사의 임무 수행 능력 유도 방법
JP6503327B2 (ja) 生理状態判定装置及び生理状態判定方法
Stemberger et al. Thermal imaging as a way to classify cognitive workload
JP2016501048A (ja) 生体パラメータを判定する方法及び装置
US10877444B1 (en) System and method for biofeedback including relevance assessment
US20190069832A1 (en) Determination result output device, determination result provision device, and determination result output system
Němcová et al. Multimodal features for detection of driver stress and fatigue
WO2020087494A1 (fr) Procédé de génération d'émotion de collecte de vidéo
KR101500888B1 (ko) 생물체의 정신생리학적 상태에 관한 정보 획득방법
Abbood et al. Prediction of driver fatigue: Approaches and open challenges
JP2021118908A (ja) ストレス判定装置、プログラム及び方法
Chang et al. Real-time physiological and facial monitoring for safe driving
Matthews et al. Monitoring task fatigue in contemporary and future vehicles: a review
KR101753834B1 (ko) 진동 이미지 기술을 이용하여 감성을 분류하는 방법
JP2016146978A (ja) 眠気検知装置
Chuang et al. Dynamically weighted ensemble-based prediction system for adaptively modeling driver reaction time
CN112888366A (zh) 脑电波解析装置、脑电波解析系统以及脑电波解析程序
KR102514577B1 (ko) 동영상을 이용한 잠재적 위험 존재 또는 이상 반응 상태 검출 방법 및 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15772489

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03/02/2017)

122 Ep: pct application non-entry in european phase

Ref document number: 15772489

Country of ref document: EP

Kind code of ref document: A1