KR20190054803A - Appartus and method for estimating condition of driver - Google Patents

Abstract

According to the present invention, provided is a method for estimating a condition of a driver, which comprises the steps of: sensing a face of a driver through a camera; sensing a color change of the sensed face in accordance with a blood pressure; estimating a heart rate corresponding to the color change; and estimating a condition of the driver based on the estimated heart rate.

Description

[0001] APPARATUS AND METHOD FOR ESTIMATING CONDITION OF DRIVER [0002]

The present invention relates to an apparatus and method for estimating a driver's condition.

Generally, the driver's surveillance apparatus uses a camera to photograph a driver who is driving, and determines the driver's driving status (whether the driver is sideview, drowsy driving, or a point of view) through the face image of the driver. Further, the driver's surveillance apparatus judges the running state of the vehicle by using a steering angle signal, a vehicle speed signal, and a driving position of the running vehicle. The driver's monitoring device uses the driver's state and running state to warn the driver that the driver is in an unsafe driving state, thereby enabling safe driving.

Japanese Patent Laid-Open Publication No. 2015-194884, published on November 5, 2015, Title: Driver's Surveillance System. International Application Number: WO2015 / 030219, Publication Date: November 19, 2015 Title: DRIVER HEALTH AND FATIGUE MONITORING SYSTEM AND METHOD. Korean Registered Patent: 10-1619651, Registered: May 02, 2016 Title: Driver's monitoring device and its lighting control method.

SUMMARY OF THE INVENTION An object of the present invention is to provide a driver condition estimating apparatus which estimates a driver's condition simply and a method thereof.

A method for estimating a driver condition in a vehicle according to an embodiment of the present invention includes: sensing a face of a driver through a camera; Detecting a change in hue of the sensed face according to a blood pressure; Estimating a heart rate corresponding to the color tone change; And estimating a state of the driver based on the estimated heart rate.

In an embodiment, the step of sensing the face includes: extracting a landmark of the face; Cropping and tracing a region of interest in the extracted facial landmarks; And detecting the skin of the driver corresponding to the area of interest.

In an embodiment, the region of interest may include the driver's nose and ball region.

In one embodiment, the step of detecting the hue change comprises: temporally filtering the data of the region of interest; And performing a temporal selection rule on the filtered data to sort the rapidly changing heart rate.

In an exemplary embodiment, the temporary filtering may include removing a variation of a reference value or more from data of the ROI using a reflectance according to a wavelength; Performing non-specification and normalization on the data from which the mutation has been removed; Performing band filtering on the normalized data to remove a background frequency domain; Extracting Xsmin? Ys features from the band-filtered data to detect pure skin color independent of color of light; And performing power spectral density (PSD) conversion on the extracted data.

In the embodiment, the step of removing the variation of the reference value or more may further include removing the steep signal variation period through the standard deviation of the interval.

In one embodiment, performing the normalization and normalization comprises: modeling a signal for heart rate estimation; And normalizing the modeled signal using a normalized least squares solution.

In an embodiment, the band filtering may include band filtering using an n-th Hamming window.

In an embodiment, performing the temporal filtering may further comprise performing moving average filtering to remove residual temporal noise.

In an embodiment, estimating the heart rate may include estimating beats per minute (bpm) over a maximum frequency value.

In an embodiment, performing the ad hoc selection rule comprises: detecting a heart rate through comparison with a heart rate result corresponding to a previous three frame result; And interpolating the average value of the previous three frame results in the case of the detected heart rate.

The method may further include the step of informing the driver of an alarm related to the estimated driver's state by voice or video or using the alert to drive the vehicle.

A driver condition estimating apparatus installed in a vehicle according to an embodiment of the present invention includes a camera for sensing a face of a driver; And a driver condition estimator that detects a change in color tone of the face detected by the camera and estimates a heart rate of the driver according to the sensed change in color tone.

In an exemplary embodiment, the driver state estimator may select and track a region of interest through the feature points of the face, remove a sharp pixel change signal in the selected region of interest, The heart rate can be estimated.

The driver condition estimating apparatus and the operating method thereof according to the embodiment of the present invention can easily estimate the driver's condition according to the heart rate by estimating the heart rate through the change of the color tone of the driver's face.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. However, the technical features of the present embodiment are not limited to the specific drawings, and the features disclosed in the drawings may be combined with each other to constitute a new embodiment.
FIG. 1 is an exemplary view showing a vehicle 10 capable of estimating a driver's condition according to an embodiment of the present invention.
2 is a flow chart illustrating an exemplary method of preventing an accident of a vehicle 10 according to an embodiment of the present invention.
FIG. 3 is a flowchart showing the driver face extraction step (S110) shown in FIG. 2 in more detail.
FIG. 4 is a detailed view showing the step S111 of extracting the face landmark of the driver shown in FIG. 3. FIG.
FIG. 5 is a flowchart showing the driver's heart rate estimation step S120 shown in FIG. 2 in more detail.
FIGS. 6 and 7 are views showing the temporary filtering step S121 shown in FIG. 5 in more detail.
FIGS. 8 and 9 are diagrams illustrating a method of eliminating a sudden variation according to an embodiment of the present invention.
10 is a diagram illustrating band filtering using an n-th order Hamming window.
11 is a view for explaining a provisional selection rule process according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms.

The terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

Other expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well. The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, the terms "comprises" or "having" are intended to specify the presence of stated features, integers, steps, operations, components, parts, or combinations thereof, wherein one or more other features, , Steps, operations, components, parts, or combinations thereof, as a matter of course. Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be construed as meaning consistent with meaning in the context of the relevant art and are not to be construed as ideal or overly formal in meaning unless expressly defined in the present application .

FIG. 1 is an exemplary view showing a vehicle 10 capable of estimating a driver's heart rate according to an embodiment of the present invention. Referring to FIG. 1, a vehicle 10 may include a camera 100 and a driver condition estimator 200.

The camera 100 may be implemented to recognize the face of the driver. In an embodiment, the camera 100 may be implemented to image an image of a predetermined area centered on a predetermined area of the face of the driver. In an embodiment, the camera 100 may be attached to the inside of the vehicle 10. In an embodiment, the camera 100 may be attached so as to freely move the direction of the sensor for capturing an image. That is, the tilting angle of the camera 100 may be arbitrarily adjustable.

In an embodiment, the camera 100 may include a charge coupled device (CCD) image sensor, a metal oxide semi-conductor (MOS) image sensor, a charge priming device (CPD) And may be implemented as any one of image sensors such as a sensor.

The driver condition estimator 200 can be implemented to estimate the heart rate HR of the driver by analyzing and processing the facial image transmitted from the camera 100. [ In an embodiment, the driver condition estimator 200 may be implemented as an electronic control unit (ECU). In an embodiment, the driver condition estimator 200 may be operated based on a driver's heart rate estimation algorithm. Here, the driver's heart rate estimation algorithm can detect the change in facial tone depending on the blood pressure and estimate the heart rate corresponding to the change in facial tone.

In addition, the driver state estimator 200 can extract face detection and region of interest in the input image, track the region of interest, extract feature signals, and estimate the heart rate. In an embodiment, the region of interest signal characteristics may be transformed to extract the feature signal. In an embodiment, frequency conversion may be used for heart rate estimation. In an embodiment, the driver condition estimator 200 may be implemented with an image sensor processor (ISP) chip. The vehicle 10 may further include a display device (for example, navigation or the like) for displaying information processed in the driver condition estimator 200. [

The vehicle 10 according to the embodiment of the present invention can estimate the health and psychological state of the driver by sensing the change in facial tone caused by the blood pressure using the driver image acquired through the camera 100 for a vehicle. In particular, the vehicle 10 according to the embodiment of the present invention can more accurately estimate the heart rate of a vehicle driver through distortion signal filtering for correcting a light change and a blurring phenomenon occurring in an outdoor environment.

2 is a flow chart illustrating an exemplary method of preventing an accident of a vehicle 10 according to an embodiment of the present invention. Referring to Figs. 1 and 2, a method of preventing an accident of the vehicle 10 can be proceeded as follows.

During driving, the camera 100 can photograph the driver in real time and extract a specific area of the driver's face (S110). The on / off state of the camera 100 during driving can be optionally determined by the driver's choice. In an embodiment, face detection and feature points of an input image may be extracted. In an embodiment, both ball-of-interest area coordinates can be detected with respect to the nose tip.

The driver condition estimator 200 receives the image information photographed from the camera 100 and can detect a change in the facial color tone of the driver according to the blood pressure (S120). In an embodiment, the driver ' s facial color change may be sensed by learning. In another embodiment, the driver ' s facial color change may be detected by comparing it with information stored in the driver at a normal time. In an embodiment, signal conversion may be performed through a combination of R, G, and B channels to estimate a heart rate that is robust to light variations. In an embodiment, a particular signal may be filtered. Such filtering is performed by dividing the converted characteristic signals at a predetermined interval, analyzing the dynamic range and variation of the divided areas, separating the distorted signals, performing detrending and bandpass filtering, And performing feature signal conversion.

The driver condition estimator 200 may estimate the heart rate corresponding to the change in facial tone (S130). The driver condition estimator 200 can improve the reliability of remote photoplethysmography (rPPG) by using a range of interest (ROI) tracking technique and frequency analysis. In an embodiment, the frequency conversion of the feature signal is performed, and the heart rate can be estimated based on the value of the highest power spectrum density (PSD).

The driver condition estimator 200 can estimate the driver's condition (drowsiness, anger, pain, etc.) based on the estimated heart rate (S140).

Thereafter, a warning corresponding to the estimated driver's condition may be notified to the driver by audio or video (output of navigation-based heart rate result), or may be transmitted to an upper-level vehicle safety-related system (power train, etc.). Thus, a driving accident can be prevented (S150).

The method of estimating driver's heart rate according to an embodiment of the present invention includes extracting a region of interest in an outdoor or vehicle image, tracking a region of interest, performing distortion signal filtering according to an outdoor environment, performing a feature signal conversion, The heart rate can be estimated based on The driver's heart rate estimation method of the present invention can perform the distortion of the signal for correcting the blur phenomenon and the light change occurring in the outdoor environment, thereby performing the heart rate of the driver more accurately.

The steps and / or operations in accordance with the present invention may occur in different orders, in parallel, or concurrently in other embodiments for other epochs or the like, as may be understood by one of ordinary skill in the art .

Depending on the embodiment, some or all of the steps and / or operations may be performed on one or more non-transitory computer-readable media, including instructions, programs, interactive data structures, At least some of which may be implemented or performed using one or more processors. The one or more non-transitory computer-readable media can be, by way of example, software, firmware, hardware, and / or any combination thereof. Further, the functions of the " module " discussed herein may be implemented in software, firmware, hardware, and / or any combination thereof.

One or more non-transitory computer-readable media and / or means for implementing / performing one or more operations / steps / modules of embodiments of the present invention may be implemented as application-specific integrated circuits (ASICs), standard integrated circuits, But are not limited to, controllers that perform appropriate instructions, including microcontrollers, and / or embedded controllers, field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs) .

FIG. 3 is a flowchart showing the driver face extraction step (S110) shown in FIG. 2 in more detail. 1 to 3, a step S110 of extracting a driver face includes extracting a face landmark of a driver S111, ROI (Interest Area) cropping and tracking ) Step S112 and skin detection step S113.

FIG. 4 is a detailed view showing the step S111 of extracting the face landmark of the driver shown in FIG. 3. FIG. Referring to FIG. 4, in step S111 of extracting a face landmark, ROI (ROI) can be extracted through 66 face landmarks in order to reduce interference of non-rigid motion such as facial expression change.

In an embodiment, the nose and ball area, where the hue signal is most clearly reflected, may be selected as the ROI.

In an embodiment, tracking can be applied to the detected area to accommodate vibration and lighting changes during travel.

FIG. 5 is a flowchart showing the driver's heart rate estimation step S120 shown in FIG. 2 in more detail. Referring to FIG. 5, the driver's heart rate estimation step S120 may include a temporal filtering step S121 and an temporal selection rule step S122 of the ROI area.

FIGS. 6 and 7 are views showing the temporary filtering step S121 shown in FIG. 5 in more detail. 6 and 7, the temporary filtering step S121 includes steps S121-1 to S121-1 of removing a sudden variation (variation of a reference value or more), detrending with smoothed prior and normalizing (S121-2), a Hamming window (hamming window) to the band filter (bandpass filtering) step (S121-3), X _{s s} minαY feature extraction step (S121-4), and Welch (welch) PSD (power spectrum density that ) Conversion step S121-5.

Hereinafter, step S121-1 for eliminating abrupt variation will be described in more detail. FIGS. 8 and 9 are views for explaining a method of eliminating abrupt variations according to an embodiment of the present invention. Referring to FIG. 8, the hemoglobin has the highest reflectance at the G (green) channel, while the reflectance at the R (red) channel is the lowest. The following equation can be used to estimate the distortion signal according to the light change in the face.

On the other hand, sudden changes in illumination occurring during driving are not effective in estimating the heart rate. As shown in FIG. 9, the steep signal variation period can be eliminated through the standard deviation std.

In the following, we will describe in more detail the steps of detrending with smoothed prior and normalizing (S121-2). If the signal for estimating the heart rate has a periodicity, it has a stationary characteristic. A detrending technique can be applied to separate the stationary signal from the non-periodic signal. Here, the input signal can be modeled by the following equation.

Also, the normalized least squares formula can satisfy the following equation.

Where z _{stat is the} stationary signal, z _{trend is the} aperiodic signal, H is the observation matrix, θ is the regression parameter, v is the error, and D _{d is the} d-th derivative operator.

At this time, the final normalized signal can satisfy the following equation.

Hereinafter, the step S121-3 of performing bandpass filtering with a hamming window will be described in detail. In the case of frequency components other than the heart rate interval, there is a possibility of an error in the heartbeat estimation. Therefore, band filtering can be performed to remove the background frequency domain. As shown in FIG. 10, the n-th order Hamming window can satisfy the following equation.

Also, in the following description, the X _s min αY _s feature extraction method is described. It can be extracted by the standardized X, Y formula below to extract skin color irrespective of color of light.

On the other hand, band-filtered X, Y signals can be used for correction after skin normalization. The response of the light change through the correction of the mutation of the signal is possible as the following equation.

On the other hand, moving average filtering may be applied to remove the residual temporary noise. The M-point moving average filtering can satisfy the following equation.

On the other hand, the Welch PSD conversion step (S121-5) will be described in detail below. A power spectrum density (PSD) conversion can be performed to estimate the dominant frequency component. In an embodiment, noise-robust Welch's method PSD transformation can be performed. The PSD conversion can satisfy the following equation.

Where Q is the segment length, and w is the gaussian weight.

On the other hand, the heart rate can be calculated from the maximum frequency value by using the following equation: beats per minute (bpm)

11 is a view for explaining a provisional selection rule process according to an embodiment of the present invention. Referring to FIG. 11, a rapidly changing heart rate can be detected by comparing the heart rate result with the result of 3 frames before. In an embodiment, the detected heart rate may be interpolated through an average of the previous three frame results as: < RTI ID = 0.0 >

In the embodiment, the threshold value thr of the variation amount can be fixed to 5. However, it should be understood that the threshold value thr of the variation amount is not limited thereto.

The driver's heartbeat estimation algorithm according to the embodiment of the present invention can perform tracking through ROI selection and KCF through feature points of a face through DRMF.

The driver's heartbeat estimation algorithm according to the embodiment of the present invention can exclude a sharp pixel value change signal through a Gover R pixel value change within a certain period.

The driver's heart rate estimation algorithm according to the embodiment of the present invention can perform XminY signal conversion and estimate the heart rate through the maximum frequency of the PSD.

The driver's heart rate estimation algorithm according to the embodiment of the present invention can perform rapid change of the heart rate value through the provisional selection rule.

The driver's heartbeat estimation algorithm according to the embodiment of the present invention can be satisfied within an error rate of 10% in the case of 10 stop DBs.

The driver's heartbeat estimation algorithm according to the embodiment of the present invention may cause an error of 10% at some intervals in 10 travel DBs.

The driver's heartbeat estimation algorithm according to the embodiment of the present invention may cause an error of more than 10% due to the strong light change in the driving and the head turning DB.

The above-described contents of the present invention are only specific examples for carrying out the invention. The present invention will include not only concrete and practical means themselves, but also technical ideas which are abstract and conceptual ideas that can be utilized as future technologies.

10: Vehicle
100: camera
200: Driver condition estimator

Claims (14)

A method for estimating a driver condition in a vehicle, comprising:
Sensing a face of a driver through a camera;
Detecting a change in hue of the sensed face according to a blood pressure;
Estimating a heart rate corresponding to the color tone change; And
And estimating a state of the driver based on the estimated heart rate. The method according to claim 1,
The method of claim 1,
Extracting a landmark on the face;
Cropping and tracing a region of interest in the extracted facial landmarks; And
And detecting the skin of the driver corresponding to the region of interest. 3. The method of claim 2,
Wherein the region of interest comprises the nose and ball region of the driver. The method according to claim 1,
Wherein the step of sensing the color tone change comprises:
Temporally filtering the data of the ROI; And
And performing a temporary selection rule for the filtered data to sort the rapidly changing heart rate. 5. The method of claim 4,
Wherein the temporary filtering comprises:
Removing a variation of a reference value or more from the data of the ROI using the reflectance along the wavelength;
Performing non-specification and normalization on the data from which the mutation has been removed;
Performing band filtering on the normalized data to remove a background frequency domain;
Extracting Xsmin? Ys features from the band-filtered data to detect pure skin color independent of color of light; And
And performing a power spectral density (PSD) conversion on the extracted data. 6. The method of claim 5,
The step of removing a variation of the reference value or more includes:
And removing the steep signal variation period through the standard deviation of the intervals. 6. The method of claim 5,
The step of performing the normalization and normalization comprises:
Modeling a signal for heart rate estimation; And
And normalizing the modeled signal using a normalized least squares solution. 6. The method of claim 5,
Wherein the band-
and band filtering using an nth order Hamming window. 6. The method of claim 5,
Wherein the performing the temporal filtering comprises:
Further comprising performing moving average filtering to remove residual temporal noise. 10. The method of claim 9,
The step of estimating the heart rate comprises:
And estimating beats per minute (bpm) through the maximum frequency value. 5. The method of claim 4,
The step of performing the temporary selection rule includes:
Detecting a heart rate through comparison with a heart rate result value corresponding to a previous three frame result value; And
Interpolating over the mean of the previous three frame results in the case of the detected heart rate. The method according to claim 1,
Further comprising the step of notifying the driver of an alarm associated with the estimated driver's condition by voice or video or using the alert to drive the vehicle. 1. A driver condition estimating apparatus installed in a vehicle, comprising:
A camera for sensing the driver's face; And
And a driver state estimator for sensing a change in color tone of the face detected by the camera and estimating a heart rate of the driver according to the sensed change in color tone. 14. The method of claim 13,
Wherein the driver state estimator comprises:
A driver for selecting a region of interest through a feature point of the face, removing a sharp pixel change signal in the selected region of interest, and estimating a heart rate corresponding to a maximum frequency through frequency spectrum conversion.

Images