KR101988581B1 - Method for estimating condition of driver - Google Patents

Abstract

A method of estimating a driver's condition in a vehicle according to the present invention may include detecting a driver's face through a camera, detecting a change in a color tone of the detected face according to blood pressure, and estimating a heart rate corresponding to the change in the color tone. And estimating the driver's condition based on the estimated heart rate.

Description

How to estimate driver status {METHOD FOR ESTIMATING CONDITION OF DRIVER}

The present invention relates to a driver state estimation apparatus and method.

In general, the driver monitoring apparatus photographs a driver who is driving by using a camera and determines a driver's driving state (squint, drowsy driving, gaze point, etc.) based on the captured face image of the driver. In addition, the driver monitoring apparatus determines the driving state of the vehicle by using the steering angle signal, the vehicle speed signal, and the driving position of the vehicle while driving. In addition, the driver monitoring apparatus may use the driver state and the driving state to warn the driver when the driver is in an unsafe driving state, thereby enabling safe driving.

Japanese Patent Application Laid-Open No. 2015-194884, Publication Date: November 05, 2015 Title: Driver Monitoring System. International application number: WO2015 / 030219, published date: 19 November 2015, title: DRIVER HEALTH AND FATIGUE MONITORING SYSTEM AND METHOD. Korean Patent Registration: 10-1619651, registered date: May 02, 2016, title: driver monitoring device and its lighting control method.

SUMMARY OF THE INVENTION An object of the present invention is to provide a driver state estimating apparatus and method thereof for simply estimating a driver's state.

According to an embodiment of the present invention, a method of estimating a driver's condition in a vehicle includes: detecting a driver's face through a camera; Detecting a change in the tone of the detected face according to blood pressure; Estimating a heart rate corresponding to the change in color tone; And estimating the driver's state based on the estimated heart rate.

The detecting of the face may include extracting a landmark of the face; Cropping and tracking a region of interest in the extracted facial landmarks; And detecting the skin of the driver corresponding to the ROI.

In an embodiment, the ROI may include a nose and a ball area of the driver.

The detecting of the color tone may include: temporarily filtering the data of the ROI; And performing a temporary selection rule on the filtered data to sort the rapidly changing heart rate.

The temporal filtering may include removing a variation of a reference value or more from data of the ROI using reflectance according to a wavelength; Performing non-trending and normalization on the data from which the mutation has been removed; Performing band filtering on the normalized data to remove a background frequency region; Extracting XsminαYs features from the band filtered data to detect pure skin color independent of the color of light; And performing a power spectral density (PSD) transformation on the extracted data.

In an embodiment, the step of removing the variation greater than or equal to the reference value may further include removing the abrupt signal variation section through the standard deviation for each section.

In an embodiment, the performing of detrending and normalization may include: modeling a signal for heart rate estimation; And normalizing the modeled signal using a normalized least squares solution.

In an embodiment, the band filtering may include band filtering using an n-th order Hamming window.

In an embodiment, the performing of the temporary filtering may further include performing moving average filtering to remove residual temporary noise.

In an embodiment, estimating the heart rate may include estimating bpm (beats per minutes) based on a maximum frequency value.

In an embodiment, the performing of the temporary selection rule may include detecting a heart rate by comparing with a heart rate result value corresponding to a previous three frame result value; And in the case of the detected heart rate, interpolating through an average value of the previous three frame result values.

In an embodiment, the method may further include notifying the driver by voice or video of an alarm related to the estimated driver's state or using the driver to drive the vehicle.

An apparatus for estimating a driver state installed in a vehicle according to an exemplary embodiment of the present disclosure may include a camera configured to detect a face of a driver; And a driver state estimator for detecting a change in the color tone of the face detected by the camera and estimating a heart rate of the driver according to the detected change in the color tone.

The driver state estimator selects and tracks a region of interest through the feature points of the face, removes a sudden pixel change signal from the selected region of interest, and corresponds to a maximum frequency through frequency spectrum conversion. The heart rate can be estimated.

An apparatus for estimating a driver's state and an operation method thereof according to an exemplary embodiment of the present invention can easily estimate a driver's state according to a heart rate by estimating a heart rate through a change in color tone of a driver's face.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are provided to facilitate understanding of the present embodiment, and provide embodiments with a detailed description. However, the technical features of the present embodiment are not limited to the specific drawings, and the features disclosed in the drawings may be combined with each other to constitute a new embodiment.
1 is a diagram illustrating a vehicle 10 capable of estimating a driver state according to an exemplary embodiment of the present invention.
2 is a flowchart illustrating an accident prevention method of the vehicle 10 according to an exemplary embodiment of the present invention.
FIG. 3 is a flowchart showing in more detail a step S110 of extracting a driver's face shown in FIG. 2.
4 is a view showing in detail the step (S111) of extracting the facial landmarks of the driver shown in FIG.
FIG. 5 is a flowchart illustrating the driver heart rate estimation step S120 shown in FIG. 2 in more detail.
6 and 7 illustrate in more detail the step S121 of temporary filtering illustrated in FIG. 5.
8 and 9 are diagrams illustrating a method of removing a sudden shift according to an embodiment of the present invention.
10 illustrates band filtering using an n-th order Hamming window.
11 illustrates a temporary selection rule process according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, the contents of the present invention will be described clearly and in detail so that those skilled in the art can easily implement the drawings.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms.

The terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that another component may be present in the middle. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

Other expressions describing the relationship between components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring", should be interpreted as well. The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this application, the terms "comprise" or "having" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof that is implemented, and that one or more other features or numbers are present. It should be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof. Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. .

1 is a diagram illustrating a vehicle 10 capable of estimating a driver's heart rate according to an exemplary embodiment of the present invention. Referring to FIG. 1, the vehicle 10 may include a camera 100 and a driver state estimator 200.

The camera 100 may be implemented to recognize a driver's face. In an embodiment of the present disclosure, the camera 100 may be implemented to capture an image corresponding to a predetermined area around the predetermined area among the driver's face. In an embodiment, the camera 100 may be attached to the inside of the vehicle 10. In an embodiment, the camera 100 may be attached to freely move the direction of the sensor for capturing an image. That is, the tilting angle of the camera 100 may be arbitrarily adjusted.

In an embodiment, the camera 100 may include a charge coupled device (CCD) image sensor, a metal oxide semi-conductor (MOS) image sensor, a charge priming device (CPD) image sensor, and a charge injection device (CID) image. The image sensor may be implemented as any one of image sensors such as a sensor.

The driver condition estimator 200 may be implemented to estimate a driver's heart rate (HR) by analyzing and processing a face image transmitted from the camera 100. In an embodiment, the driver state estimator 200 may be implemented as an electronic control unit (ECU). In an embodiment, the driver state estimator 200 may be operated based on a driver heartbeat estimation algorithm. Here, the driver's heartbeat estimation algorithm detects a change in facial tone according to blood pressure and estimates a heart rate corresponding to the change in facial tone.

The driver state estimator 200 may extract a face detection and a region of interest in the input image, track a region of interest, extract a feature signal, and estimate a heart rate. In an embodiment, the ROI feature may be converted to extract the feature signal. In an embodiment, frequency conversion may be used for heart rate estimation. In an embodiment, the driver state estimator 200 may be implemented as an image sensor processor (ISP) chip. Although not shown, the vehicle 10 may further include a display device (for example, a navigation device) for displaying information processed by the driver state estimator 200.

The vehicle 10 according to an exemplary embodiment of the present disclosure may estimate a driver's health and psychological state by detecting a change in facial tone due to blood pressure using the driver's image acquired through the vehicle camera 100. In particular, the vehicle 10 according to an exemplary embodiment of the present invention may more accurately estimate the heart rate of the vehicle driver through filtering distortion signals for correcting light changes and blurring occurring in an outdoor environment.

2 is a flowchart illustrating an accident prevention method of the vehicle 10 according to an exemplary embodiment of the present invention. 1 and 2, the accident prevention method of the vehicle 10 may proceed as follows.

When driving, the camera 100 may photograph the driver in real time and extract a specific region of the driver's face (S110). On / off of the camera 100 during driving may be optionally determined by the driver's selection. In an embodiment, face detection and feature points of the input image may be extracted. In an embodiment, both ball ROI coordinates may be detected based on the tip of the nose.

The driver state estimator 200 may receive image information captured by the camera 100 and may detect a change in the facial tone of the driver according to blood pressure (S120). In an embodiment, the change in the facial tone of the driver may be sensed by learning. In another embodiment, the change in the facial tone of the driver may be sensed by comparing with information stored in the driver. In an embodiment, the signal conversion may be performed through a combination of the R, G, and B channels in order to estimate the heart rate that is robust to light changes. In an embodiment, a particular signal may be filtered. This filtering divides the transformed feature signals into overlapping at regular intervals, analyzes the dynamic range and variation of the divided regions, separates the distortion signals, detrends and bandpasses the filtering, and XminY. Performing feature signal conversion.

The driver state estimator 200 may estimate a heart rate corresponding to a change in facial tone (S130). The driver condition estimator 200 may improve the reliability of remote photoplethysmography (rPPG) by using a range of interest (ROI) tracking technique and frequency analysis. According to an embodiment, the frequency conversion of the feature signal may be performed, and the heart rate may be estimated based on a frequency having the highest power spectrum density (PSD).

The driver state estimator 200 may estimate a driver's state (drowsiness, anger, pain, etc.) based on the estimated heart rate (S140).

Thereafter, a warning corresponding to the estimated driver's state may be notified to the driver by voice or video (navigation-based heart rate result output) or transmitted to a vehicle safety related upper system (powertrain, etc.). As a result, a driving accident may be prevented (S150).

The driver heart rate estimation method according to an embodiment of the present invention extracts a region of interest from an outdoor or vehicle image, tracks a region of interest, performs distortion signal filtering according to an outdoor environment, performs a characteristic signal conversion, and analyzes frequencies. Based on the heart rate can be estimated. The driver's heart rate estimation method of the present invention can perform the heart rate of the vehicle driver more precisely by performing distortion signal filtering for light change and blur phenomenon occurring in an outdoor environment.

The steps and / or actions according to the invention may occur simultaneously in different embodiments in different order, in parallel, or for other epochs, etc., as would be understood by one of ordinary skill in the art. Can be.

In some embodiments, some or all of the steps and / or actions may be directed to instructions, programs, interactive data structures, clients, and / or servers stored on one or more non-transitory computer-readable media. At least some may be implemented or performed using one or more processors. One or more non-transitory computer-readable media may be illustratively software, firmware, hardware, and / or any combination thereof. In addition, the functionality of the "module" discussed herein may be implemented in software, firmware, hardware, and / or any combination thereof.

One or more non-transitory computer-readable media and / or means for implementing / performing one or more operations / steps / modules of embodiments of the present invention may be used in application-specific integrated circuits (ASICs), standard integrated circuits, A controller that performs appropriate instructions, including a microcontroller, and / or an embedded controller, field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and the like. Does not.

FIG. 3 is a flowchart showing in more detail a step S110 of extracting a driver's face shown in FIG. 2. 1 to 3, the step S110 of extracting a driver's face may include extracting a face landmark of a driver (S111), ROI cropping and tracking (ROI). ) Step S112 and skin detection step S113.

4 is a view showing in detail the step (S111) of extracting the facial landmarks of the driver shown in FIG. Referring to FIG. 4, in step S111 of extracting facial landmarks, ROIs (regions of interest) may be precisely extracted through 66 facial landmarks in order to reduce interference of non-rigid motion such as facial expression changes.

In an embodiment, the nose and cheek area where the color signal is most clearly reflected may be selected as the ROI.

In an embodiment, tracking may be applied to the detected area in order to cope with vibrations and changes in illumination when driving.

FIG. 5 is a flowchart illustrating the driver heart rate estimation step S120 shown in FIG. 2 in more detail. Referring to FIG. 5, the driver heart rate estimation step S120 may include a step S121 of temporally filtering the ROI region and a temporal selection rule step S122.

6 and 7 illustrate in more detail the step S121 of temporary filtering illustrated in FIG. 5. 6 and 7, the temporary filtering step S121 may include removing a sudden variation (a variation of a reference value or more) (S121-1), detrending with smoothed priority, and normalizing it. S121-2, bandpass filtering with a hamming window S121-3, X _s minαY _s feature extraction step S121-4, and weld PSD power spectrum density ) May include a conversion step (S121-5).

Hereinafter, the step (S121-1) of removing the sudden mutation will be described in more detail. 8 and 9 are diagrams illustrating a method of removing a sudden mutation according to an embodiment of the present invention. Referring to FIG. 8, the hemoglobin has the highest reflectance in the G (green) channel while the lowest in the R (red) channel. The following equation may be used to estimate the distortion signal according to the change of light in the face.

On the other hand, sudden changes in illuminance generated during driving are not effective for estimating heart rate. As shown in FIG. 9, the abrupt signal fluctuation period may be removed through the standard deviation std for each period.

In addition, the following steps (S121-2) for detrending with smoothed prior and normalizing (normalizing) will be described in more detail. The signal for heart rate estimation has a stationary characteristic if it has periodicity. Detrending techniques can be applied to separate stationary signals from aperiodic signals. In this case, the input signal may be modeled by the following equation.

In addition, the normalized least squares formula may satisfy the following equation.

Where z _stat : stationary signal, z _trend: aperiodic signal, H: observation matrix, θ: regression parameter, v: error, and D _d : d-th derivative operator.

In this case, the final normalized signal may satisfy the following equation.

In addition, below (S121-3) will be described in detail for the bandpass filtering with a hamming window (hamming window). In the case of frequency components other than the heart rate section, there is a possibility of error in heart rate estimation. Accordingly, band filtering may be performed to remove the background frequency domain. As shown in FIG. 10, the n-th order Hamming window may satisfy the following equation.

In addition, the following describes the X _s minαY _s feature extraction method. Standardized X, Y (X _s , Y _s ) for extraction of skin color irrespective of light color can be extracted with the following formula.

Meanwhile, band-filtered X and Y signals X _f and Y _f may be used for correction after normalization of the skin. It is possible to cope with the light change through the correction of the signal variation as shown below.

Meanwhile, moving average filtering may be applied to remove residual temporal noise. M-point moving average filtering may satisfy the following equation.

In the following description, the weld PSD conversion step S121-5 will be described in detail. Power spectrum density (PSD) transformation may be performed to estimate the major frequency components. In an embodiment, PSD conversion of the Welch's method, which is robust to noise, may be performed. PSD conversion may satisfy the following equation.

Where Q is segment length and w is gaussian weight.

On the other hand, heart rate estimation can be calculated by the following equation bpm (beats per minute) through the maximum frequency value.

11 illustrates a temporary selection rule process according to an embodiment of the present invention. Referring to FIG. 11, a rapidly changing heart rate may be detected by comparing a result value before three frames and a heart rate result value. In an embodiment, the detected heart rate may be interpolated as shown in the following equation through an average value of the previous three frame result values.

In an embodiment, the threshold thr of the variation amount may be fixed to five. However, it will be understood that the threshold thr of variation is not limited thereto.

The driver's heartbeat estimation algorithm according to an embodiment of the present invention may perform ROI selection and tracking through the KCF through the feature points of the face through the DRMF.

The driver's heartbeat estimation algorithm according to an exemplary embodiment of the present invention may exclude a sudden pixel value change signal through a change in a GoverR pixel value within a predetermined section.

The driver's heartbeat estimation algorithm according to an embodiment of the present invention may perform the XminY signal conversion and estimate the heart rate through the maximum frequency of the PSD.

The driver's heartbeat estimation algorithm according to an embodiment of the present invention may perform a rapidly changing heart rate value alignment through a temporary selection rule.

The driver's heartbeat estimation algorithm according to an embodiment of the present invention may satisfy 10% or less of an error rate in case of 10 stop DBs.

In the driver's heartbeat estimation algorithm according to an exemplary embodiment of the present invention, an error of 10% early in some sections may occur in the case of 10 driving DBs.

In the driver's heartbeat estimation algorithm according to an exemplary embodiment of the present invention, an error of 10% or more may occur due to a strong light change in the case of driving and a head turn DB.

On the other hand, the content of the present invention described above is only specific embodiments for carrying out the invention. The invention will include not only specific and practically available means per se, but also technical ideas as abstract and conceptual ideas that may be utilized in future technology.

10: vehicle
100: camera
200: driver condition estimator

Claims (14)

CLAIMS What is claimed is: 1. A method for estimating a driver condition in a vehicle;
Detecting a region of interest on the face of the driver through a camera;
Detecting a change in color tone of the region of interest according to blood pressure;
Estimating a heart rate corresponding to the change in color tone; And
Estimating a state of the driver based on the estimated heart rate,
Detecting the change in color tone,
Temporarily filtering the data of the ROI; And
Performing a temporal selection rule on the filtered data to sort the rapidly changing heart rate,
The temporary filtering step,
Removing a variation of a reference value or more from the data of the region of interest using reflectance according to a wavelength;
Performing non-trending and normalization on the data from which the mutation has been removed;
Performing band filtering on the normalized data to remove a background frequency region;
Extracting data according to a pure skin color irrespective of light color from the band filtered data; And
Performing a power spectral density (PSD) transformation on the extracted data. The method of claim 1,
Detecting the ROI of the face,
Extracting a landmark of the face;
Cropping and tracking the region of interest in the extracted facial landmarks; And
Detecting the skin of the driver corresponding to the region of interest. The method of claim 2,
The region of interest includes the nose and ball area of the driver. delete delete The method of claim 1,
Removing the variation above the reference value,
And eliminating abrupt signal fluctuation intervals through the section standard deviations. The method of claim 1,
Performing the de-trending and normalization,
Modeling a signal for heart rate estimation; And
Normalizing the modeled signal using a normalized least squares solution. The method of claim 1,
The band filtering step,
band filtering using an nth order Hamming window. The method of claim 1,
The performing of the temporary filtering may include:
Performing moving average filtering to remove remaining temporary noise. The method of claim 9,
Estimating the heart rate,
Estimating bpm (beats per minutes) through the maximum frequency value. The method of claim 1,
The step of performing the temporary selection rule is
Detecting a heart rate by comparing the heart rate result value corresponding to the previous three frame result value; And
Interpolating through the average of the previous three frame result values for the detected heart rate. The method of claim 1,
And notifying the driver by voice or video of the alarm related to the estimated driver's condition or driving the vehicle. delete delete

Images