KR20200116825A - Apparatus and method for driver condition recognition - Google Patents

Abstract

The present invention provides an apparatus and a method for recognizing the state of a driver by using a face image photographed under the face of the driver. In addition, the present invention provides a driver-wearable image collection device for collecting a face image to be used for driver state recognition. According to the present invention, the apparatus for recognizing the state of the driver comprises: a storage for recognizing the state of the driver based on a face image of the driver, which is captured by at least one camera, and storing the face image; and a processor.

Description

Driver condition recognition device and method {APPARATUS AND METHOD FOR DRIVER CONDITION RECOGNITION}

The present invention relates to an apparatus and method for recognizing a driver's condition, and more particularly, to an apparatus and method for recognizing a driver's condition based on a driver's face image.

In order to prevent accidents caused by drivers' drowsy driving and provide personalized and differentiated services, studies such as the development of a driver's drowsiness detection device are being conducted.

In the prior art, in order to analyze the driver's fatigue, the blinking speed is decreased or the eyes are closed for a long time through a camera. In addition, with the development of image processing processors, image processing speed is improved, and deep learning research is grafted together to introduce various image processing technologies applicable to embedded systems.

Meanwhile, in order to grasp the driver's condition, the facial expression and voice of the driver's face are analyzed, and for this, a camera, a microphone, and a sensor for measuring biometric signals are also used.

In order to check the driver's condition and emotion through facial expressions, a method of attaching a device to the outside is used, but when wearing sunglasses, bowing his face, or turning his head to one side, it is difficult to detect eyes, nose, and mouth. There is. In particular, it is difficult to detect a face due to reflection of shade or light, and it may be difficult to detect a face with only an image due to actions of touching the face with a hand or covering the face.

When predicting the condition by photographing the driver's front face, the burden that the face screen is being directly photographed may give the driver a feeling of hesitation and make it difficult for the driver to concentrate on driving.

In order to solve the above-described problem, the present invention provides an apparatus and method for recognizing a driver's state by using a face image photographed under the driver's face.

In addition, the present invention provides a driver wearable image collection device for collecting face images to be used for driver state recognition.

The above-described objects and other objects, advantages, and features of the present invention, and methods of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings.

An apparatus for recognizing a driver's state according to an aspect of the present invention is an apparatus for recognizing a driver's state based on a face image of a driver photographed with at least one camera, and includes a storage and a processor for storing the face image, and the processor, It may be set to acquire a driver's face image, extract a reference region from the face image, extract a feature boundary from the face image based on the reference region, and determine the driver's state based on a change in the feature boundary.

In one example, the processor is further configured to determine a response indication based on the driver's condition.

An image collection device according to another aspect of the present invention includes a body wearable by a user and at least one LED camera cell disposed on the body to capture a face image of the user.

In one example, the body may be formed in a neck band type.

In an example, at least one LED camera cell may photograph a user's face in a direction looking up from the bottom to the top.

In an example, at least one LED camera cell may capture a face image of a user's first view and a face image of a second view.

In an example, the image collection device further includes a communication module that transmits a face image and receives a response instruction for a user's state information determined based on the face image.

In an example, the image collection device further includes an output unit that outputs a response to the user based on the response instruction.

A driver state recognition method according to another aspect of the present invention is a method of recognizing a driver's state based on a driver's face image captured by at least one camera, comprising: acquiring a driver's face image, a reference region from the face image And extracting, extracting a feature boundary from the face image based on the reference region, and determining a driver's state based on a change in the feature boundary.

In an example, the face image may be an image taken of a driver's face in a direction looking up from the bottom to the top.

In an example, in the step of extracting the reference region, a region of the face image corresponding to the driver's nose region may be extracted as the reference region.

In an example, the reference area may have a triangular shape.

In one example, extracting the feature boundary includes extracting an ROI of a face image corresponding to an area including the driver's eyes and mouth, and determining the ROI as a feature boundary based on the positional relationship between the reference region and the ROI. It includes the step of.

In one example, the feature boundary may have a historical trapezoidal shape.

In an example, acquiring the face image includes acquiring a first face image of a first view and acquiring a second face image of a second view.

In an example, in determining the driver's state, the driver's state may be determined based on a change in a feature boundary between the first face image and the second face image.

In an example, the characteristic boundary is an inverse trapezoidal shape, and the change in the characteristic boundary may be at least one of a height change, a length change of a lower side, and an area change of the inverse trapezoidal shape.

According to the present invention, it is possible to recognize and predict the driver's state even in a distorted face image other than a front face image.

In addition, according to the present invention, it is not necessary to separately mount a camera on the vehicle, and the number of feature points of the distorted face image is smaller than that of the front face image, so that the processing speed is high.

In addition, the image collection device according to the present invention may acquire face images from various angles.

1 shows a driver state recognition system according to an embodiment.
2 shows exemplary driver state information that can be recognized based on a face image.
3 shows an image collection device according to an embodiment.
4 is a block diagram of an apparatus for recognizing a driver's state according to an embodiment.
5 is a flowchart illustrating a driver state recognition process according to an embodiment.
6 shows a process of recognizing a driver's state according to an embodiment.
7 shows a process of recognizing a driver's state according to an embodiment.
8 is a flowchart illustrating a process of determining a driver's state according to an example.

The aspect in which the present invention is implemented will be described with reference to each of the following preferred embodiments. It is obvious that the present invention is not limited to the embodiments disclosed below, but can be implemented in other various forms within the scope of the technical idea of the present invention. The terms used in the present specification are also intended to describe embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, "comprise" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the stated component, step, action and/or element is Or does not preclude addition.

1 shows a driver state recognition system according to an embodiment.

The driver state recognition system is a system that analyzes and predicts a current state of a driver in a vehicle, and includes an image collection device 150 and a driver state recognition device 100.

The image collection device 150 acquires a driver's face image using at least one camera and transmits the acquired face image to the driver state recognition device 100. Here, the image collection device 150 may acquire a distorted face image rather than a front face image of the driver. For example, the image collection device 150 acquires an image photographed in a direction from the lower portion of the driver's face toward the upper portion and transmits the acquired image to the driver state recognition apparatus 100. A face image photographed in a direction from the lower portion of the driver's face toward the upper portion is referred to as a distorted face image hereinafter.

The driver state recognition device 100 determines a current state of the driver by analyzing the face image acquired from the image collection device 150.

The driver state recognition device 100 may generate a response based on the driver's current state and transmit it to the image collection device 150. For example, the response can be a response indication.

The image collection device 150 may generate an appropriate output based on the received response. For example, the image collection device 150 may generate at least one of an auditory signal, a visual signal, and a vibration signal as an output and transmit it to the driver.

In an example, the image collection device 150 and the driver state recognition device 100 may be physically separated devices. In this case, the image collection device 150 and the driver state recognition device 100 may communicate with each other by providing a wired or wireless communication means. For example, the driver state recognition device 100 may be implemented as a server located inside or outside the vehicle and may interwork with at least one image collection device 150.

In another example, the image collection device 150 and the driver state recognition device 100 may be implemented integrally. For example, the image collection device 150 may be embedded with a hardware module implementing the driver state recognition apparatus 100 to be described later with reference to FIG. 4.

2 shows exemplary driver state information that can be recognized based on a face image.

Conventionally, in order to determine the driver's current state, the driver's front face was detected and features within the face were utilized. Existing methods perform eye, nose, mouth and face extraction, edge shape extraction of each part, and eye opening angle extraction to find facial feature points. Instead of performing the image preprocessing process for finding feature points, a method for finding features in the network by inputting them into a convolutional neural network (CNN) can be used, and the two methods are also used in combination.

In FIG. 2, seven or more emotions (neutral, anger, sadness, surprise, disgust, anxiety, and happiness) may be classified for one face image in order to distinguish various emotional states.

According to an apparatus and method for recognizing a driver's state according to an embodiment, a distorted face image may be obtained to recognize a driver's emotions, fatigue, and drowsiness.

3 shows an image collection device according to an embodiment.

The image collection device 150 includes a body 310 wearable by the user and at least one LED camera cell 320 disposed on the body 310 to capture a face image of the user.

In one example, the body 310 of the image collection device 150 may be formed in a wearable shape so that the user can wear it. In one example, the body 310 of the image collection device 150 may be formed in a neck band type as exemplarily shown in FIGS. 3(a) and 3(c). For example, the shape of the neckband can be transformed into various forms that can be worn on the neck such as ∩, ∪, c, ㅁ, and O-shaped. In another example, the image collection device 150 may be implemented as a bracelet, brooch, headset, or belt wearable by the user.

On the other hand, in FIGS. 3(a) and 3(c), at least one LED camera cell 320 is arranged in a line on the body 310 at regular intervals, but various arrangements of other forms are possible. For example, at least one LED camera cell 320 may be arranged in several rows. For example, at least one LED camera cell 320 may be arranged in a zigzag. For example, at least one LED camera cell 320 constitutes an array of small cameras.

The image collection device 150 may capture a face image of a user from various angles by driving at least one LED camera cell 320. In contrast, when a camera is mounted on a vehicle, a face image can be photographed only from a limited angle, whereas an image analysis can be performed based on a face image photographed from various angles, thereby improving the accuracy of the recognized driver's state.

3(b) shows an exemplary LED camera cell 320. The LED camera cell 320 may include a camera 322 and a plurality of LEDs 324 disposed around the camera 322. Since the LED camera cell 320 of the image recognition apparatus 150 is equipped with at least one LED 324, the LED 324 can be operated to take a picture with the camera 322 even in a dark place.

3(c) shows an example in which the image collection device 150 is mounted by a user. The user is wearing an image collection device 150 including at least one LED camera cell 320 including at least one LED 324 and a camera 322.

As illustrated by way of example in FIG. 3C, the image recognition apparatus 150 may photograph a face surface in a direction from a lower face to an upper face when the user wears it. That is, at least one LED camera cell 320 may photograph the user's face in a direction looking up from the bottom to the top.

The image recognition device 150 drives at least one LED camera cell 320 to capture a face image of a user's first viewpoint and a face image of a second viewpoint. The first time point and the second time point may be two different time points in a continuous time. For example, at least one LED camera cell 320 periodically captures a face image of a first view and a face image of a second view according to a preset time interval. Here, the time interval may be adjusted according to the current driver condition.

In an example, the image recognition apparatus 150 may further include a biometric sensor. For example, the image recognition apparatus 150 may include a heart rate sensor, a blood pressure sensor, a blood glucose sensor, and/or a temperature sensor. Here, the image recognition apparatus 150 may adjust a time period for acquiring a face image by driving at least one LED camera cell 320 based on a biosignal received from the biometric sensor. For example, the image recognition apparatus 150 may acquire a face image more often when the user's heart rate received from the biometric sensor is lower than usual.

The image collection device 150 transmits the photographed face image to the state recognition device 100, and receives a response instruction for the user's state information determined by the state recognition device 100 based on the corresponding face image. It may contain more.

In addition, the image collection device 150 may further include an output unit that outputs a response to a user based on a response instruction received from the state recognition device 100. For example, the response may include at least one of an auditory signal, a visual signal, and a vibration signal.

In another example, the image collection device 150 may be mounted on a vehicle. For example, the image collection device 150 may be mounted on a wheel of a vehicle. In this case, the handle corresponds to the body 310 of the image collection device 150.

4 is a block diagram of an apparatus for recognizing a driver's state according to an embodiment.

The driver state recognition apparatus 100 recognizes the driver state based on the driver's face image captured with at least one camera. For example, the driver's state recognition device 100 may recognize the driver's state based on a driver's face image captured by at least one LED camera cell included in the image collection device 150.

As shown in the functional block diagram of FIG. 4, the driver state recognition device 100 includes a storage unit 420 that stores an acquired face image, and a communication unit that communicates with the image collection device 150 including at least one camera ( 430) and a controller 410 that determines the driver's state based on the acquired face image.

The driver state recognition apparatus 100 includes a storage unit corresponding to the storage unit 420 for storing the acquired face image and a processor for driving the control unit 410.

Additionally, the driver state recognition apparatus 100 may include a wired/wireless communication module corresponding to the communication unit 430.

The processor of the driver state recognition apparatus 100 acquires a driver's face image, extracts a reference region from the face image, extracts a feature boundary from the face image based on the reference region, and provides the driver based on the change in the feature boundary. It is set to drive the control unit 410 that determines the state of. In addition, the processor of the driver state recognition apparatus 100 may be further configured to drive the controller 410 that determines a response instruction based on the driver's state. A detailed operation of the control unit 410 will be described later with reference to FIG. 5.

5 is a flowchart illustrating a driver state recognition process according to an embodiment.

The driver's state recognition method recognizes the driver's state based on a driver's face image captured by at least one camera. As described above, for example, the driver state recognition method may recognize the driver state based on the driver's face image captured by at least one LED camera cell included in the image collection device 150.

In step 510, with reference to FIG. 4, the controller 410 acquires a face image of the driver. For example, the control unit 410 acquires a face image captured by the image collection device 150 in a direction in which the driver's face is looking up from the bottom to the top through the communication unit 430. For example, the controller 410 acquires a first face image of a first view and subsequently acquires a second face image of a second view. For example, the controller 410 periodically acquires a face image according to a preset time interval.

In step 520, the controller 410 extracts a reference region from the face image acquired in step 510.

The reference area is a reference area for finding a feature boundary in a distorted face shape.

In an example, the controller 410 extracts a region of a face image corresponding to the driver's nose region as a reference region. In step 510, the controller 410 may extract a feature point from the acquired face image using a feature point extraction technique, and extract a region corresponding to the driver's nose region as a reference region therefrom. For example, the control unit 410 may extract the reference region in a triangular shape. In an example, the controller 410 may extract a reference region by analyzing a face image based on a landmark database that has been pre-built.

In one example, the control unit 410 may select and use an image from which a reference region is clearly extracted from images captured by at least one LED camera cell of the image recognition device 150.

In step 530, the controller 410 extracts a feature boundary from the face image based on the reference region extracted in step 520.

The feature boundary is a target area used to determine the driver's current state in the distorted face shape. The characteristic boundary changes in length or area according to the driver's current state.

In one example, the controller 410 extracts an ROI of a face image corresponding to a region including the driver's eyes and mouth. For example, the controller 410 may extract a feature point from the acquired face image using a feature point extraction technique, and extract a region corresponding to the maximum region including the driver's eyes and mouth as the ROI. For example, the region of interest is extracted as a historical trapezoid. In an example, the controller 410 may extract a region of interest by analyzing a face image based on a landmark database that has been pre-built.

Subsequently, in step 530, the controller 410 determines the region of interest as a feature boundary based on the positional relationship between the reference region extracted in step 520 and the previously extracted region of interest. For example, when more than half of the reference region overlaps the region of interest, the controller 410 determines the region of interest as an effective feature boundary. For example, when the reference area has a triangular shape, when at least two vertices of the corresponding triangle are located within the ROI, the controller 410 determines the ROI as an effective feature boundary.

In one example, the control unit 410 extracts the feature boundary into an inverse trapezoid shape. In an example, the controller 410 may select and use an image from which a feature boundary is clearly extracted from images captured by at least one LED camera cell of the image recognition device 150.

In step 540, the control unit 410 determines the driver's state based on the change in the feature boundary extracted in step 530.

For example, the controller 410 may determine the driver's state based on a change in a feature boundary between a first face image of a first view and a second face image of a second view acquired in operation 510. Here, the change of the feature boundary includes at least one of a change in the shape of the feature boundary, a change in length of one side, and a change in area. For example, the characteristic boundary has an inverse trapezoid shape, and the controller 410 determines the driver's state based on at least one of a change in height, a change in length of the lower side, and a change in area of the inverse trapezoid shape of the characteristic boundary.

6 shows a process of recognizing a driver's state according to an embodiment.

Specifically, FIG. 6 shows a distorted image 610 obtained in step 510 with reference to FIG. 5, a reference region 620 extracted in step 520, and a feature boundary 630 extracted in step 530. It is shown for each driver state recognized at (540). For example, the reference region 620 was extracted as a triangular shape, and the feature boundary 630 was extracted as an inverse trapezoidal shape.

6(a) shows a reference area 620 and a characteristic boundary 630 when the driver's state is a normal state. 6(b) shows a reference area 620 and a characteristic boundary 630 when the driver's state is a yawning state. 6(c) shows a reference area 620 and a characteristic boundary 630 when the driver's eyes are closed. 6(d) shows a reference area 620 and a characteristic boundary 630 when the driver's mouth is tightly closed.

Referring to FIG. 4, in step 540, the controller 410 includes a reference region 620 and a characteristic boundary 630 in the normal state of FIG. 6(a), and FIGS. 6(b), 6(c), and The reference area 620 and the characteristic boundary 630 of FIG. 6D are compared, and a current state of the driver is determined based on the change of the characteristic boundary 630.

7 shows a process of recognizing a driver's state according to an embodiment.

7 schematically illustrates an arrangement relationship between the reference region 620 and the feature boundary 630 according to the driver's state. Exemplarily, the reference region 620 has a triangular shape, and the feature boundary 630 has an inverse trapezoid shape.

7(a) shows a reference area 620 and a characteristic boundary 630 when the driver's state is a normal state corresponding to FIG. 7(a). 7(b) shows a reference area 620 and a characteristic boundary 630 when the driver's state is a yawning state, corresponding to FIG. 6(b). 7(c) shows the reference area 620 and the feature boundary 630 when the driver's eyes are closed in correspondence with FIG. 6(c). 7(d) shows a reference area 620 and a characteristic boundary 630 when the driver's mouth is tightly closed in correspondence with FIG. 6(d).

In one example, the control unit 410 sets the upper side (a1, a2, a3, a4), the lower side (b1, b2, b3, b4) and the height (h1, h2, h3, h4) with respect to the feature boundary 630. Decide.

Referring to FIG. 5, in step 540, the control unit 410, for example, changes in heights (h1, h2, h3, h4), changes in lengths of the lower sides (b1, b2, b3, b4), and feature boundaries ( The current state of the driver may be determined based on at least one of a change in the area of 630 and a change in a ratio of the length to height h1, h2, h3, and h4 of the lower sides b1, b2, b3, and b4.

8 is a flowchart illustrating a process of determining a driver's state according to an example.

That is, FIG. 8 exemplarily shows a process of determining the driver's state in step 540 with reference to FIG. 5.

For example, in step 810, if the height h2 in the characteristic boundary 630 obtained in step 530 increases from the height h1 in the normal state, the control unit 410 returns to the yawning state in step 815. I can judge. This corresponds to the yawning state of Fig. 7(b).

For example, in step 820, the control unit 410 closes the eyes in step 825 when the height h3 in the feature boundary 630 obtained in step 530 decreases from the height h1 in the normal state. It can be judged as being present. This corresponds to the state in which the eyes are closed in Fig. 7(c).

For example, in step 830, if the length of the lower side b4 in the characteristic boundary 630 obtained in step 530 increases than the length of the lower side b1 in the normal state, the step ( 835) can be judged as the state of the mouth tight. This corresponds to the state in which the mouth is tightly closed in Fig. 7(d).

According to the driver's state recognition apparatus and method of the present invention, it is possible to distinguish the driver's state even from a distorted face image other than a front face image. In particular, since facial features are relatively concentrated in a distorted face image, the size of an image to be processed for driver state recognition is small, and since the number of pixels to be actually processed is small, processing speed is high.

The apparatus and method for recognizing driver status according to an embodiment of the present invention may be implemented in a computer system or recorded on a recording medium. The computer system may include at least one processor, memory, user input device, data communication bus, user output device, and storage. Each of the above-described components communicates data through a data communication bus.

The computer system may further include a network interface coupled to the network. The processor may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in a memory and/or storage.

The memory and storage may include various types of volatile or nonvolatile storage media. For example, the memory may include ROM and RAM.

The method for recognizing driver status according to an embodiment of the present invention may be implemented in a computer-executable method. When the method for recognizing a driver's state according to an embodiment of the present invention is performed in a computer device, commands readable by a computer may perform the method for recognizing a driver's state according to the present invention.

The driver state recognition method according to the present invention described above can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media in which data that can be decoded by a computer system is stored. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, and the like. In addition, the computer-readable recording medium can be distributed to a computer system connected through a computer communication network, and stored and executed as code that can be read in a distributed manner.

So far, the present invention has been looked at based on examples. Those of ordinary skill in the art to which the present invention pertains will appreciate that the present invention may be implemented in variously modified or modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an exemplary point of view for description and not a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: driver status recognition device
150: video acquisition device

Claims (15)

In the device for recognizing a driver's state based on a driver's face image taken with at least one camera,
A storage for storing the face image, and
Including a processor, the processor,
Acquire the driver's face image,
Extracting a reference region from the face image,
Extracting a feature boundary from the face image based on the reference region,
Driver state recognition device configured to determine the state of the driver based on a change in the characteristic boundary.
The method of claim 1.
The processor,
Driver condition recognition device further configured to determine a response instruction based on the driver's condition.
In the image collection device,
A user-wearable body; And
At least one LED camera cell disposed on the body to capture the user's face image
Image collection device comprising a.
The method of claim 3,
The body is an image collection device formed in a neckband type.
The method of claim 3,
The at least one LED camera cell is an image collection device for photographing the user's face in a direction looking up from the bottom to the top.
The method of claim 3,
The at least one LED camera cell is an image collection device for capturing a face image of a first viewpoint and a face image of a second viewpoint of the user.
The method of claim 3,
A communication module for transmitting the face image and receiving a response instruction for the user's state information determined based on the face image; And
An output unit that outputs a response to the user based on the response instruction
An image collection device further comprising a.
In a method of recognizing a driver's state based on a driver's face image taken with at least one camera,
Acquiring a driver's face image;
Extracting a reference region from the face image;
Extracting a feature boundary from the face image based on the reference region, and
Determining the state of the driver based on the change in the characteristic boundary
Driver state recognition method comprising a.
The method of claim 8,
The face image is an image of the driver's face in a direction looking up from the bottom to the top.
The method of claim 8,
The step of extracting the reference region,
A driver state recognition method for extracting an area of the face image corresponding to the driver's nose area as the reference area.
The method of claim 10,
The reference area is a driver state recognition method of a triangular shape.
The method of claim 8,
The step of extracting the feature boundary,
Extracting an ROI of the face image corresponding to an area including the driver's eyes and mouth, and
Determining the region of interest as the feature boundary based on a positional relationship between the reference region and the region of interest
Driver state recognition method comprising a.
The method of claim 12,
The characteristic boundary is a driver state recognition method having a reverse trapezoid shape.
The method of claim 8,
Acquiring the face image,
Acquiring a first face image of a first viewpoint, and
Acquiring a second face image of a second viewpoint
Including,
The step of determining the state of the driver,
A driver state recognition method for determining a state of the driver based on a change in a feature boundary between the first and second face images.
The method of claim 8,
The characteristic boundary is a historical trapezoidal shape,
The change of the characteristic boundary is at least one of a change in height of the inverse trapezoid shape, a change in length of a lower side, and a change in area.

Images