WO2020261832A1 - Image processing device, monitoring device, control system, image processing method, and program - Google Patents

Abstract

An image processing device that processes an image input from an imaging unit, said image processing device being equipped with a face detection unit for detecting a face region while scanning a search region of the image. The face detection unit is equipped with: a hierarchically structured ordinary face classifier that uses a feature amount extracted from the search region by a first feature amount extraction unit and an ordinary face feature amount stored in a face feature amount storage unit to determine whether the search region is the face of an ordinary person or is a non-face; and a specific individual face determination unit that, when the ordinary face classifier determines at any level that the search region is a non-face, uses the feature amount extracted from the search region and a face feature amount of the specific individual stored in the face feature amount storage unit to determine whether the search region is the face of the specific individual, or is a non-face.

Description

Image processing equipment, monitoring equipment, control systems, image processing methods, and programs

The present invention relates to an image processing device, a monitoring device, a control system, an image processing method, and a program.

Patent Document 1 below discloses a robot device used as a service providing device that can switch to an appropriate service according to the situation of a target (person) to which the service is provided.
The robot device is equipped with a first camera, a second camera, and an information processing device including a CPU, and the CPU includes a face detection unit, an attribute determination unit, a person detection unit, a person position calculation unit, and an information processing unit. It is equipped with a movement vector detector and the like.

According to the robot device, when the service is provided to a group of people who have a relationship such as communicating with each other, the first service of providing information based on close communication is performed. To determine. On the other hand, when the service is provided to a group of people whose relationship such as communication with each other is unknown, the second service provides information unilaterally without exchanging information. Decide to do. As a result, it is possible to provide appropriate services according to the situation of the service provision target.

Japanese Unexamined Patent Publication No. 2014-14899

In the robot device, the face detection unit is configured to detect a person's face using the first camera, and a known technique can be used for the face detection.
However, with conventional face detection technology, if a part of the facial organs such as eyes, nose, and mouth is missing or greatly deformed due to injury, a large mole, wart, or body decoration such as tattoo is applied to the face. Such specific individuals (in other words, age difference, gender, and person), such as when the facial organs are displaced from their average position due to treatment or a disease such as a hereditary disease. There is a problem that the accuracy of face detection for a specific individual (a specific individual) having characteristics different from those of a general person, which is common regardless of the difference in species, is lowered.

The present invention has been made in view of the above problems, and an image processing device, a monitoring device, a control system, an image processing method, and an image processing method capable of accurately detecting even the face of a specific individual as described above in real time. The purpose is to provide a program.
The image processing apparatus (1) according to the present disclosure in order to achieve the above object is an image processing apparatus that processes an image input from an imaging unit.
As the learned facial features that have been learned to detect the face from the image, a facial feature storage unit that stores the facial features of a specific individual and the normal facial features, and
A face detection unit that detects a face area while scanning a search area with respect to the image is provided.
The face detection unit
A first feature amount extraction unit that extracts facial feature amounts from the search area, and
A hierarchically structured normal face discriminator that discriminates whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount.
When it is determined that the face is non-face in any layer of the normal face discriminator, the search area is used by using the feature amount extracted from the search area and the face feature amount of the specific individual. Is provided with a specific individual face determination unit for determining whether the face is the face of the specific individual or the non-face.

According to the image processing device (1), the normal face discriminator hierarchically discriminates whether the search area is a face or a non-face by using the normal face feature amount. , The face area is detected. Further, even when it is determined that the face is non-faced in any layer of the normal face discriminator, the specific individual face determination unit uses the face feature amount of the specific individual to search the search area. By determining whether is the face of the specific individual or non-face, the face region including the face of the specific individual is detected. Thereby, the face region can be detected with high accuracy regardless of whether it is the normal face or the face of the specific individual. Further, since the normal face discriminator and the specific individual face determination unit use the common feature amount extracted from the search region, the real-time property related to the detection of the face region can be maintained.

The image processing device (2) according to the present disclosure is the above-mentioned image processing device (1) used in one layer of the normal face discriminator that the specific individual face determination unit has determined to be non-face. An index showing the correlation between the feature amount and the face feature amount of the specific individual corresponding to the one layer is calculated, and based on the calculated index, whether the search area is the face of the specific individual or not. It is characterized in that it determines whether or not it is non-faced.

According to the image processing device (2), the feature amount used in one layer of the normal face discriminator determined to be non-face and the face feature amount of the specific individual corresponding to the one layer. Based on the index showing the correlation with, it is possible to efficiently determine whether the search area is the face or non-face of the specific individual, and the normal face discriminator determines that the search area is non-face. Even if it is done, it is possible to accurately determine the case where the face is the specific individual. The index may be an index value indicating that the larger the value is, the higher the relationship is, for example, a correlation coefficient, the reciprocal of the square error, or the non-face. It may be an index value indicating the degree of similarity between the feature amount used in one layer of the normal face discriminator determined to be, and the face feature amount of the specific individual corresponding to the one layer. ..

In the image processing device (3) according to the present disclosure, in the image processing device (2), when the specific individual face determination unit indicates that the index is larger than a predetermined threshold value, the search area is the face of the specific individual. When the index is equal to or less than the predetermined threshold value, it is determined that the search region is the non-face.
According to the image processing device (3), when the index is larger than a predetermined threshold value, it is determined that the search area is the face of the specific individual, and when the index is equal to or less than the predetermined threshold value, the search area is determined. Is determined to be the non-face. The processing efficiency of the determination can be improved by the processing of comparing the index with the predetermined threshold value.

When the image processing device (4) according to the present disclosure is determined by the specific individual face determination unit to be the face of the specific individual in any of the image processing devices (1) to (3), the usual A discrimination progressing unit that advances discrimination to the next level of the face discriminator,
When the specific individual face determination unit determines that the face is non-face, the specific individual face determination unit is provided with a discrimination cutoff unit that terminates the discrimination by the normal face discriminator.

According to the image processing device (4), when the specific individual face determination unit determines that the face is the specific individual's face, the determination proceeds to the next layer of the normal face discriminator, and the discrimination process is expedited. Continued to. On the other hand, when the specific individual face determination unit determines that the face is non-face, the determination by the normal face discriminator is terminated. Therefore, it is possible to perform the process of determining the face of the specific individual while maintaining the efficiency of the normal face discriminator.

The image processing device (5) according to the present disclosure is one or more of the image processing devices (1) to (4) in which the face detection unit is determined to be the face by the normal face discriminator. The face region integration unit that integrates the candidates for the face region and the second feature quantity extraction unit that extracts the facial feature quantity from the integrated face region are provided, and the face region is extracted from the integrated face region. It is characterized by including a specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual by using the feature amount and the face feature amount of the specific individual.

According to the image processing apparatus (5), one or more candidates for the face region determined to be the face are integrated, the feature amount extracted from the integrated face region, and the specific individual. Using the face feature amount, it is determined whether or not the face in the face region is the face of the specific individual. Therefore, it is possible to accurately determine whether the face region integrated by the face region integrating unit is the face of the specific individual or the face of a normal person.

The monitoring device (1) according to the present disclosure is based on any of the above image processing devices (1) to (5), an image pickup unit that captures an image input to the image processing device, and image processing by the image processing device. It is characterized by having an output unit that outputs information.
According to the monitoring device (1), not only the face of the normal person but also the face of the specific individual can be accurately detected and monitored, and information based on the image processing can be detected from the output unit. Can be output, so it is possible to easily construct a monitoring system or the like that uses the information.

The control system (1) according to the present disclosure is one or more that is communicably connected to the monitoring device (1) and executes a predetermined process based on the information output from the monitoring device. It is characterized by being equipped with a control device of.
According to the control system (1), it is possible to execute a predetermined process by one or more of the control devices based on the information output from the monitoring device. Therefore, it is possible to construct a system that can utilize not only the monitoring result of the normal person but also the monitoring result of the specific individual.

In the control system (1), the control system (2) according to the present disclosure is a device for monitoring the driver of the vehicle, and the control device is mounted on the vehicle. It is characterized by including an electronic control unit.
According to the control system (2), even when the driver of the vehicle is the specific individual, the face of the specific individual can be accurately monitored, and the electronic device is based on the monitoring result. It is possible to make the control unit appropriately execute a predetermined control. This makes it possible to construct a highly safe in-vehicle system that allows even the specific individual to drive with peace of mind.

The image processing method according to the present disclosure is an image processing method for processing an image input from an imaging unit.
A face detection step of detecting a face area while scanning the search area with respect to the image is included.
The face detection step
A feature amount extraction step for extracting facial feature amounts from the search area, and
The search area is a face or a non-face using the feature amount extracted by the feature amount extraction step and a trained normal face feature amount that has been trained to detect a face. A normal face discrimination step that hierarchically discriminates whether or not
When it is determined that the face is non-face in any of the layers of the normal face discrimination step, the extracted feature amount and the learned specific individual who has been trained to detect the face of the specific individual. It is characterized by including a specific individual face determination step of determining whether the search area is the face of the specific individual or the non-face by using the face feature amount of the above.

According to the image processing method, the face is determined by hierarchically determining whether the search area is a face or a non-face by using the normal face feature amount in the normal face discrimination step. The area is detected. Further, even when it is determined that the face is non-face in any layer of the normal face determination step, the search area is used by the specific individual face determination step using the face feature amount of the specific individual. By determining whether is the face of the specific individual or non-face, the face region including the face of the specific individual is detected. Thereby, the face region can be detected with high accuracy regardless of whether it is the normal face or the face of the specific individual. Further, since the common feature amount extracted from the search area is used in the normal face determination step and the specific individual face determination step, the real-time property related to the detection of the face area can be maintained. Therefore, the face of the specific individual can be detected accurately in real time.

The program according to the present disclosure is a program for causing at least one or more computers to process an image input from an imaging unit.
To at least one of the above computers
A face detection step of detecting a face area while scanning the search area with respect to the image is included.
The face detection step
A feature amount extraction step for extracting facial feature amounts from the search area, and
The search area is a face or a non-face using the feature amount extracted by the feature amount extraction step and a trained normal face feature amount that has been trained to detect a face. A normal face discrimination step that hierarchically discriminates whether or not
When it is determined that the face is non-face in any of the layers of the normal face discrimination step, the extracted feature amount and the learned specific individual who has been trained to detect the face of the specific individual. It is a program for executing a specific individual face determination step for determining whether the search area is the face of the specific individual or the non-face by using the face feature amount of the above. ..

According to the above program, the search area is hierarchically determined whether the search area is a face or a non-face by using the normal face feature amount on the at least one computer in the normal face discrimination step. It is possible to detect the face region. Further, even when it is determined that the face is non-face in any layer of the normal face determination step, the search area is used by the specific individual face determination step using the face feature amount of the specific individual. It is possible to detect the face region including the face of the specific individual by determining whether the face is the face of the specific individual or not. As a result, the face region can be accurately detected regardless of whether it is the normal face or the face of the specific individual. Further, since the common feature amount extracted from the search area is used in the normal face determination step and the specific individual face determination step, the real-time property related to the detection of the face area can be maintained. Therefore, it is possible to construct a device or system capable of accurately sensing each face, whether it is the specific individual or an ordinary person other than the specific individual. The above program may be a program stored in a storage medium, a program that can be transferred via a communication network, or a program that is executed via a communication network. ..

It is a schematic diagram which shows an example of the in-vehicle system including the driver monitoring apparatus which concerns on embodiment of this invention. It is a block diagram which shows an example of the hardware composition of the in-vehicle system including the driver monitoring apparatus which concerns on embodiment. It is a block diagram which shows the functional structure example of the image processing part of the driver monitoring apparatus which concerns on embodiment. It is a block diagram which shows the functional structure example of the face detection part included in the image processing part. It is a schematic diagram for demonstrating the processing operation example performed in the face detection part. It is a schematic diagram for demonstrating the processing operation example performed in the face detection part. It is a flowchart which shows an example of the processing operation performed by the image processing part of the driver monitoring apparatus which concerns on embodiment. It is a flowchart which shows an example of the face detection processing operation and the specific individual judgment processing operation performed by the image processing unit of the driver monitoring apparatus which concerns on embodiment. It is a flowchart which shows an example of the discrimination processing operation performed by the image processing part of the driver monitoring apparatus which concerns on embodiment.

Hereinafter, an image processing device, a monitoring device, a control system, an image processing method, and an embodiment of a program according to the present invention will be described with reference to the drawings.
The image processing apparatus according to the present invention can be widely applied to, for example, an apparatus or system for monitoring an object such as a person using a camera. The image processing device according to the present invention operates or monitors, for example, various facilities such as machines and devices in a factory, in addition to devices and systems for monitoring drivers (operators) of various moving objects such as vehicles. It can also be applied to devices and systems that monitor people who perform predetermined work.

[Application example]
FIG. 1 is a schematic view showing an example of an in-vehicle system including the driver monitoring device according to the embodiment. In this application example, an example in which the image processing apparatus according to the present invention is applied to the driver monitoring apparatus 10 will be described.
The in-vehicle system 1 includes a driver monitoring device 10 that monitors the state of the driver 3 of the vehicle 2 (for example, facial behavior), and one or more ECUs (Electronic Control Units) that control the running, steering, or braking of the vehicle 2. ) 40, and one or more sensors 41 for detecting the state of each part of the vehicle, the state around the vehicle, and the like are included, and these are connected via the communication bus 43. The in-vehicle system 1 is configured as, for example, an in-vehicle network system that communicates according to a CAN (Controller Area Network) protocol. As the communication standard of the in-vehicle system 1, a communication standard other than CAN may be adopted. The driver monitoring device 10 is an example of the "monitoring device" of the present invention, and the in-vehicle system 1 is an example of the "control system" of the present invention.

The driver monitoring device 10 transmits information based on image processing by the camera 11 for capturing the face of the driver 3, the image processing unit 12 that processes the image input from the camera 11, and the image processing unit 12, and the communication bus 43. It is configured to include a communication unit 16 that performs processing such as output to a predetermined ECU 40 via the above. The image processing unit 12 is an example of the "image processing device" of the present invention. The camera 11 is an example of the "imaging unit" of the present invention.

The driver monitoring device 10 detects the face of the driver 3 from the image captured by the camera 11, and detects the behavior of the face such as the direction of the face of the detected driver 3, the direction of the line of sight, or the open / closed state of the eyes. The driver monitoring device 10 may determine the state of the driver 3, such as forward gaze, inattentiveness, dozing, backward facing, and prone, based on the detection results of these facial behaviors. Further, the driver monitoring device 10 outputs a signal based on the state determination of the driver 3 to the ECU 40, and the ECU 40 performs attention and warning processing to the driver 3 or operation control of the vehicle 2 (for example, deceleration) based on the signal. Control, guidance control to the road shoulder, etc.) may be executed.

In the driver monitoring device 10, face sensing for the driver 3, particularly whether the driver 3 is a specific individual or a normal person other than the specific individual, can accurately detect the faces of these subjects in real time. One of the purposes is to do.
In the conventional driver monitoring device, the driver 3 of the vehicle 2 has a part of facial organs such as eyes, nose, and mouth missing or greatly deformed due to, for example, an injury, or a large mole or wart on the face, or Accuracy of detecting the face from the image captured by the camera when the facial organs are displaced from the average position due to body decoration such as tattoo or a disease such as a hereditary disease. There was a problem that the

Further, if the face detection accuracy is lowered, the post-face detection processing such as the face orientation estimation processing is not properly performed, so that the driver 3 cannot properly perform the state determination such as inattentiveness or dozing. Further, there is a problem that various controls to be executed by the ECU 40 based on the state determination may not be appropriately performed.
In order to solve the problem, the driver monitoring device 10 according to the embodiment is a general person who is common regardless of a specific individual, in other words, a difference in age, gender, race, etc. (individual difference). The following configuration was adopted in order to enable real-time and accurate face detection for a specific individual who has features different from those of a normal person (also called a normal person).

In the image processing unit 12, as the learned facial features that have been learned to detect the face from the image, the facial features of a specific individual and the normal facial features (in other words, the face of a normal person) The amount of facial features used for detection) is stored.
The image processing unit 12 performs face detection processing for detecting a face area while scanning a search area of a predetermined size with respect to an input image of the camera 11 and extracting a feature amount for detecting a face from the search area. Do. Then, the image processing unit 12 hierarchically determines whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount. The face area is detected from the input image by the normal face discrimination process.

In addition, when the image processing unit 12 determines that the face is non-face in any layer of the normal face discrimination process, the feature amount extracted from the search area and the face feature amount of the specific individual Using and, a specific individual face determination for determining whether the search area is a face of a specific individual or a non-face is performed.
In the specific individual face determination process, the relationship between the feature amount used in one layer determined to be non-face in the normal face determination process and the face feature amount of the specific individual corresponding to the one layer is determined. An index to be shown, for example, a correlation coefficient may be calculated, and based on the calculated correlation coefficient, it may be determined whether the search area is a face or a non-face of the specific individual.

For example, when the correlation coefficient is larger than a predetermined threshold value, it may be determined that the search area is the face of the specific individual, and the discrimination process may proceed to the next layer in the normal face discrimination process. Further, when the correlation coefficient is equal to or less than the predetermined threshold value, it may be determined that the search area is the non-face, and the normal face discrimination process for the search area may be terminated. In the specific individual face determination process, an index other than the correlation coefficient may be used.

In this way, in the driver monitoring device 10, the normal face discrimination process is used to hierarchically determine whether the search region is a face or a non-face using the normal face feature amount, and the face region is determined. Is detected. Further, even when it is determined that the face is non-faced in any layer of the normal face determination process, the search area is used by the specific individual face determination process using the face feature amount of the specific individual. The face region including the face of the specific individual is detected by determining whether is the face of the specific individual or a non-face.

Further, when it is determined that the face is non-face in any layer of the normal face discrimination process and the specific individual face determination process determines that the face is non-face, the search area is defined as non-face. The process is terminated as (other than the face), and the process proceeds to the next search area.
With these configurations, it is possible to accurately detect the face region regardless of whether it is the normal face or the face of the specific individual. Further, since the normal face discrimination process and the specific individual face determination process use the common feature amount extracted from the search area, it is possible to maintain the real-time property related to the detection of the face area. ..
Therefore, the driver 3 can accurately detect each face in real time (in other words, by high-speed processing) regardless of whether the driver 3 is a specific individual or an ordinary person other than the specific individual. ..

[Hardware configuration example]
FIG. 2 is a block diagram showing an example of the hardware configuration of the in-vehicle system 1 including the driver monitoring device 10 according to the embodiment.

The in-vehicle system 1 includes a driver monitoring device 10, 1 or more ECUs 40 for monitoring the state of the driver 3 of the vehicle 2, and 1 or more sensors 41, which are connected via a communication bus 43. Further, one or more actuators 42 are connected to the ECU 40.
The driver monitoring device 10 includes a camera 11, an image processing unit 12 that processes an image input from the camera 11, and a communication unit 16 for exchanging data and signals with an external ECU 40 and the like. There is.

The camera 11 is a device that captures an image including the face of the driver 3 seated in the driver's seat. For example, a lens unit, an image sensor unit, a light irradiation unit, an interface unit, a camera control unit that controls each of these units, and the like. Can be configured to include. The image sensor unit may include an image sensor such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor), a filter, a microlens, and the like. The image pickup device unit may be an element capable of forming an image pickup image by receiving light in a visible region, or an element capable of forming an image pickup image by receiving light in a near infrared region. The light irradiation unit is configured to include a light emitting element such as an LED (Light Emitting Diode), and may include a near infrared LED or the like so that the driver 3's face can be imaged day or night. The camera 11 captures an image at a predetermined frame rate (for example, several tens of frames per second), and the data of the captured image is input to the image processing unit 12. The camera 11 may be an external type as well as an integrated type.

The image processing unit 12 is configured as an image processing device including one or more CPU (Central Processing Unit) 13, ROM (Read Only Memory) 14, and RAM (Random Access Memory) 15. The ROM 14 includes a program storage unit 141 and a facial feature amount storage unit 142, and the RAM 15 includes an image memory 151 for storing an input image from the camera 11. The driver monitoring device 10 may be further equipped with another storage unit, and the other storage unit may be used as the program storage unit 141, the facial feature amount storage unit 142, and the image memory 151. The other storage unit may be a semiconductor memory or a storage medium that can be read by a disk drive or the like.

The CPU 13 is an example of a hardware processor, and by reading, interpreting, and executing data such as a program stored in the program storage unit 141 of the ROM 14 and the face feature amount stored in the face feature amount storage unit 142. , Processing of the image input from the camera 11, for example, face image processing such as face detection processing is performed. Further, the CPU 13 performs a process of outputting the result (for example, processing data, determination signal, control signal, etc.) obtained by the face image processing to the ECU 40 or the like via the communication unit 16.

The face feature amount storage unit 142 contains the face feature amount 142a of a specific individual and the normal face feature amount 142b as learned face feature amounts that have been learned (for example, machine learning) to detect a face from an image. And (see FIGS. 3 and 4) are stored.
As the learned facial features, various feature quantities effective for detecting a face from an image can be used. For example, a feature amount (Haar-like feature amount) focusing on the difference in brightness (difference in average brightness) of a local region of the face may be used. Alternatively, a feature amount (LBP (Local Binary Pattern) feature amount) focusing on a combination of brightness distributions in the local region of the face may be used, or the distribution of the brightness in the local region of the face in the gradient direction may be used. Features (HOG (Histogram of Oriented Gradients) features) focusing on the combination may be used.

The face feature amount stored in the face feature amount storage unit 142 is extracted as an effective feature amount for face detection by using, for example, various machine learning methods. Machine learning is a process of finding a pattern inherent in data (learning data) by a computer. For example, AdaBoost may be used as an example of a statistical learning method. AdaBoost selects a large number of discriminators (weak discriminators) with low discriminating ability, selects a weak discriminator with a small error rate from these many weak discriminators, adjusts parameters such as weights, and has a hierarchical structure. It is a learning algorithm that can construct a strong discriminator by setting. The discriminator may be referred to as a discriminator, a classifier, or a learner.

For example, the strong discriminator is configured to discriminate one feature amount effective for face detection by one weak discriminator, and a large number of weak discriminators and their combinations are selected by AdaBoost, and these are used hierarchically. The structure may be constructed. Note that one weak discriminator may output information such as 1 for a face and 0 for a non-face. Further, as the learning method, a learning method called Real AdaBoost, which can output a real number from 0 to 1 instead of 0 or 1, may be used. Further, as these learning methods, a neural network having an input layer, an intermediate layer, and an output layer may be used.

A large number of face images captured under various conditions and a large number of non-face images (non-face images) are given as training data to a learning device equipped with such a learning algorithm, learning is repeated, weighting, etc. By adjusting the parameters and optimizing it, it is possible to construct a strong discriminator having a hierarchical structure capable of detecting a face with high accuracy. Then, one or more feature amounts used in the weak discriminators of each layer constituting such a strong discriminator can be used as the learned facial feature amounts.

For example, the face feature amount 142a of a specific individual individually captures a face image of the specific individual at a predetermined place under various conditions (conditions such as various face orientations, line-of-sight directions, or eye open / closed states). Then, these a large number of captured images are input to the learning device as teacher data, and are parameters that indicate the facial features of a specific individual adjusted by the learning process. The facial feature amount 142a of the specific individual may be, for example, a combination pattern of the difference in brightness of the local region of the face obtained by the learning process. The facial feature amount 142a of a specific individual stored in the facial feature amount storage unit 142 may be only the facial feature amount of one specific individual, or can be used when a plurality of specific individuals drive the vehicle 2. , The facial features of a plurality of specific individuals may be stored.

The normal facial feature amount 142b is the above-mentioned learning device using images of a normal human face image captured under various conditions (conditions such as various face orientations, line-of-sight directions, or eye open / closed states) as teacher data. It is a parameter indicating the characteristics of a normal human face, which is input to and adjusted by the learning process. The normal facial feature amount 142b may be, for example, a combination pattern of light and dark differences in a local region of the face obtained by a learning process. Further, as the normal facial feature amount 142b, the information registered in the predetermined facial feature amount database may be used.

The learned facial feature amount stored in the facial feature amount storage unit 142 is fetched from a server on the cloud or the like via a communication network such as the Internet or a mobile phone network and stored in the facial feature amount storage unit 142. It may be configured as such.
The ECU 40 is composed of a computer device including one or more processors, a memory, a communication module, and the like. Then, the processor mounted on the ECU 40 reads, interprets, and executes the program stored in the memory, so that predetermined control for the actuator 42 and the like is executed.

The ECU 40 includes, for example, at least one of a traveling system ECU, a driving support system ECU, a body system ECU, and an information system ECU.
The traveling system ECU includes, for example, a drive system ECU, a chassis system ECU, and the like. The drive system ECU includes a control unit related to a "running" function such as engine control, motor control, fuel cell control, EV (Electric Vehicle) control, or transmission control. The chassis-based ECU includes a control unit related to a "stop, turn" function such as brake control or steering control.

The driving support system ECU has, for example, an automatic braking support function, a lane keeping support function (also referred to as LKA / Lane Keep Assist), a constant speed driving / inter-vehicle distance support function (also referred to as ACC / Adaptive Cruise Control), and a forward collision warning function. , Lane departure warning function, blind spot monitoring function, traffic sign recognition function, etc., functions that automatically improve safety or realize comfortable driving by linking with driving ECUs (driving support function or automatic driving function) It may be configured to include at least one control unit with respect to.

The driving support system ECU includes, for example, Level 1 (driver assistance), Level 2 (partially automatic driving), and Level 3 (conditional automatic driving) at the automatic driving level presented by the American Society of Automotive Engineers of Japan (SAE). ) May be equipped with at least one of the functions. Further, the functions of level 4 (highly automatic driving) or level 5 (fully automatic driving) of the automatic driving level may be equipped, and only the functions of level 1 and 2 or only level 2 and 3 are equipped. May be good. Further, the in-vehicle system 1 may be configured as an automatic driving system.

The body system ECU may be configured to include at least one control unit related to the function of the vehicle body such as a door lock, a smart key, a power window, an air conditioner, a light, an instrument panel, or a winker.
The information system ECU may be configured to include, for example, an infotainment device, a telematics device, or an ITS (Intelligent Transport Systems) related device. The infotainment device may include, for example, an HMI (Human Machine Interface) device that functions as a user interface, a car navigation device, an audio device, and the like. The telematics device may include a communication unit or the like for communicating with the outside. The ITS-related device may include an ETC (Electronic Toll Collection System), a communication unit for performing road-to-vehicle communication with a roadside machine such as an ITS spot, or vehicle-to-vehicle communication.

The sensor 41 may include various in-vehicle sensors that acquire sensing data necessary for controlling the operation of the actuator 42 by the ECU 40. For example, in addition to vehicle speed sensors, shift position sensors, accelerator opening sensors, brake pedal sensors, steering sensors, etc., peripheral monitoring of external imaging cameras, millimeter-wave radar (Radar), riders (LIDER), ultrasonic sensors, etc. A sensor or the like may be included.

The actuator 42 is a device that executes operations related to traveling, steering, braking, etc. of the vehicle 2 based on a control signal from the ECU 40, and includes, for example, an engine, a motor, a transmission, a hydraulic cylinder, an electric cylinder, and the like.

[Functional configuration example]
FIG. 3 is a block diagram showing a functional configuration example of the image processing unit 12 of the driver monitoring device 10 according to the embodiment.
The image processing unit 12 includes an image input unit 21, a face detection unit 22, a specific individual determination unit 25, a first face image processing unit 26, a second face image processing unit 30, an output unit 34, and a face feature amount storage unit 142. It is configured to include.
The image input unit 21 performs a process of capturing an image including the face of the driver 3 captured by the camera 11.

The face detection unit 22 performs a process of detecting the face area while scanning the search area of a predetermined size with respect to the input image and extracting the feature amount of the face from the search area. The face detection unit 22 includes a first feature amount extraction unit 221, a normal face discriminator 222, and a specific individual face determination unit 223. The face detection unit 22 may further include a face region integration unit 224 and a second feature amount extraction unit 225.

FIG. 4 is a block diagram showing a functional configuration example of the face detection unit 22.
5 and 6 are schematic views for explaining an example of processing operation performed by the face detection unit 22.
In the present embodiment, the face detection unit 22 uses, for example, Haar-like features as image features, and has a hierarchical structure normal face discriminator 222 and a normal face discriminator constructed by using the learning algorithm of AdaBoost. It is configured to use the specific individual face determination unit 223 added to 222.

The Haar-like feature is also called a rectangular feature, for example, the feature amount is the difference in the average brightness of the two rectangular areas. For example, the eye area in the image has low brightness and is around the eye (under the eye). , Next to the eyes) utilizes the feature that the brightness is high. As the Haar-like feature, a rectangular feature that is a combination of two, three, or four rectangles may be used. Features (highly important) effective for face detection and their combinations are selected using a learning algorithm and stored in the face feature storage unit 142. The face feature amount storage unit 142 stores a normal face feature amount 142b used for processing by the normal face discriminator 222 and a specific individual face feature amount 142a used for processing by the specific individual face determination unit 223. Has been done.

As shown in FIG. 4, the normal face discriminator 222 includes the first discriminator 222a to the Nth discriminator 222n, and has a hierarchical structure (also referred to as a cascade structure) in which a plurality of these are connected. Each of these discriminators uses one or more feature quantities 221a extracted from a search area 210 of a predetermined size cut out from an image and is effective for face detection, and the search area 210 is a face or a non-face (face). Other than). In the first discriminator of the hierarchical structure such as the first discriminator 222a, a feature amount 221a that roughly captures the face, such as whether or not there are eyes, is used. Further, in a discriminator having a deeper hierarchical structure such as the Nth discriminator 222n, for example, whether there are eyes, nose, mouth, front face, diagonal face, or profile. , A feature amount 221a that captures the details of the face is used.

The first feature amount extraction unit 221 extracts from the search area 210 one or more feature amounts 221a that are usually set to be discriminated by each discriminator constituting the face discriminator 222.
The normal face discriminator 222 uses the feature amount 221a extracted from the search area 210 by the first feature amount extraction unit 221 and the normal face feature amount 142b, and the search area 210 is face or non-face. The presence or absence is hierarchically determined in the order of the first discriminator 222a to the Nth discriminator 222n.

As shown in FIG. 4, the specific individual face determination unit 223 includes the first determination unit 223a to the Nth determination unit 223n, and determines that the search area 210 is non-face in any layer of the normal face determination device 222. When this is done, the feature amount 221a extracted from the search area 210 and the face feature amount 142a of the specific individual are used to determine whether the search area 210 is the face of the specific individual or not.

For example, when the search area 210 is determined to be non-face by the second discriminator 222b, the second determination unit 223b extracts the feature amount 221a (used by the second discriminator 222b) from the search region 210. And the face feature amount 142a of the specific individual corresponding to the hierarchy are used to determine whether the search area 210 is a face of the specific individual or a non-face.
When the second determination unit 223b determines that the search area 210 is the face of a specific individual, the process proceeds to the determination process by the third discriminator 222c, while when it is determined that the search area 210 is a non-face, the second determination unit 210 The discrimination process is terminated by the discriminator 222b, and the process proceeds to the face detection process for the next search area.

Then, when the discrimination process proceeds and the Nth discriminator 222n determines that the search area 210 is a face, or the Nth determination unit 223n determines that the search area 210 is a face of a specific individual. The search area 210 is stored as a candidate for the face area.
The face detection unit 22 generates reduced images 20a and 20b obtained by reducing the input image 20 by a plurality of magnifications, as shown in FIG. 5, for example, in order to detect faces of various sizes. A search area 210 of a predetermined size may be cut out from the reduced images 20a and 20b of the above, and a normal face detector 222 may be used to determine whether the search area 210 is a face or a non-face. Then, by scanning the search area 210 in the input image 20, the reduced images 20a, and 20b, faces of various sizes in the image 20 and the positions of the faces may be detected. The search area 210 may have any shape other than a rectangle.

Further, the face detection unit 22 may be configured to detect a face facing (rotated) in various directions or a face tilted at various angles. For example, the first feature amount extraction unit 221 extracts a feature amount 221a effective for discriminating the face orientation and the face inclination from the search area 210, and the normal face discriminator 222 usually uses the face orientation and the face inclination. The search area 210 may be configured to be able to discriminate whether it is a face or a non-face by using a trained discriminator that has learned about each feature amount.

For example, as shown in FIG. 6, in order to detect the front face, the diagonal face, and the profile, respectively, the normal face discriminator 222 normally uses the front (0 degree), left diagonal (45 degrees), and left side (90 degrees). ) May be provided with a discriminator that has learned each feature amount. In this case, learning may be performed so that one discriminator can cover a predetermined angle (for example, 22.5 degrees) or more. Further, the right angle (45 degrees) can be determined by flipping the left diagonal (45 degrees) horizontally, and the right side (90 degrees) can be determined by flipping the left side (90 degrees) horizontally. Good. Further, as shown in FIG. 6, the normal face discriminator 222 may be provided with a discriminator that learns each feature amount for each predetermined tilt so that the tilt of the face can be detected.

The face region integration unit 224 performs a process of integrating one or more face region candidates determined to be a face by the normal face discriminator 222. The method of integrating the candidates of one or more face regions is not particularly limited. For example, it may be integrated based on the average value of the region center of one or more face region candidates and the average value of the region size.
The second feature amount extraction unit 225 performs a process of extracting facial feature amounts from the face area integrated by the face area integration unit 224.

The specific individual determination unit 25 uses the feature amount of the face area detected by the face detection unit 22 and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142 to detect the face in the face area. Performs a process of determining whether is the face of a specific individual or the face of a normal person other than the specific individual.
The specific individual determination unit 25 calculates a correlation coefficient as an index showing the relationship between the feature amount extracted from the face region and the face feature amount 142a of the specific individual, for example, an index showing the correlation, and the calculated correlation coefficient. It may be determined whether or not the face in the face region is the face of a specific individual based on. Then, when the correlation coefficient is larger than a predetermined threshold value, it is determined that the face in the detected face area is the face of a specific individual, and when the correlation coefficient is equal to or less than the predetermined threshold value, the face in the detected face area is a specific individual. It may be determined that it is not the face of.

Further, the specific individual determination unit 25 may determine whether or not the face in the detected face region is the face of the specific individual based on the result of determination for one frame of the input image from the camera 11. Based on the result of determination for a plurality of frames of the input image from the camera 11, it may be determined whether or not the face in the detected face region is the face of a specific individual.
When the specific individual determination unit 25 determines that the face is a specific individual's face, the first face image processing unit 26 performs face image processing for the specific individual. The first face image processing unit 26 includes a face orientation estimation unit 27 of a specific individual, an eye opening / closing detection unit 28 of the specific individual, and a line-of-sight direction estimation unit 29 of the specific individual, but is still different. It may include a configuration for estimating or detecting facial behavior. In addition, the first face image processing unit 26 may perform any processing of the face image processing for the specific individual by using the face feature amount 142a of the specific individual. Further, the face feature amount storage unit 142 stores the learned feature amount that has been learned to perform the face image processing for the specific individual, and uses the learned feature amount for the specific individual. Any processing of face image processing may be performed.

The face orientation estimation unit 27 of the specific individual performs a process of estimating the face orientation of the specific individual. The face orientation estimation unit 27 of a specific individual detects, for example, the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected by the face detection unit 22, and the detected position and shape of the facial organs. The process of estimating the orientation of the face is performed based on.
The method for detecting the facial organs from the facial region in the image is not particularly limited, but it is preferable to adopt a method capable of detecting the facial organs at high speed and with high accuracy. For example, a method of creating a three-dimensional face shape model, fitting it to a face region on a two-dimensional image, and detecting the position and shape of each organ of the face can be adopted. As a technique for fitting a three-dimensional face shape model to a human face in an image, for example, the technique described in Japanese Patent Application Laid-Open No. 2007-249280 can be applied, but the technique is not limited thereto.

Further, the face orientation estimation unit 27 of the specific individual can use the estimation data of the face orientation of the specific individual, for example, the pitch angle of vertical rotation (around the X axis) included in the parameters of the three-dimensional face shape model. The yaw angle of the left-right rotation (around the Y axis) and the roll angle of the entire rotation (around the Z axis) may be output.
The eye opening / closing detection unit 28 of the specific individual performs a process of detecting the opening / closing state of the eyes of the specific individual. The eye opening / closing detection unit 28 of the specific individual, for example, is based on the position and shape of the facial organs obtained by the face orientation estimation unit 27 of the specific individual, particularly the position and shape of the feature points (eyelids, pupils) of the eyes. Detects the open / closed state, for example, whether the eyes are open or closed. For the open / closed state of the eye, for example, the feature amount of the image of the eye (the position of the eyelid, the shape of the pupil (black eye), the area size of the white eye part and the black eye part, etc.) in various open / closed states of the eye is previously learned. It may be detected by learning using the data and evaluating the degree of similarity with the learned feature data.

The line-of-sight direction estimation unit 29 of the specific individual performs a process of estimating the line-of-sight direction of the specific individual. The line-of-sight direction estimation unit 29 of a specific individual is based on, for example, the orientation of the face of the driver 3 and the position and shape of the facial organs of the driver 3, particularly the position and shape of the feature points of the eyes (outer corners of eyes, inner corners of eyes, pupils). Estimate the direction of the line of sight. The direction of the line of sight is the direction in which the driver 3 is looking, and is determined by, for example, a combination of the direction of the face and the direction of the eyes.

In addition, the direction of the line of sight is, for example, the feature amount of the image of the eye in various combinations of face orientation and eye orientation (relative position of outer corner, inner corner of eye, pupil, relative position of white eye portion and black eye portion, shading, etc. (Texture, etc.) may be detected by learning in advance using a learning device and evaluating the degree of similarity with the learned feature amount data. In addition, the line-of-sight direction estimation unit 29 of the specific individual estimates the size and center position of the eyeball from the size and orientation of the face, the position of the eyes, etc., using the fitting result of the three-dimensional face shape model, and the pupil. The position of the eyeball may be detected, and the vector connecting the center of the eyeball and the center of the pupil may be detected as the line-of-sight direction.

When the specific individual determination unit 25 determines that the face is not the face of a specific individual, the second face image processing unit 30 performs normal face image processing. The second face image processing unit 30 includes a normal face orientation estimation unit 31, a normal eye opening / closing detection unit 32, and a normal line-of-sight direction estimation unit 33, but has yet another face behavior. It may include a configuration for estimating or detecting. In addition, the second face image processing unit 30 may perform any processing of the normal face image processing using the normal face feature amount 142b. In addition, the face feature amount storage unit 142 stores the learned feature amount that has been learned for performing the normal face image processing, and uses the learned feature amount to perform the normal face image processing. Either process may be performed. The processing performed by the normal face orientation estimation unit 31, the normal eye opening / closing detection unit 32, and the normal line-of-sight direction estimation unit 33 includes the face orientation estimation unit 27 of the specific individual and the eye opening / closing detection of the specific individual. Since it is basically the same as the unit 28 and the line-of-sight direction estimation unit 29 of a specific individual, the description thereof will be omitted here.

The output unit 34 performs a process of outputting information based on the image processing by the image processing unit 12 to the ECU 40 or the like. The information based on the image processing may be, for example, information on the behavior of the face such as the direction of the face of the driver 3, the direction of the line of sight, or the open / closed state of the eyes, or the driver 3 determined based on the detection result of the behavior of the face. Information on the state of (for example, forward gaze, inattentiveness, dozing, backward facing, prone, etc.) may be used. Further, the information based on the image processing may be a predetermined control signal (control signal for performing caution or warning processing, control signal for performing operation control of the vehicle 2, etc.) based on the state determination of the driver 3.

[Processing operation example]
FIG. 7 is a flowchart showing an example of a processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. For example, the camera 11 captures an image of several tens of frames per second, and this processing is performed for each frame or every frame at regular intervals.
First, in step S1, the CPU 13 operates as an image input unit 21, performs a process of reading an image (an image including the face of the driver 3) captured by the camera 11, and proceeds to step S2.
In step S2, the CPU 13 operates as the face detection unit 22, performs face detection processing for detecting the face area while scanning the search area for the input image, and proceeds to step S3. A specific example of the face detection process in step S2 will be described later.

In step S3, the CPU 13 operates as the specific individual determination unit 25, and uses the feature amount of the face region detected in step S2 and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142. A process of determining whether or not the face in the detected face area is a face of a specific individual is performed, and the process proceeds to step S4.
In step S4, the CPU 13 determines whether or not the result of the determination process in step S3 is the face of a specific individual, and if it is determined that the result is the face of a specific individual, the process proceeds to step S5.

In step S5, the CPU 13 operates as a face orientation estimation unit 27 of a specific individual, and for example, detects and detects the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S2. The orientation of the face is estimated based on the position and shape of the facial organ, and the process proceeds to step S6.
In step S6, the CPU 13 operates as an eye opening / closing detection unit 28 of a specific individual, and is based on, for example, the position and shape of the facial organs obtained in step S5, particularly the position and shape of eye feature points (eyelids, pupils). , The open / closed state of the eyes, for example, whether the eyes are open or closed is detected, and the process proceeds to step S7.

In step S7, the CPU 13 operates as a line-of-sight direction estimation unit 29 of a specific individual, and for example, the orientation of the face, the position and shape of the facial organs obtained in step S5, particularly the feature points of the eyes (outer corners of eyes, inner corners of eyes, pupils) The direction of the line of sight is estimated based on the position and shape, and then the process is finished.
On the other hand, in step S4, if the CPU 13 determines that it is not the face of a specific individual, in other words, it is a normal face, the process proceeds to step S8.

In step S8, the CPU 13 operates as a normal face orientation estimation unit 31, and for example, detects the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S2, and detects the face. The orientation of the face is estimated based on the position and shape of the organ, and the process proceeds to step S9.
In step S9, the CPU 13 operates as a normal eye opening / closing detection unit 32, and for example, based on the position and shape of the facial organs obtained in step S8, particularly the position and shape of the feature points (eyelids, pupils) of the eyes. The open / closed state of the eyes, for example, whether the eyes are open or closed is detected, and the process proceeds to step S10.

In step S10, the CPU 13 operates as a normal line-of-sight direction estimation unit 33, and for example, the orientation of the face and the position and shape of the facial organs obtained in step S8, particularly the positions of the feature points of the eyes (outer corners of eyes, inner corners of eyes, pupils) The direction of the line of sight is estimated based on the shape and shape, and then the process is completed.
FIG. 8 is a flowchart showing an example of a processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. This processing operation is an example of the face detection processing operation of step S2 and the specific individual determination processing operation of step S3 shown in FIG. 7, and is an example of processing operation for one input image (1 frame).

First, the CPU 13 starts the loop process L1 for detecting the size (size) of the face in step S21, and starts the loop process L2 for detecting the rotation angle (direction and inclination) of the face in the next step S22. In the next step S23, the loop process L3 for detecting the position of the face is started, and the process proceeds to step S24.
The loop processing L1 is repeated according to the number of reduced images (for example, 20a and 20b shown in FIG. 5) generated to detect faces of various sizes. The loop processing L2 is repeated according to the setting of the discriminator for discriminating the rotation angle of the face (for example, the front face, the oblique face, the profile, and the inclination shown in FIG. 6). The loop process L3 is repeated for the number of positions where the search area 210 is scanned to detect the position of the face.

In step S24, the CPU 13 operates as the first feature amount extraction unit 221 and performs a process of extracting the facial feature amount 221a from the search area 210 under each condition of the loop processes L1, L2, and L3.
In step S25, the CPU 13 normally operates as a face discriminator 222 and a specific individual face determination unit 223, and hierarchically determines whether the search area 210 is a face (face) or a non-face (other than a face). When it is determined that the search area 210 is a non-face in any of the layers, a process of determining whether the search area 210 is a face of a specific individual or a non-face is performed.

For example, the CPU 13 scans the search area 210 for all the reduced images, and when the face detection (in other words, the detection of the face area candidate) in all the search areas 210 is completed, the loop process is performed in step S26. L1 is completed, loop processing L2 is completed in step S27, loop processing L3 is completed in step S28, and then processing proceeds to step S29.
In step S29, the CPU 13 operates as the face area integration unit 224, performs a process of integrating one or more face area candidates determined to be faces in the processes of steps S21 to S28, and proceeds to step S30. .. The method of integrating the candidates of one or more face regions is not particularly limited. For example, it may be integrated based on the average value of the region center of one or more face region candidates and the average value of the region size.

In step S30, the CPU 13 operates as the second feature amount extraction unit 225, performs a process of extracting the facial feature amount from the face region integrated in step S29, and proceeds to the process in step S31.
In step S31, the CPU 13 operates as the specific individual determination unit 25, and in step S30, the feature amount extracted from the integrated face area and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142. The process of calculating the correlation coefficient of is performed, and the process proceeds to step S32.

In step S32, the CPU 13 determines whether or not the calculated correlation coefficient is larger than a predetermined threshold value for determining whether or not the face is a specific individual, and the correlation coefficient is larger than the predetermined threshold value, in other words. Then, if it is determined that the feature amount extracted from the face area has a high correlation (in other words, the similarity is high) with the face feature amount 142a of the specific individual, the process proceeds to step S33.
In step S33, the CPU 13 determines that the face detected in the face area is the face of a specific individual, and then ends the process.

On the other hand, in step S32, the correlation coefficient is equal to or less than a predetermined threshold value, in other words, the correlation between the feature amount extracted from the face region and the face feature amount 142a of the specific individual is low (in other words, the degree of similarity). Is low), the process proceeds to step S34.
In step S34, the CPU 13 determines that the face is not a specific individual's face, in other words, a normal face, and then ends the process.

FIG. 9 is a flowchart showing an example of a face detection processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. This processing operation is an example of the discrimination processing operation in step S25 shown in FIG.
First, in step S41, the CPU 13 reads the feature amount 221a extracted from the search area 210 in step S24 of FIG. 8, and proceeds to step S42.

In step S42, the CPU 13 sets 0 in the discriminator counter (i) that counts the hierarchy of the normal face discriminator 222, and sets n indicating the number of hierarchical structures of the normal face discriminator 222. , 1 is set as the initial value, and the process proceeds to step S43.
In step S43, the CPU 13 normally operates as a face discriminator 222, and performs a process of discriminating whether the search area 210 is a face or a non-face by the nth discriminator.

In step S44, the CPU 13 determines whether the search area 210 is a face or a non-face, and if it is determined that the search area 210 is a face, the process proceeds to step S45.
In step S45, the CPU 13 adds 1 to the discriminator counter (i) and proceeds to step S46.
In step S46, the CPU 13 determines whether or not the discriminator counter (i) is less than N indicating the number of discriminators, and if it is determined that the number of discriminators is less than N, the process proceeds to step S47. In step S47, 1 is added to n in order to proceed to the process by the discriminator of the next layer, and then the process returns to step S43 and the process is repeated.

On the other hand, in step S46, if the CPU 13 determines that the discriminator counter (i) is not less than the number of discriminators N, in other words, the discriminator counter (i) has reached the number of discriminators N, the process is performed in step S48. Proceed.
In step S48, the CPU 13 determines that the search area 210 is a candidate for the face area, stores the information of the search area as a candidate for the face area, finishes the face detection process for the search area, and finishes the next search area. The face detection process for is repeated.

On the other hand, if the CPU 13 determines in step S44 that the search area 210 is non-faced, the process proceeds to step S49.
In step S49, the CPU 13 operates as the specific individual face determination unit 223, and the feature amount used in the nth discriminator (feature amount extracted from the search area 210) and the specific individual corresponding to the hierarchy of the nth discriminator. The process of calculating the correlation coefficient with the facial feature amount 142a is performed, and the process proceeds to step S50.

In step S50, the CPU 13 determines whether or not the calculated correlation coefficient is larger than a predetermined threshold value for determining whether or not the face is a specific individual, and the correlation coefficient is larger than the predetermined threshold value. If it is determined (that is, the correlation is high) (in other words, if it is determined that the search area 210 is the face of a specific individual), the process proceeds to step S45, and the discrimination process by the normal face discriminator 222 proceeds. ..
On the other hand, in step S50, if the CPU 13 determines that the correlation coefficient is equal to or less than a predetermined threshold value (that is, the correlation is low) (in other words, if it determines that the search area 210 is faceless), the step. The process proceeds to S51.
In step S51, the CPU 13 determines that the search area is a non-face (other than a face), and in the next step S52, the discrimination process after the nth discriminator by the normal face discriminator 222 is terminated, and then the search is performed. The face detection process for the area is completed, and the face detection process for the next search area is repeated.

[Action / Effect]
According to the driver monitoring device 10 according to the above-described embodiment, the face feature amount 142a of a specific individual and the normal face feature amount 142b are stored as the learned face feature amount in the face feature amount storage unit 142. The face region is detected by the normal face discriminator 222 hierarchically discriminating whether the search region 210 cut out from the image is a face or a non-face by using the normal face feature amount 142b. To.

Further, even when it is determined that the face is not a face in any layer of the normal face discriminator 222, the specific individual face determination unit 223 uses the face feature amount 142a of the specific individual to search the search area 210. By determining whether the face of a specific individual is a face or a non-face, a face region including the face of the specific individual is detected.
As a result, the face region can be accurately detected from the image regardless of whether it is a normal face or a face of a specific individual. Further, since the normal face discriminator 222 and the specific individual face determination unit 223 use a common feature amount extracted from the search area 210, there is no need for a separate process of extracting the feature amount, so that the face area can be detected. The real-time property can be maintained.

Therefore, the driver 3 can accurately detect each face in real time (in other words, by high-speed processing) regardless of whether the driver 3 is a specific individual or a normal person other than the specific individual.
Further, the feature amount used in one layer of the normal face discriminator 222 determined to be non-face by the specific individual face determination unit 223 and the face feature amount 142a of the specific individual corresponding to the one layer. Based on the correlation, it is possible to efficiently determine whether the search area is a face of a specific individual or a non-face. Therefore, even when the face discriminator 222 determines that the face is non-face, it is possible to accurately determine the face of a specific individual.

Further, according to the driver monitoring device 10, when the specific individual face determination unit 223 determines that the face is a specific individual face, the determination proceeds to the next layer of the normal face discriminator 222, and the discrimination process is swiftly performed. Will be continued. On the other hand, when the specific individual face determination unit 223 determines that the face is non-face, the determination by the normal face discriminator 222 is terminated. Therefore, it is possible to perform a process of determining the face of a specific individual while maintaining the efficiency of the normal face discriminator 222.

Further, according to the driver monitoring device 10, one or more face region candidates determined to be faces are integrated by the face region integration unit 224 via the normal face discriminator 222, and the specific individual determination unit 25 integrates the candidates. It is determined whether or not the face in the face region is the face of the specific individual by using the feature amount extracted from the integrated face region and the face feature amount 142a of the specific individual. Therefore, based on the face region integrated by the face region integration unit 224, it is possible to accurately determine whether the face is a specific individual's face or a normal person's face.
Further, the in-vehicle system 1 includes a driver monitoring device 10 and one or more ECUs 40 that execute a predetermined process based on the monitoring result output from the driver monitoring device 10. Therefore, based on the result of the monitoring, the ECU 40 can appropriately execute a predetermined control. This makes it possible to construct a highly safe in-vehicle system that allows even a specific individual to drive with peace of mind.

[Modification example]
Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and changes can be made without departing from the scope of the present invention.
In the above embodiment, the case where the image processing device according to the present invention is applied to the driver monitoring device 10 has been described, but the application example is not limited to this. For example, in a device or system for monitoring a person who operates, monitors, or performs a predetermined work of various facilities such as machines and devices in a factory, the above-mentioned specific individual is included in the monitoring target person. In some cases, the image processing apparatus according to the present invention can be applied.

[Additional Notes]
Embodiments of the present invention may also be described as, but are not limited to, the following appendices.
(Appendix 1)
An image processing device (12) that processes an image input from the image pickup unit (11).
A face feature storage unit that stores a specific individual's face feature (142a) and a normal face feature (142b) as learned face features that have been learned to detect a face from the image. (142) and
A face detection unit (22) that detects a face area while scanning the search area with respect to the image is provided.
The face detection unit (22)
A first feature amount extraction unit (221) that extracts facial feature amounts from the search area, and
A hierarchically structured normal face discriminator that discriminates whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount (142b). (222) and
When the non-face is determined by any layer of the normal face discriminator (222), the feature amount extracted from the search area and the face feature amount of the specific individual are used. An image processing device including a specific individual face determination unit (223) for determining whether the search area is the face of the specific individual or the non-face.

(Appendix 2)
An image processing method for processing an image input from the image pickup unit (11).
A face detection step (S2) of detecting a face area while scanning the search area with respect to the image is included.
The face detection step (S2)
A feature amount extraction step (S24) for extracting facial feature amounts from the search area, and
The search area is a face using the feature amount extracted by the feature amount extraction step (S24) and the learned normal face feature amount (142b) that has been learned to detect a face. The normal face discrimination step (S43, S44) for hierarchically discriminating whether the face is non-face or non-face,
Learning to detect the extracted feature amount and the face of a specific individual when the non-face is determined in any layer of the normal face discrimination step (S43, S44). With the specific individual face determination step (S49, S50) for determining whether the search area is the face of the specific individual or the non-face using the completed face feature amount (142a) of the specific individual. An image processing method characterized by including.

1 In-vehicle system 2 Vehicle 3 Driver 10 Driver monitoring device 11 Camera 12 Image processing unit 13 CPU
14 ROM
141 Program storage unit 142 Face feature amount Storage unit 142a Specific individual face feature amount 142b Normal face feature amount 15 RAM
151 Image memory 16 Communication unit 20 Image (input image)
20a, 20b Reduced image 21 Image input unit 22 Face detection unit 210 Search area 221 First feature amount extraction unit 221a Feature amount 222 Normal face discriminator 223 Specific individual face determination unit 224 Face area integration unit 225 Second feature amount extraction unit 25 Specific individual determination unit 26 First face image processing unit 27 Face orientation estimation unit for a specific individual 28 Eye opening / closing detection unit for a specific individual 29 Line-of-sight direction estimation unit 30 for a specific individual Second face image processing unit 31 Normal face orientation estimation unit 32 Normal eye opening / closing detection unit 33 Normal line-of-sight direction estimation unit 34 Output unit 40 ECU
41 Sensor 42 Actuator 43 Communication bus

Claims (10)

1. An image processing device that processes images input from the image pickup unit.
   As the learned facial features that have been learned to detect the face from the image, a facial feature storage unit that stores the facial features of a specific individual and the normal facial features, and
   A face detection unit that detects the face area while scanning the search area for the image, and
   With
   The face detection unit
   A first feature amount extraction unit that extracts facial feature amounts from the search area, and
   A hierarchically structured normal face discriminator that discriminates whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount.
   When it is determined that the face is non-face in any layer of the normal face discriminator, the search area is used by using the feature amount extracted from the search area and the face feature amount of the specific individual. The specific individual face determination unit that determines whether is the face of the specific individual or the non-face, and
   An image processing device equipped with.
2. The specific individual face determination unit
   An index showing the correlation between the feature amount used in one layer of the normal face discriminator determined to be non-face and the face feature amount of the specific individual corresponding to the one layer was calculated.
   Based on the calculated index, it is determined whether the search area is the face of the specific individual or the non-face.
   The image processing apparatus according to claim 1.
3. The specific individual face determination unit
   When the index is larger than a predetermined threshold value, it is determined that the search area is the face of the specific individual.
   When the index is equal to or less than the predetermined threshold value, it is determined that the search area is the non-face.
   The image processing apparatus according to claim 2.
4. When the specific individual face determination unit determines that the face is the specific individual's face, the determination progress unit advances the determination to the next layer of the normal face discriminator.
   When the specific individual face determination unit determines that the face is non-face, the specific individual face determination unit is provided with a discrimination cutoff unit that terminates the discrimination by the normal face discriminator.
   The image processing apparatus according to any one of claims 1 to 3.
5. The face detection unit
   A face region integration unit that integrates one or more candidates for the face region determined to be the face by the normal face discriminator, and a face region integration unit.
   It is provided with a second feature amount extraction unit that extracts facial feature amounts from the integrated face area.
   A specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual by using the feature amount extracted from the integrated face region and the face feature amount of the specific individual. Has,
   The image processing apparatus according to any one of claims 1 to 4.
6. The image processing apparatus according to any one of claims 1 to 5.
   An image pickup unit that captures an image to be input to the image processing device,
   An output unit that outputs information based on image processing by the image processing device, and
   A monitoring device equipped with.
7. The monitoring device according to claim 6 and
   It includes one or more control devices that are communicably connected to the monitoring device and execute a predetermined process based on the information output from the monitoring device.
   Control system.
8. The monitoring device is a device for monitoring the driver of the vehicle.
   The control device includes an electronic control unit mounted on the vehicle.
   The control system according to claim 7.
9. It is an image processing method that processes an image input from an imaging unit.
   A face detection step of detecting a face area while scanning the search area with respect to the image is included.
   The face detection step
   A feature amount extraction step for extracting facial feature amounts from the search area, and
   The search area is a face or a non-face using the feature amount extracted by the feature amount extraction step and a trained normal face feature amount that has been trained to detect a face. A normal face discrimination step that hierarchically discriminates whether or not
   When it is determined that the face is non-face in any of the layers of the normal face discrimination step, the extracted feature amount and the learned specific individual who has been trained to detect the face of the specific individual. A specific individual face determination step for determining whether the search area is the face of the specific individual or the non-face using the facial features of
   Image processing method including.
10. A program for causing at least one or more computers to process an image input from an imaging unit.
    To at least one of the above computers
    A face detection step of detecting a face area while scanning the search area with respect to the image is included.
    The face detection step
    A feature amount extraction step for extracting facial feature amounts from the search area, and
    The search area is a face or a non-face using the feature amount extracted by the feature amount extraction step and a trained normal face feature amount that has been trained to detect a face. A normal face discrimination step that hierarchically discriminates whether or not
    When it is determined that the face is non-face in any of the layers of the normal face discrimination step, the extracted feature amount and the learned specific individual who has been trained to detect the face of the specific individual. A specific individual face determination step for determining whether the search area is the face of the specific individual or the non-face using the facial features of
    A program to execute.

Images