WO2020261820A1 - Image processing device, monitoring device, control system, image processing method, and program - Google Patents

Abstract

An image processing device for processing an image inputted from an imaging unit, the image processing device being provided with: a facial feature amount storage unit for storing, as learned facial feature amounts, a facial feature amount of a specific individual and a normal facial feature amount; a face detection unit for detecting a face region while extracting the feature amount of a face from the image; a specific individual assessment unit for assessing, using the feature amount of the detected face region and the facial feature amount of the specific individual, whether the face in the face region is the face of the specific individual; a first facial image processing unit for performing facial image processing for a specific individual when it is assessed that the face is the face of the specific individual; and a second facial image processing unit for performing normal facial image processing when it is assessed that the face is not the face of the specific individual.

Description

Image processing equipment, monitoring equipment, control systems, image processing methods, and programs

The present invention relates to an image processing device, a monitoring device, a control system, an image processing method, and a program.

Patent Document 1 below discloses a robot device used as a service providing device that can switch to an appropriate service according to the situation of a target (person) to which the service is provided.
The robot device is equipped with a first camera, a second camera, and an information processing device including a CPU, and the CPU includes a face detection unit, an attribute determination unit, a person detection unit, a person position calculation unit, and an information processing unit. It is equipped with a movement vector detector and the like.

According to the robot device, when the service is provided to a group of people who have a relationship such as communicating with each other, the first service of providing information based on close communication is performed. To determine. On the other hand, when the service is provided to a group of people whose relationship such as communication with each other is unknown, the second service provides information unilaterally without exchanging information. Decide to do. As a result, it is possible to provide appropriate services according to the situation of the service provision target.

Japanese Unexamined Patent Publication No. 2014-14899

In the robot device, the face detection unit is configured to detect a person's face using the first camera, and a known technique can be used for the face detection.
However, with conventional face detection technology, if a part of the facial organs such as eyes, nose, and mouth is missing or greatly deformed due to injury, a large mole, wart, or body decoration such as tattoo is applied to the face. Such specific individuals (in other words, age difference, gender, and person), such as when the facial organs are displaced from their average position due to treatment or a disease such as a hereditary disease. There is a problem that the accuracy of face detection for a specific individual (a specific individual) having characteristics different from those of a general person, which is common regardless of the difference in species, is lowered.

The present invention has been made in view of the above problems, and provides an image processing device, a monitoring device, a control system, an image processing method, and a program capable of improving the accuracy of face sensing for a specific individual as described above. The purpose is.
The image processing apparatus (1) according to the present disclosure in order to achieve the above object is an image processing apparatus that processes an image input from an imaging unit.
As the learned facial features that have been learned to detect the face from the image, a facial feature storage unit that stores the facial features of a specific individual and the normal facial features, and
A face detection unit that detects a face region while extracting a feature amount for detecting a face from the image, and a face detection unit.
Using the detected feature amount of the face region and the face feature amount of the specific individual, a specific individual determination unit for determining whether or not the face in the face region is the face of the specific individual.
When the specific individual determination unit determines that the face is the face of the specific individual, the first face image processing unit that performs face image processing for the specific individual and the first face image processing unit
When it is determined by the specific individual determination unit that it is not the face of the specific individual, it is characterized by including a second face image processing unit that performs normal face image processing.

According to the image processing device (1), the face feature amount of the specific individual and the normal face feature amount (in other words, the specific individual) are used as the learned face feature amount in the face feature amount storage unit. The facial feature amount used when the person is a person other than the above) is stored, and the feature amount of the face region detected by the face detection unit and the facial feature amount of the specific individual are stored by the specific individual determination unit. It is used to determine whether or not the face in the face region is the face of the specific individual. By using the facial feature amount of the specific individual, it is possible to accurately determine whether or not the face is the face of the specific individual.

Further, when it is determined that the face of the specific individual is the face, the face image processing of the specific individual can be accurately performed by the first face image processing unit. On the other hand, when it is determined that the face is not the face of the specific individual, in other words, the face of a person other than the specific individual, the second face image processing unit performs the normal face image processing. It can be carried out with high accuracy. Therefore, both the specific individual and the ordinary person other than the specific individual can accurately perform the sensing of each face.

Further, in the image processing device (2) according to the present disclosure, in the image processing device (1), the specific individual determination unit correlates the feature amount extracted from the face region with the face feature amount of the specific individual. It is characterized in that an index indicating the above is calculated, and based on the calculated index, it is determined whether or not the face in the face region is the face of the specific individual.
According to the image processing apparatus (2), an index showing the correlation between the feature amount extracted from the face area and the face feature amount of the specific individual is calculated, and the face area is based on the calculated index. It is determined whether or not the face of the specific individual is the face of the specific individual. Thereby, it is possible to efficiently determine whether or not the face in the face region is the face of the specific individual based on the index. The index may be an index value indicating that the larger the value is, the higher the relationship is, for example, a correlation coefficient, the reciprocal of the square error, or other extraction from the face region. It may be an index value or the like indicating the degree of similarity of the relationship between the said feature amount and the face feature amount of the specific individual.

Further, in the image processing device (3) according to the present disclosure, in the image processing device (2), when the specific individual determination unit indicates that the index is larger than a predetermined threshold value, the face in the face region is the face of the specific individual. When the index is equal to or less than the predetermined threshold value, it is determined that the face in the face region is not the face of the specific individual.
According to the image processing device (3), when the index is larger than a predetermined threshold value, it is determined that the face in the face region is the face of the specific individual, and when the index is equal to or less than the predetermined threshold value, the face is described. It is determined that the face in the face area is not the face of the specific individual. The processing efficiency of the determination can be further improved by the process of comparing the index with the predetermined threshold value.

Further, in the image processing device (4) according to the present disclosure, in any of the image processing devices (1) to (3), the specific individual determination unit is based on the result of determination for one frame of the image. It is characterized in that it is determined whether or not the face in the face region is the face of the specific individual.
According to the image processing device (4), it is determined whether or not the face in the face region is the face of the specific individual based on the result of the determination for one frame of the image, so that the determination is high speed. Can be achieved.

Further, in the image processing device (5) according to the present disclosure, in any of the image processing devices (1) to (3), the specific individual determination unit is based on the result of determination for a plurality of frames of the image. It is characterized in that it is determined whether or not the face in the face region is the face of the specific individual.
According to the image processing device (5), it is determined whether or not the face in the face region is the face of the specific individual based on the result of the determination for a plurality of frames of the image, so that the accuracy of the determination is determined. Can be enhanced.

Further, the image processing device (6) according to the present disclosure is any of the above image processing devices (1) to (5), and the face image processing includes face detection processing, face orientation estimation processing, and eye direction estimation processing. And at least one of the eye opening / closing detection processing is included.
According to the image processing apparatus (6), the face image processing includes at least one of face detection processing, face orientation estimation processing, line-of-sight direction estimation processing, and eye opening / closing detection processing. It is possible to accurately perform processing for estimating and detecting various facial behaviors of the specific individual or a (normal) person other than the specific individual.

Further, the monitoring device (1) according to the present disclosure is used for any of the above image processing devices (1) to (6), an image pickup unit for capturing an image to be input to the image processing device, and image processing by the image processing device. It is characterized by having an output unit that outputs based information.
According to the monitoring device (1), not only the face of the normal person but also the face of the specific individual can be accurately monitored, and information based on the image processing can be output from the output unit. Therefore, it is possible to easily construct a monitoring system or the like that uses the information.

Further, the control system (1) according to the present disclosure is communicably connected to the monitoring device (1) and executes a predetermined process based on the information output from the monitoring device. It is characterized by having the above-mentioned control device.
According to the control system (1), it is possible to execute a predetermined process by one or more of the control devices based on the information output from the monitoring device. Therefore, it is possible to construct a system that can utilize not only the monitoring result of the normal person but also the monitoring result of the specific individual.

Further, the control system (2) according to the present disclosure is the control system (1) described above.
The monitoring device is a device for monitoring the driver of the vehicle.
The control device is characterized by including an electronic control unit mounted on the vehicle.
According to the control system (2), even when the driver of the vehicle is the specific individual, the face of the specific individual can be accurately monitored, and the electronic device is based on the monitoring result. It is possible to make the control unit appropriately execute a predetermined control. This makes it possible to construct a highly safe in-vehicle system that allows even the specific individual to drive with peace of mind.

Further, the image processing method according to the present disclosure is an image processing method for processing an image input from an imaging unit.
A face detection step of detecting a face region while extracting facial features from the image, and
The face of the face region is used by using the feature amount of the face region detected by the face detection step and the learned face feature amount of the specific individual who has been trained to detect the face of the specific individual. A specific individual determination step for determining whether or not is the face of the specific individual, and
When the face of the specific individual is determined by the specific individual determination step, the first face image processing step of performing the face image processing for the specific individual and the first face image processing step
When it is determined by the specific individual determination step that the face is not the face of the specific individual, it is characterized by including a second face image processing step of performing normal face image processing.

According to the image processing method, the face in the face region is formed by using the feature amount of the face region detected in the face detection step and the face feature amount of the specific individual in the specific individual determination step. Whether or not it is the face of the specific individual is determined. By using the facial feature amount of the specific individual, it is possible to accurately determine whether or not the face is the face of the specific individual.
Further, when it is determined that the face of the specific individual is the face, the face image processing of the specific individual can be accurately performed by the first face image processing step. On the other hand, when it is determined that the face is not the specific individual's face, in other words, it is a normal face that is not the specific individual, the normal face image processing is performed with high accuracy by the second face image processing step. Can be done. Therefore, both the specific individual and a normal person other than the specific individual can accurately sense each face.

Further, the program according to the present disclosure is a program for causing at least one or more computers to process an image input from an imaging unit.
To at least one of the above computers
A face detection step of detecting a face region while extracting facial features from the image, and
The face of the face region is used by using the feature amount of the face region detected by the face detection step and the learned face feature amount of the specific individual who has been trained to detect the face of the specific individual. A specific individual determination step for determining whether or not is the face of the specific individual, and
When the face of the specific individual is determined by the specific individual determination step, the first face image processing step of performing the face image processing for the specific individual and the first face image processing step
When it is determined by the specific individual determination step that the face is not the face of the specific individual, the second face image processing step of performing normal face image processing is executed.

According to the above program, whether or not the face in the face region is the face of the specific individual by using the feature amount of the face region and the face feature amount of the specific individual on at least one computer. It is possible to determine whether or not the face is the specific individual's face with high accuracy.
Further, when it is determined that the face of the specific individual is the face, the face image processing of the specific individual can be performed with high accuracy. On the other hand, when it is determined that the face is not the specific individual's face, in other words, it is a normal face that is not the specific individual, the normal face image processing can be performed with high accuracy. Therefore, it is possible to construct a device or system capable of accurately sensing each face regardless of whether the specific individual or an ordinary person other than the specific individual. The above program may be a program stored in a storage medium, a program that can be transferred via a communication network, or a program that is executed via a communication network. ..

It is a schematic diagram which shows an example of the in-vehicle system including the driver monitoring apparatus which concerns on embodiment of this invention. It is a block diagram which shows an example of the hardware composition of the in-vehicle system including the driver monitoring apparatus which concerns on embodiment. It is a block diagram which shows the functional structure example of the image processing part of the driver monitoring apparatus which concerns on embodiment. It is a flowchart which shows an example of the processing operation performed by the image processing part of the driver monitoring apparatus which concerns on embodiment. It is a flowchart which shows an example of the specific individual judgment processing operation performed by the image processing part of the driver monitoring apparatus which concerns on embodiment. It is a flowchart which shows another example of the specific individual determination processing operation performed by the image processing part of the driver monitoring apparatus which concerns on embodiment.

Hereinafter, an image processing device, a monitoring device, a control system, an image processing method, and an embodiment of a program according to the present invention will be described with reference to the drawings.
The image processing apparatus according to the present invention can be widely applied to, for example, an apparatus or system for monitoring an object such as a person using a camera. The image processing device according to the present invention operates or monitors, for example, various facilities such as machines and devices in a factory, in addition to devices and systems for monitoring drivers (operators) of various moving objects such as vehicles. It can also be applied to devices and systems that monitor people who perform predetermined work.

[Application example]
FIG. 1 is a schematic view showing an example of an in-vehicle system including the driver monitoring device according to the embodiment. In this application example, an example in which the image processing apparatus according to the present invention is applied to the driver monitoring apparatus 10 will be described.
The in-vehicle system 1 includes a driver monitoring device 10 that monitors the state of the driver 3 of the vehicle 2 (for example, facial behavior), and one or more ECUs (Electronic Control Units) that control the running, steering, or braking of the vehicle 2. ) 40, and one or more sensors 41 for detecting the state of each part of the vehicle, the state around the vehicle, and the like are included, and these are connected via the communication bus 43. The in-vehicle system 1 is configured as, for example, an in-vehicle network system that communicates according to a CAN (Controller Area Network) protocol. As the communication standard of the in-vehicle system 1, a communication standard other than CAN may be adopted. The driver monitoring device 10 is an example of the "monitoring device" of the present invention, and the in-vehicle system 1 is an example of the "control system" of the present invention.

The driver monitoring device 10 transmits information based on image processing by the camera 11 for capturing the face of the driver 3, the image processing unit 12 that processes the image input from the camera 11, and the image processing unit 12, and the communication bus 43. It is configured to include a communication unit 16 that performs processing such as output to a predetermined ECU 40 via the above. The image processing unit 12 is an example of the "image processing device" of the present invention. The camera 11 is an example of the "imaging unit" of the present invention.

The driver monitoring device 10 detects the face of the driver 3 from the image captured by the camera 11, and detects the behavior of the face such as the direction of the face of the detected driver 3, the direction of the line of sight, or the open / closed state of the eyes. The driver monitoring device 10 may determine the state of the driver 3, such as forward gaze, inattentiveness, dozing, backward facing, and prone, based on the detection results of these facial behaviors. Further, the driver monitoring device 10 outputs a signal based on the state determination of the driver 3 to the ECU 40, and the ECU 40 performs attention and warning processing to the driver 3 or operation control of the vehicle 2 (for example, deceleration) based on the signal. Control, guidance control to the road shoulder, etc.) may be executed.

One of the purposes of the driver monitoring device 10 is to improve the accuracy of face sensing for a specific individual.
In the conventional driver monitoring device, the driver 3 of the vehicle 2 has a part of facial organs such as eyes, nose, and mouth missing or greatly deformed due to, for example, an injury, or a large mole or wart on the face, or Accuracy of detecting the face from the image captured by the camera when the facial organs are displaced from the average position due to body decoration such as tattoo or a disease such as a hereditary disease. There was a problem that the

Further, if the face detection accuracy is lowered, the post-face detection processing such as the face orientation estimation processing is not properly performed, so that the driver 3 cannot properly perform the state determination such as inattentiveness or dozing. Further, there is a problem that various controls to be executed by the ECU 40 based on the state determination may not be appropriately performed.
In order to solve the problem, the driver monitoring device 10 according to the embodiment is a general person who is common regardless of a specific individual, in other words, a difference in age, gender, race, etc. (individual difference). In order to improve the accuracy of face detection for a specific individual who has features different from the face features of (ordinary person), the following configuration was adopted.

In the image processing unit 12, as the learned facial features that have been learned to detect the face from the image, the facial features of a specific individual and the normal facial features (in other words, a person other than the specific individual). The amount of facial features used in the case of) is stored.
The image processing unit 12 performs face detection processing for detecting a face region while extracting a feature amount for detecting a face from an input image of the camera 11. Then, the image processing unit 12 determines whether or not the face in the face region is the face of the specific individual by using the detected feature amount of the face region and the face feature amount of the specific individual. Performs specific individual judgment processing.

In the specific individual determination process, a correlation coefficient is calculated and calculated as an index showing the relationship between the feature amount extracted from the face region and the face feature amount of the specific individual, for example, an index showing the correlation. Based on the number of relationships, it may be determined whether or not the face in the face region is the face of the specific individual.
For example, when the correlation coefficient is larger than a predetermined threshold value, it is determined that the face in the face region is the face of the specific individual, and when the correlation coefficient is equal to or less than the predetermined threshold value, the face in the face region is It may be determined that it is not the face of the specific individual. In the specific individual determination process, an index other than the correlation coefficient may be used.

Further, in the specific individual determination process, it may be determined whether or not the face in the face region is the face of the specific individual based on the result of determination for one frame of the input image from the camera 11. Based on the result of determination for a plurality of frames of the input image from the camera 11, it may be determined whether or not the face in the face region is the face of the specific individual.
As described above, in the driver monitoring device 10, the learned facial feature amount of the specific individual is stored in advance in the image processing unit 12, and the face feature amount of the specific individual is used to obtain the face of the specific individual. It is possible to accurately determine whether or not it is.

Further, when the face of the specific individual is determined by the specific individual determination process, the image processing unit 12 executes the face image process for the specific individual, so that the face image process of the specific individual is accurately performed. It becomes possible to do.
On the other hand, when it is determined that the face is not the face of the specific individual, in other words, a normal face (in other words, a face other than the specific individual), the image processing unit 12 executes the normal face image processing. Therefore, the normal face image processing can be performed with high accuracy. Therefore, whether the driver 3 is a specific individual or an ordinary person other than the specific individual, it is possible to accurately perform sensing of each face.

[Hardware configuration example]
FIG. 2 is a block diagram showing an example of the hardware configuration of the in-vehicle system 1 including the driver monitoring device 10 according to the embodiment.
The in-vehicle system 1 includes a driver monitoring device 10, 1 or more ECUs 40 for monitoring the state of the driver 3 of the vehicle 2, and 1 or more sensors 41, which are connected via a communication bus 43. Further, one or more actuators 42 are connected to the ECU 40.
The driver monitoring device 10 includes a camera 11, an image processing unit 12 that processes an image input from the camera 11, and a communication unit 16 for exchanging data and signals with an external ECU 40 and the like. There is.

The camera 11 is a device that captures an image including the face of the driver 3 seated in the driver's seat. For example, a lens unit, an image sensor unit, a light irradiation unit, an interface unit, a camera control unit that controls each of these units, and the like. Can be configured to include. The image sensor unit may include an image sensor such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor), a filter, a microlens, and the like. The image pickup device unit may be an element capable of forming an image pickup image by receiving light in a visible region, or an element capable of forming an image pickup image by receiving light in a near infrared region. The light irradiation unit is configured to include a light emitting element such as an LED (Light Emitting Diode), and may include a near infrared LED or the like so that the driver's face can be imaged day or night. The camera 11 captures an image at a predetermined frame rate (for example, several tens of frames per second), and the data of the captured image is input to the image processing unit 12. The camera 11 may be an external type as well as an integrated type.

The image processing unit 12 is configured as an image processing device including one or more CPU (Central Processing Unit) 13, ROM (Read Only Memory) 14, and RAM (Random Access Memory) 15. The ROM 14 includes a program storage unit 141 and a facial feature amount storage unit 142, and the RAM 15 includes an image memory 151 for storing an input image from the camera 11. The driver monitoring device 10 may be provided with another storage unit, and the storage unit may be used as the program storage unit 141, the facial feature amount storage unit 142, and the image memory 151. The other storage unit may be a semiconductor memory or a storage medium that can be read by a disk drive or the like.

The CPU 13 is an example of a hardware processor, and by reading, interpreting, and executing data such as a program stored in the program storage unit 141 of the ROM 14 and the face feature amount stored in the face feature amount storage unit 142. , Processing of the image input from the camera 11, for example, face image processing such as face detection processing is performed. Further, the CPU 13 performs a process of outputting the result (for example, processing data, determination signal, control signal, etc.) obtained by the face image processing to the ECU 40 or the like via the communication unit 16.

In the face feature amount storage unit 142, as the learned face feature amount that has been learned (for example, machine learning) to detect the face from the image, the face feature amount 142a of the specific individual shown in FIG. The facial feature amount 142b is stored.
As the learned facial features, various feature quantities effective for detecting a face from an image can be used. For example, a feature amount (Haar-like feature amount) focusing on the difference in brightness (difference in average brightness between two rectangular areas of various sizes) in a local area of the face may be used. Alternatively, a feature amount (LBP (Local Binary Pattern) feature amount) focusing on a combination of brightness distributions in the local region of the face may be used, or the distribution of the brightness in the local region of the face in the gradient direction may be used. Features (HOG (Histogram of Oriented Gradients) features) focusing on the combination may be used.

The face feature amount stored in the face feature amount storage unit 142 is extracted as an effective feature amount for face detection by using, for example, various machine learning methods. Machine learning is a process of finding a pattern inherent in data (learning data) by a computer. For example, AdaBoost may be used as an example of a statistical learning method. AdaBoost selects a large number of discriminators (weak discriminators) with low discriminating ability, selects a weak discriminator with a small error rate from these many weak discriminators, adjusts parameters such as weights, and has a hierarchical structure. It is a learning algorithm that can construct a strong discriminator by setting. The discriminator may be referred to as a discriminator, a classifier, or a learner.

For example, the strong discriminator is configured to discriminate one feature amount effective for face detection by one weak discriminator, and a large number of weak discriminators and their combinations are selected by AdaBoost, and these are used hierarchically. The structure may be constructed. Note that one weak discriminator may output information such as 1 for a face and 0 for a non-face. Further, as the learning method, a learning method called Real AdaBoost, which can output a real number from 0 to 1 instead of 0 or 1, may be used. Further, as these learning methods, a neural network having an input layer, an intermediate layer, and an output layer may be used.

A large number of face images captured under various conditions and a large number of non-face images (non-face images) are given as training data to a learning device equipped with such a learning algorithm, learning is repeated, weighting, etc. By adjusting the parameters and optimizing it, it is possible to construct a strong discriminator having a hierarchical structure capable of detecting a face with high accuracy. Then, one or more feature amounts used in the weak discriminators of each layer constituting such a strong discriminator can be used as the learned facial feature amounts.

For example, the face feature amount 142a of a specific individual individually captures a face image of the specific individual at a predetermined place under various conditions (conditions such as various face orientations, line-of-sight directions, or eye open / closed states). Then, these a large number of captured images are input to the learning device as teacher data, and are parameters that indicate the facial features of a specific individual adjusted by the learning process. The facial feature amount 142a of the specific individual may be, for example, a combination pattern of the difference in brightness of the local region of the face obtained by the learning process. The facial feature amount 142a of a specific individual stored in the facial feature amount storage unit 142 may be only the facial feature amount of one specific individual, or can be used when a plurality of specific individuals drive the vehicle 2. , The facial features of a plurality of specific individuals may be stored.

The normal facial feature amount 142b is the above-mentioned learning device using images of a normal human face image captured under various conditions (conditions such as various face orientations, line-of-sight directions, or eye open / closed states) as teacher data. It is a parameter indicating the characteristics of a normal human face, which is input to and adjusted by the learning process. The normal facial feature amount 142b may be, for example, a combination pattern of light and dark differences in a local region of the face obtained by a learning process. Further, as the normal facial feature amount 142b, the information registered in the predetermined facial feature amount database may be used.

The learned facial feature amount stored in the facial feature amount storage unit 142 is fetched from a server on the cloud or the like via a communication network such as the Internet or a mobile phone network and stored in the facial feature amount storage unit 142. It may be configured as such.
The ECU 40 is composed of a computer device including one or more processors, a memory, a communication module, and the like. Then, the processor mounted on the ECU 40 reads, interprets, and executes the program stored in the memory, so that predetermined control for the actuator 42 and the like is executed.

The ECU 40 includes, for example, at least one of a traveling system ECU, a driving support system ECU, a body system ECU, and an information system ECU.
The traveling system ECU includes, for example, a drive system ECU, a chassis system ECU, and the like. The drive system ECU includes a control unit related to a "running" function such as engine control, motor control, fuel cell control, EV (Electric Vehicle) control, or transmission control. The chassis-based ECU includes a control unit related to a "stop, turn" function such as brake control or steering control.

The driving support system ECU has, for example, an automatic braking support function, a lane keeping support function (also referred to as LKA / Lane Keep Assist), a constant speed driving / inter-vehicle distance support function (also referred to as ACC / Adaptive Cruise Control), and a forward collision warning function. , Lane departure warning function, blind spot monitoring function, traffic sign recognition function, etc., functions that automatically improve safety or realize comfortable driving by linking with driving ECUs (driving support function or automatic driving function) It may be configured to include at least one control unit with respect to.

The driving support system ECU includes, for example, Level 1 (driver assistance), Level 2 (partially automatic driving), and Level 3 (conditional automatic driving) at the automatic driving level presented by the American Society of Automotive Engineers of Japan (SAE). ) May be equipped with at least one of the functions. Further, the functions of level 4 (highly automatic driving) or level 5 (fully automatic driving) of the automatic driving level may be equipped, and only the functions of level 1 and 2 or only level 2 and 3 are equipped. May be good. Further, the in-vehicle system 1 may be configured as an automatic driving system.

The body system ECU may be configured to include at least one control unit related to the function of the vehicle body such as a door lock, a smart key, a power window, an air conditioner, a light, an instrument panel, or a winker.
The information system ECU may be configured to include, for example, an infotainment device, a telematics device, or an ITS (Intelligent Transport Systems) related device. The infotainment device may include, for example, an HMI (Human Machine Interface) device that functions as a user interface, a car navigation device, an audio device, and the like. The telematics device may include a communication unit or the like for communicating with the outside. The ITS-related device may include an ETC (Electronic Toll Collection System), a communication unit for performing road-to-vehicle communication with a roadside machine such as an ITS spot, or vehicle-to-vehicle communication.

The sensor 41 may include various in-vehicle sensors that acquire sensing data necessary for controlling the operation of the actuator 42 by the ECU 40. For example, in addition to vehicle speed sensors, shift position sensors, accelerator opening sensors, brake pedal sensors, steering sensors, etc., peripheral monitoring of external imaging cameras, millimeter-wave radar (Radar), riders (LIDER), ultrasonic sensors, etc. A sensor or the like may be included.

The actuator 42 is a device that executes operations related to traveling, steering, braking, etc. of the vehicle 2 based on a control signal from the ECU 40, and includes, for example, an engine, a motor, a transmission, a hydraulic cylinder, an electric cylinder, and the like.

[Functional configuration example]
FIG. 3 is a block diagram showing a functional configuration example of the image processing unit 12 of the driver monitoring device 10 according to the embodiment.
The image processing unit 12 includes an image input unit 21, a face detection unit 22, a specific individual determination unit 25, a first face image processing unit 26, a second face image processing unit 30, an output unit 34, and a face feature amount storage unit 142. It is configured to include.
The image input unit 21 performs a process of capturing an image including the face of the driver 3 captured by the camera 11.

The face detection unit 22 is configured to include a face detection unit 23 of a specific individual and a normal face detection unit 24, and performs a process of detecting a face region while extracting a feature amount for detecting a face from an input image. Do.
The face detection unit 23 of the specific individual uses the face feature amount 142a of the specific individual read from the face feature amount storage unit 142 to perform a process of detecting the face region from the input image.
The normal face detection unit 24 uses the normal face feature amount 142b read from the face feature amount storage unit 142 to perform a process of detecting a face region from an input image.

The method of detecting the face area from the image is not particularly limited, but a method of detecting the face area at high speed and with high accuracy is adopted. The face detection unit 22 extracts, for example, a feature amount for detecting a face in each search area while scanning a predetermined search area (search window) for the input image. The face detection unit 22 may extract, for example, the difference in brightness (luminance difference) of a local region of the face, the edge strength, or the relationship between these local regions as a feature amount. Then, the face detection unit 22 uses the feature amount extracted from the search area, the normal face feature amount 142b read from the face feature amount storage unit 142, or the face feature amount 142a of a specific individual, and has a hierarchical structure ( A detector (hierarchical structure that captures the details of the face from the hierarchy that roughly captures the face) determines whether the face is face or non-face, and performs processing to detect the face area from the image.

The specific individual determination unit 25 uses the feature amount of the face area detected by the face detection unit 22 and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142 to detect the face in the face area. Performs a process of determining whether or not is the face of a specific individual.
The specific individual determination unit 25 calculates a correlation coefficient as an index showing the relationship between the feature amount extracted from the face region and the face feature amount 142a of the specific individual, for example, an index showing the correlation, and the calculated correlation coefficient. It may be determined whether or not the face in the face region is the face of a specific individual based on. For example, the correlation of feature quantities such as Haar-like features (luminance difference) of one or more local regions in the face region may be obtained. Then, when the correlation coefficient is larger than a predetermined threshold value, it is determined that the face in the detected face area is the face of a specific individual, and when the correlation coefficient is equal to or less than the predetermined threshold value, the face in the detected face area is a specific individual. It may be determined that it is not the face of.

Further, the specific individual determination unit 25 includes a learned learning device that has been machine-learned using the facial feature amount 142a of the specific individual as a parameter in order to determine whether or not the face is the face of the specific individual. By inputting the feature amount extracted from the face area into the learned learner and performing arithmetic processing of the learned learner, for example, correlation analysis processing of the feature amount, whether or not the face is a specific individual's face. The determination information may be acquired from the learning device. The learned learner may be configured to include a non-linear discriminator, or may be configured to include a linear discriminator. For example, the trained learner may be configured to include a support vector machine or may be configured to include a neural network.

Further, the specific individual determination unit 25 may determine whether or not the face in the detected face region is the face of the specific individual based on the result of determination for one frame of the input image from the camera 11. Based on the result of determination for a plurality of frames of the input image from the camera 11, it may be determined whether or not the face in the detected face region is the face of a specific individual.
When the specific individual determination unit 25 determines that the face is a specific individual's face, the first face image processing unit 26 performs face image processing for the specific individual. The first face image processing unit 26 includes a face orientation estimation unit 27 of a specific individual, an eye opening / closing detection unit 28 of the specific individual, and a line-of-sight direction estimation unit 29 of the specific individual, but is still different. It may include a configuration for estimating or detecting facial behavior. In addition, the first face image processing unit 26 may perform any processing of the face image processing for the specific individual by using the face feature amount 142a of the specific individual. Further, the face feature amount storage unit 142 stores the learned feature amount that has been machine-learned to perform the face image processing for the specific individual, and uses the learned feature amount for the specific individual. You may perform any processing of the face image processing of.

The face orientation estimation unit 27 of the specific individual performs a process of estimating the face orientation of the specific individual. The face orientation estimation unit 27 of the specific individual detects, for example, the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face region detected by the face detection unit 23 of the specific individual, and the detected facial organs. Performs processing to estimate the orientation of the face based on the position and shape.
The method for detecting the facial organs from the facial region in the image is not particularly limited, but it is preferable to adopt a method capable of detecting the facial organs at high speed and with high accuracy. For example, a method of creating a three-dimensional face shape model, fitting it to a face region on a two-dimensional image, and detecting the position and shape of each organ of the face can be adopted. As a technique for fitting a three-dimensional face shape model to a human face in an image, for example, the technique described in Japanese Patent Application Laid-Open No. 2007-249280 can be applied, but the technique is not limited thereto.

Further, the face orientation estimation unit 27 of the specific individual can use the estimation data of the face orientation of the specific individual, for example, the pitch angle of vertical rotation (around the X axis) included in the parameters of the three-dimensional face shape model. The yaw angle of the left-right rotation (around the Y axis) and the roll angle of the entire rotation (around the Z axis) may be output.
The eye opening / closing detection unit 28 of the specific individual performs a process of detecting the opening / closing state of the eyes of the specific individual. The eye opening / closing detection unit 28 of the specific individual, for example, is based on the position and shape of the facial organs obtained by the face orientation estimation unit 27 of the specific individual, particularly the position and shape of the feature points (eyelids, pupils) of the eyes. Detects the open / closed state, for example, whether the eyes are open or closed. For the open / closed state of the eye, for example, the feature amount of the image of the eye (the position of the eyelid, the shape of the pupil (black eye), the area size of the white eye part and the black eye part, etc.) in various open / closed states of the eye is previously learned. It may be detected by learning using the data and evaluating the degree of similarity with the learned feature data.

The line-of-sight direction estimation unit 29 of the specific individual performs a process of estimating the line-of-sight direction of the specific individual. The line-of-sight direction estimation unit 29 of a specific individual is based on, for example, the orientation of the face of the driver 3 and the position and shape of the facial organs of the driver 3, particularly the position and shape of the feature points of the eyes (outer corners of eyes, inner corners of eyes, pupils). Estimate the direction of the line of sight. The direction of the line of sight is the direction in which the driver 3 is looking, and is determined by, for example, a combination of the direction of the face and the direction of the eyes.

In addition, the direction of the line of sight is, for example, the feature amount of the image of the eye in various combinations of face orientation and eye orientation (relative position of outer corner, inner corner of eye, pupil, relative position of white eye portion and black eye portion, shading, etc. (Texture, etc.) may be detected by learning in advance using a learning device and evaluating the degree of similarity with the learned feature amount data. In addition, the line-of-sight direction estimation unit 29 of the specific individual estimates the size and center position of the eyeball from the size and orientation of the face, the position of the eyes, etc., using the fitting result of the three-dimensional face shape model, and the pupil. The position of the eyeball may be detected, and the vector connecting the center of the eyeball and the center of the pupil may be detected as the line-of-sight direction.

When the specific individual determination unit 25 determines that the face is not the face of a specific individual, the second face image processing unit 30 performs normal face image processing. The second face image processing unit 30 includes a normal face orientation estimation unit 31, a normal eye opening / closing detection unit 32, and a normal line-of-sight direction estimation unit 33, but has yet another face behavior. It may include a configuration for estimating or detecting. In addition, the second face image processing unit 30 may perform any processing of the normal face image processing using the normal face feature amount 142b. Further, the face feature amount storage unit 142 stores the learned feature amount obtained by machine learning for performing the normal face image processing, and the learned feature amount is used for the normal face image processing. Any of the above processes may be performed. The processing performed by the normal face orientation estimation unit 31, the normal eye opening / closing detection unit 32, and the normal line-of-sight direction estimation unit 33 includes the face orientation estimation unit 27 of the specific individual and the eye opening / closing detection of the specific individual. Since it is basically the same as the unit 28 and the line-of-sight direction estimation unit 29 of a specific individual, the description thereof will be omitted here.

The output unit 34 performs a process of outputting information based on the image processing by the image processing unit 12 to the ECU 40 or the like. The information based on the image processing may be, for example, information on the behavior of the face such as the direction of the face of the driver 3, the direction of the line of sight, or the open / closed state of the eyes, or the driver 3 determined based on the detection result of the behavior of the face. Information on the state of (for example, forward gaze, inattentiveness, dozing, backward facing, prone, etc.) may be used. Further, the information based on the image processing may be a predetermined control signal (control signal for performing caution or warning processing, control signal for performing operation control of the vehicle 2, etc.) based on the state determination of the driver 3.

[Processing operation example]
FIG. 4 is a flowchart showing an example of a processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. For example, the camera 11 captures an image of several tens of frames per second, and this processing is performed for each frame or every frame at regular intervals.
First, in step S1, the CPU 13 operates as an image input unit 21, performs a process of reading an image (an image including the face of the driver 3) captured by the camera 11, and proceeds to step S2.
In step S2, the CPU 13 operates as a normal face detection unit 24, performs normal face detection processing on the input image, and proceeds to step S3. For example, the CPU 13 scans a predetermined search area (search window) for an input image and extracts a feature amount for detecting a face in each search area. Then, the CPU 13 determines whether the face is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount 142b read from the face feature amount storage unit 142, and detects the face area from the image. Perform the process of

In step S3, the CPU 13 operates as the face detection unit 23 of the specific individual, performs face detection processing of the specific individual on the input image, and proceeds to step S4. For example, the CPU 13 scans a predetermined search area (search window) for an input image and extracts a feature amount for detecting a face in each search area. Then, the CPU 13 determines whether the face is a face or a non-face by using the feature amount extracted from the search area and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142, and determines the face area from the image. Perform the detection process. The processes of steps S2 and S3 may be performed in parallel in one step or may be performed in combination.

In step S4, the CPU 13 operates as the specific individual determination unit 25, and uses the feature amount of the face region detected in steps S2 and S3 and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142. Then, a process of determining whether or not the face in the face area is a face of a specific individual is performed, and the process proceeds to step S5.
In step S5, the CPU 13 determines whether or not the result of the determination process in step S4 is the face of a specific individual, and if it is determined that the result is the face of a specific individual, the process proceeds to step S6.

In step S6, the CPU 13 operates as a face orientation estimation unit 27 of a specific individual, and for example, detects and detects the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S3. The orientation of the face is estimated based on the position and shape of the facial organ, and the process proceeds to step S7.
In step S7, the CPU 13 operates as an eye opening / closing detection unit 28 for a specific individual, and is based on, for example, the position and shape of the facial organs obtained in step S6, particularly the position and shape of eye feature points (eyelids, pupils). , The open / closed state of the eyes, for example, whether the eyes are open or closed is detected, and the process proceeds to step S8.

In step S8, the CPU 13 operates as a line-of-sight direction estimation unit 29 of a specific individual, and for example, the orientation of the face, the position and shape of the facial organs obtained in step S6, particularly the feature points of the eyes (outer corners of eyes, inner corners of eyes, pupils). The direction of the line of sight is estimated based on the position and shape, and then the process is finished.
On the other hand, in step S5, if the CPU 13 determines that it is not the face of a specific individual, in other words, it is a normal face, the process proceeds to step S9.

In step S9, the CPU 13 operates as a normal face orientation estimation unit 31, and for example, detects the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S2, and detects the face. The orientation of the face is estimated based on the position and shape of the organ, and the process proceeds to step S10.
In step S10, the CPU 13 operates as a normal eye opening / closing detection unit 32, and for example, based on the position and shape of the facial organs obtained in step S9, particularly the position and shape of the feature points (eyelids, pupils) of the eyes. The open / closed state of the eyes, for example, whether the eyes are open or closed is detected, and the process proceeds to step S11.

In step S11, the CPU 13 operates as a normal line-of-sight direction estimation unit 33, and for example, the orientation of the face and the position and shape of the facial organs obtained in step S9, particularly the positions of the feature points of the eyes (outer corners of eyes, inner corners of eyes, pupils). The direction of the line of sight is estimated based on the shape and shape, and then the process is completed.
FIG. 5 is a flowchart showing an example of a specific individual determination processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. This processing operation is an example of the specific individual determination processing operation in step S4 shown in FIG. 4, and is an example of the processing operation in the case of determining with one input image (1 frame).

First, in step S21, the CPU 13 reads the feature amount extracted from the face area detected by the face detection processes of steps S2 and S3 shown in FIG. 4, and in the next step S22, learns from the face feature amount storage unit 142. The completed facial feature amount 142a of the specific individual is read, and the process proceeds to step S23.
In step S23, the CPU 13 performs a process of calculating the correlation coefficient between the feature amount extracted from the face area read in step S21 and the face feature amount 142a of the specific individual read in step S22, and processes in step S24. To proceed.

In step S24, the CPU 13 determines whether or not the calculated correlation coefficient is larger than a predetermined threshold value for determining whether or not the individual is a specific individual, and the correlation coefficient is larger than the predetermined threshold value, in other words. If it is determined that the feature amount extracted from the face area and the face feature amount 142a of the specific individual have a high correlation (in other words, the similarity is high), the process proceeds to step S25.
In step S25, the CPU 13 determines that the face detected in the face area is the face of a specific individual, and then ends the process.

On the other hand, in step S24, the correlation coefficient is equal to or less than a predetermined threshold value, in other words, the correlation between the feature amount extracted from the face region and the face feature amount 142a of the specific individual is low (in other words, the degree of similarity). Is low), the process proceeds to step S26.
In step S26, the CPU 13 determines that the face is not a specific individual's face, in other words, a normal face, and then ends the process.

FIG. 6 is a flowchart showing an example of a specific individual determination processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment.
This processing operation is another example of the specific individual determination processing operation in step S4 shown in FIG. 4, and is a processing operation example in the case of determining with a plurality of input images (multiple frames).
First, in step S31, the CPU 13 sets the counter (numSp) of the image in which the face of a specific individual is detected to 0, and in step S32, the counter (i) of the input image to be determined is set to 0 and step S33. Proceed to processing.

In step S33, the CPU 13 performs a specific individual determination process (for example, the processes of steps S21 to S26 shown in FIG. 5) for one input image (1 frame), and proceeds to step S34.
In step S34, the CPU 13 determines whether or not the result of the determination process in step S33 is the face of a specific individual, and if it is determined that the result is the face of a specific individual, the process proceeds to step S35.

In step S35, the CPU 13 adds 1 to the counter (numSp) of the face image of the specific individual and proceeds with the process in step S36, and in step S36, adds 1 to the counter (i) of the input image and steps. The process proceeds to S37.
On the other hand, in step S34, if the CPU 13 determines that the face is not the face of a specific individual, the process proceeds to step S36, and in step S36, 1 is added to the counter (i) of the input image and the process proceeds to step S37. .. That is, in this case, 1 is not added to the counter (numSp) of the face image of the specific individual.

In step S37, the CPU 13 determines whether or not the input image counter (i) is less than the predetermined number of images N, and the counter (i) is less than N (in other words, the predetermined number of images (N). If it is determined that the determination of (sheets) has not been completed), the process returns to step S33, and the determination process of the specific individual in the next input image is repeated.
On the other hand, in step S37, if the CPU 13 determines that the counter (i) of the input image is not less than N (in other words, the determination of the predetermined number of images (N) has been completed), the processing is performed in step S38. Proceed.

In step S38, the CPU 13 determines whether or not the counter (numSp) of the face image of the specific individual is larger than a predetermined threshold value for determining the specific individual. The predetermined threshold value may be set to N / 2 or a value larger than N / 2 when the number of input images used for determination is N (N frames), for example. ..
In step S38, if the CPU 13 determines that the face image counter (numSp) of the specific individual is larger than the predetermined threshold value, the process proceeds to step S39, and in step S39, it is determined that the face of the specific individual is the face of the specific individual. After finishing the determination process of the specific individual, the face image processing for the specific individual is performed.

On the other hand, in step S38, if the CPU 13 determines that the counter (numSp) of the specific individual face image is equal to or less than a predetermined threshold value, the process proceeds to step S40, and in step S40, the face is not the face of the specific individual (in other words, For example, it is a normal face), the judgment process of a specific individual is completed, and then the normal face image processing is performed.
According to the driver monitoring device 10 according to the above-described embodiment, the face feature amount 142a of a specific individual and the normal face feature amount 142b are stored as the learned face feature amount in the face feature amount storage unit 142. The specific individual determination unit 25 determines whether or not the face in the face region is the face of a specific individual by using the feature amount of the face region detected by the face detection unit 22 and the face feature amount 142a of the specific individual. Will be done. Therefore, by using the face feature amount 142a of the specific individual, it is possible to accurately determine whether or not the face is the face of the specific individual, and it is possible to reduce the load on the determination process.

Further, when the specific individual determination unit 25 determines that the face is a specific individual, the first face image processing unit 26 can accurately perform the face image processing of the specific individual. On the other hand, when the specific individual determination unit 25 determines that the face is not a specific individual's face, in other words, a normal face (a face of a person other than the specific individual), the second face image processing unit 30 determines that the face is a normal face. Image processing can be performed with high accuracy. Therefore, whether the driver 3 is a specific individual or an ordinary person other than the specific individual, it is possible to accurately perform sensing of each face.

Further, the specific individual determination unit 25 calculates a correlation coefficient as an index showing the correlation between the feature amount extracted from the face region and the face feature amount 142a of the specific individual, and based on the calculated correlation coefficient, It is determined whether or not the face in the face region is the face of the specific individual. As a result, it is possible to efficiently determine whether or not the face in the face region is the face of the specific individual based on the correlation coefficient, and a process of comparing the correlation coefficient with a predetermined threshold value. Therefore, the processing efficiency of the determination can be further improved.

Further, the in-vehicle system 1 includes a driver monitoring device 10 and one or more ECUs 40 that execute a predetermined process based on the monitoring result output from the driver monitoring device 10. Therefore, based on the result of the monitoring, the ECU 40 can appropriately execute a predetermined control. This makes it possible to construct a highly safe in-vehicle system that allows even a specific individual to drive with peace of mind.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and changes can be made without departing from the scope of the present invention.
In the above embodiment, the case where the image processing device according to the present invention is applied to the driver monitoring device 10 has been described, but the application example is not limited to this. For example, in a device or system for monitoring a person who operates, monitors, or performs a predetermined work of various facilities such as machines and devices in a factory, the above-mentioned specific individual is included in the monitoring target person. In some cases, the image processing apparatus according to the present invention can be applied.

[Additional Notes]
Embodiments of the present invention may also be described as, but are not limited to, the following appendices.
(Appendix 1)
An image processing device (12) that processes an image input from the image pickup unit (11).
A face feature storage unit that stores a specific individual's face feature (142a) and a normal face feature (142b) as learned face features that have been learned to detect a face from the image. (142) and
A face detection unit (22) that detects a face region while extracting a feature amount for detecting a face from the image, and a face detection unit (22).
A specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual by using the detected feature amount of the face region and the face feature amount (142a) of the specific individual. (25) and
When the specific individual determination unit (25) determines that the face is the specific individual's face, the first face image processing unit (26) that performs face image processing for the specific individual and
Image processing characterized by including a second face image processing unit (30) that performs normal face image processing when the specific individual determination unit (25) determines that the face is not the face of the specific individual. apparatus.

(Appendix 2)
The imaging unit (11), the storage unit (14),
Using a device with at least one or more processors (13)
An image processing method for processing an image input from the imaging unit (11).
The storage unit (14)
A face feature storage unit that stores a specific individual's face feature (142a) and a normal face feature (142b) as learned face features that have been learned to detect a face from the image. With (142)
The processor (13)
Face detection steps (S2, S3) for detecting a face region while extracting facial features from the image, and
Using the feature amount of the face region detected by the face detection steps (S2, S3) and the face feature amount (142a) of the specific individual read from the face feature amount storage unit (142), the said A specific individual determination step (S4) for determining whether or not the face in the face region is the face of the specific individual, and
When the face of the specific individual is determined by the specific individual determination step (S4), the first face image processing step (S6, S7, S8) for performing the face image processing for the specific individual and
When it is determined by the specific individual determination step (S4) that the face is not the specific individual's face, it includes a second face image processing step (S9, S10, S11) for performing normal face image processing. Image processing method.

1 In-vehicle system 2 Vehicle 3 Driver 10 Driver monitoring device 11 Camera 12 Image processing unit 13 CPU
14 ROM
141 Program storage unit 142 Face feature amount Storage unit 142a Specific individual face feature amount 142b Normal face feature amount 15 RAM
151 Image memory 16 Communication unit 21 Image input unit 22 Face detection unit 23 Specific individual face detection unit 24 Normal face detection unit 25 Specific individual judgment unit 26 First face image processing unit 27 Specific individual face orientation estimation unit 28 Specific individual Eye opening / closing detection unit 29 Line-of-sight direction estimation unit 30 of a specific individual Second face image processing unit 31 Normal face orientation estimation unit 32 Normal eye opening / closing detection unit 33 Normal line-of-sight direction estimation unit 34 Output unit 40 ECU
41 Sensor 42 Actuator 43 Communication bus

Claims (11)

1. An image processing device that processes images input from the image pickup unit.
   As the learned facial features that have been learned to detect the face from the image, a facial feature storage unit that stores the facial features of a specific individual and the normal facial features, and
   A face detection unit that detects a face region while extracting a feature amount for detecting a face from the image, and a face detection unit.
   Using the detected feature amount of the face region and the face feature amount of the specific individual, a specific individual determination unit for determining whether or not the face in the face region is the face of the specific individual.
   When the specific individual determination unit determines that the face is the face of the specific individual, the first face image processing unit that performs face image processing for the specific individual and the first face image processing unit
   When the specific individual determination unit determines that the face is not the specific individual's face, the second face image processing unit that performs normal face image processing and
   An image processing device equipped with.
2. The specific individual judgment unit
   An index showing the correlation between the feature amount extracted from the face region and the facial feature amount of the specific individual was calculated.
   Based on the calculated index, it is determined whether or not the face in the face region is the face of the specific individual.
   The image processing apparatus according to claim 1.
3. The specific individual judgment unit
   When the index is larger than a predetermined threshold value, it is determined that the face in the face region is the face of the specific individual.
   When the index is equal to or less than the predetermined threshold value, it is determined that the face in the face region is not the face of the specific individual.
   The image processing apparatus according to claim 2.
4. The specific individual judgment unit
   Based on the result of the determination for one frame of the image, it is determined whether or not the face in the face region is the face of the specific individual.
   The image processing apparatus according to any one of claims 1 to 3.
5. The specific individual judgment unit
   Based on the result of determination for a plurality of frames of the image, it is determined whether or not the face in the face region is the face of the specific individual.
   The image processing apparatus according to any one of claims 1 to 3.
6. The face image processing includes at least one of a face detection process, a face orientation estimation process, a line-of-sight direction estimation process, and an eye opening / closing detection process.
   The image processing apparatus according to any one of claims 1 to 5.
7. The image processing apparatus according to any one of claims 1 to 6.
   An image pickup unit that captures an image to be input to the image processing device,
   An output unit that outputs information based on image processing by the image processing device, and
   A monitoring device equipped with.
8. The monitoring device according to claim 7 and
   It includes one or more control devices that are communicably connected to the monitoring device and execute a predetermined process based on the information output from the monitoring device.
   Control system.
9. The monitoring device is a device for monitoring the driver of the vehicle.
   The control device includes an electronic control unit mounted on the vehicle.
   The control system according to claim 8.
10. It is an image processing method that processes an image input from an imaging unit.
    A face detection step of detecting a face region while extracting facial features from the image, and
    The face of the face region is used by using the feature amount of the face region detected by the face detection step and the learned face feature amount of the specific individual who has been trained to detect the face of the specific individual. A specific individual determination step for determining whether or not is the face of the specific individual, and
    When the face of the specific individual is determined by the specific individual determination step, the first face image processing step of performing the face image processing for the specific individual and the first face image processing step
    When it is determined by the specific individual determination step that it is not the face of the specific individual, it includes a second face image processing step of performing normal face image processing.
    Image processing method.
11. A program for causing at least one or more computers to process an image input from an imaging unit.
    To at least one of the above computers
    A face detection step of detecting a face region while extracting facial features from the image, and
    The face of the face region is used by using the feature amount of the face region detected by the face detection step and the learned face feature amount of the specific individual who has been trained to detect the face of the specific individual. A specific individual determination step for determining whether or not is the face of the specific individual, and
    When the face of the specific individual is determined by the specific individual determination step, the first face image processing step of performing the face image processing for the specific individual and the first face image processing step
    A program for executing a second face image processing step of performing normal face image processing when it is determined by the specific individual determination step that the face is not the face of the specific individual.

Images