JP7230710B2 - Image processing device, monitoring device, control system, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, a monitoring device, a control system, an image processing method, and a program.

Patent Literature 1 below discloses a robot device used as a service providing device capable of switching to an appropriate service according to the situation of a target (person) to whom the service is provided.

The robot device is equipped with a first camera, a second camera, and an information processing device including a CPU. The CPU includes a face detection unit, an attribute determination unit, a person detection unit, a person position calculation unit, and It is equipped with a movement vector detection unit.

According to the above-described robot device, when the target of service provision is a group of persons who have established relationships such as mutual communication, the first service of providing information based on close exchanges is provided. to decide. On the other hand, if the target of service provision is a group of people whose relationship such as mutual communication is unknown, the second service provides information unilaterally without any interaction. decide to do As a result, appropriate services can be provided according to the situation of the service provider.

[Problems to be solved by the invention]
In the robot device, the face detection unit detects a person's face using the first camera, and known technology can be used for the face detection.
However, in the conventional face detection technology, when a part of the facial organs such as eyes, nose, mouth, etc. is missing or greatly deformed due to injury, etc., there is a large mole, wart, or body decoration such as a tattoo on the face. Such a specific individual (in other words, age difference, gender, and person There is a problem that the accuracy of face detection for a specific individual having features different from the facial features of general people common regardless of species etc. is lowered.

JP 2014-14899 A

Means to solve the problem and its effect

The present invention has been made in view of the above problems, and an image processing device, a monitoring device, a control system, an image processing method, and an image processing device capable of accurately detecting even the face of a specific individual as described above in real time. The purpose is to provide a program.

To achieve the above object, an image processing device (1) according to the present disclosure is an image processing device that processes an image input from an imaging unit,
a facial feature amount storage unit that stores a specific individual's facial feature amount and a normal facial feature amount as learned facial feature amounts that have been trained to detect a face from the image;
a face detection unit that detects a face area while scanning a search area of the image;
The face detection unit
a first feature quantity extraction unit that extracts a face feature quantity from the search area;
a hierarchical normal face discriminator that discriminates whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount;
When the face is determined to be a non-face in any hierarchy of the normal face classifier, the search region is determined using the feature quantity extracted from the search region and the face feature quantity of the specific individual. is the face of the specific individual or the non-face of the specific individual.

According to the image processing device (1), the normal face discriminator uses the normal face feature amount to hierarchically discriminate whether the search area is a face or a non-face, , the face region is detected. Further, even when the normal face discriminator determines that the face is non-face in any of the hierarchies, the specific individual face determination unit uses the specific individual's face feature amount to determine the search area. is the specific individual's face or non-face, the face region containing the specific individual's face is detected. As a result, the face region can be detected with high accuracy whether it is the normal face or the face of the specific individual. In addition, since the common feature amount extracted from the search area is used in the normal face discriminator and the specific individual face determination unit, real-time performance regarding detection of the face area can be maintained.

The image processing device (2) according to the present disclosure is the image processing device (1), wherein the specific individual face determination unit uses in one layer of the normal face discriminator determined as the non-face. An index indicating a correlation between the feature amount and the face feature amount of the specific individual corresponding to the one layer is calculated, and based on the calculated index, whether the search area is the face of the specific individual or not It is characterized by determining whether it is a non-face.

According to the image processing device (2), the feature amount used in one layer of the normal face discriminator that was determined to be a non-face, and the face feature amount of the specific individual corresponding to the one layer It is possible to efficiently determine whether the search area is the face of the specific individual or a non-face based on the index showing the correlation with the normal face discriminator. Even if it is the face of the specific individual, it can be determined with high accuracy. The index may be an index value indicating that the higher the value, the higher the relationship, for example, a correlation coefficient, the reciprocal of the squared error, or the non-face. It may be an index value indicating the similarity of the relationship between the feature amount used in one layer of the normal face discriminator that discriminated as and the face feature amount of the specific individual corresponding to the one layer. .

The image processing device (3) according to the present disclosure is the image processing device (2), wherein the specific individual face determining unit determines that the search area is the face of the specific individual when the index is larger than a predetermined threshold. and when the index is equal to or less than the predetermined threshold value, it is determined that the search area is the non-face.

According to the image processing device (3), if the index is greater than a predetermined threshold, the search area is determined to be the face of the specific individual, and if the index is less than or equal to the predetermined threshold, the search area is determined to be the non-face. Processing efficiency of the determination can be improved by the process of comparing the index and the predetermined threshold.

The image processing device (4) according to the present disclosure, in any one of the image processing devices (1) to (3), when the specific individual face determination unit determines that the face is the specific individual, the normal a discrimination progressing unit that advances discrimination to the next layer of the face discriminator;
and a determination terminating unit that terminates determination by the normal face discriminator when the specific individual face determining unit determines that the face is a non-face.

According to the image processing device (4), when the specific individual face determination unit determines that the face is the specific individual face, the determination proceeds to the next layer of the normal face discriminator, and the determination process proceeds quickly. to be continued. On the other hand, when the specific individual face determining unit determines that the face is the non-face, the determination by the normal face determining device is terminated. Therefore, the process of determining the face of the specific individual can be performed while maintaining the efficiency of the normal face discriminator.

The image processing device (5) according to the present disclosure is any one of the image processing devices (1) to (4), wherein the face detection unit is one or more determined to be the face by the normal face discriminator. and a second feature quantity extraction unit for extracting a face feature quantity from the integrated face region, wherein the face region extracted from the integrated face region A specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual by using the feature quantity and the face feature quantity of the specific individual.

According to the image processing device (5), one or more candidates for the face area determined to be the face are integrated, and the feature amount extracted from the integrated face area and the specific individual's It is determined whether or not the face in the face region is the face of the specific individual by using the face feature amount. Therefore, it is possible to accurately determine whether the face area integrated by the face area integration unit is the face of the specific individual or the face of the ordinary person.

A monitoring device (1) according to the present disclosure includes any one of the image processing devices (1) to (5), an imaging unit that captures an image to be input to the image processing device, and image processing by the image processing device. and an output unit for outputting information.

According to the monitoring device (1), it is possible to accurately detect and monitor not only the face of the ordinary person but also the face of the specific individual, and output the information based on the image processing from the output unit. can be output, it is possible to easily construct a monitoring system or the like that uses the information.

A control system (1) according to the present disclosure is connected to the monitoring device (1) so as to be able to communicate with the monitoring device, and based on the information output from the monitoring device, one or more that execute a predetermined process and a control device.

According to the above control system (1), it is possible to cause one or more of the control devices to execute a predetermined process based on the information output from the monitoring device. Therefore, it is possible to construct a system that can use not only the monitoring results of the ordinary person but also the monitoring results of the specific individual.

A control system (2) according to the present disclosure is the control system (1), wherein the monitoring device (1) is a device for monitoring a driver of a vehicle, and the control device is mounted on the vehicle. It is characterized by including an electronic control unit.

According to the control system (2), even if the driver of the vehicle is the specific individual, the face of the specific individual can be monitored with high accuracy, and based on the monitoring results, the electronic It is possible to cause the control unit to appropriately execute predetermined control. As a result, it is possible to construct a highly safe vehicle-mounted system that allows even the specific individual to drive with peace of mind.

An image processing method according to the present disclosure is an image processing method for processing an image input from an imaging unit,
A face detection step of detecting a face region while scanning a search region of the image;
The face detection step includes:
a feature quantity extraction step of extracting a face feature quantity from the search area;
The search area is determined to be a face or a non-face by using the feature quantity extracted by the feature quantity extraction step and a learned normal face feature quantity that has been trained to detect a face. a normal face discrimination step for hierarchically discriminating whether
The feature amount extracted when the non-face is discriminated in any hierarchy of the normal face discriminating step, and the learned specific individual who has undergone learning to detect the face of the specific individual. and a specific individual face determination step of determining whether the search area is the face of the specific individual or the non-face by using the facial feature amount of .

According to the above image processing method, in the normal face determination step, the normal face feature amount is used to hierarchically determine whether the search area is a face or a non-face, thereby determining whether the search area is a face or not. A region is detected. Further, even if the non-face is determined in any layer of the normal face determination step, the specific individual face determination step uses the face feature amount of the specific individual to determine the search area. is the specific individual's face or non-face, the face region containing the specific individual's face is detected. As a result, the face region can be detected with high accuracy whether it is the normal face or the face of the specific individual. In addition, since the common feature amount extracted from the search area is used in the normal face determination step and the specific individual face determination step, the real-time nature of detection of the face area can be maintained. Therefore, the face of the specific individual can be detected in real time with high accuracy.

A program according to the present disclosure is a program for causing at least one or more computers to process an image input from an imaging unit,
to said at least one computer;
A face detection step of detecting a face region while scanning a search region of the image;
The face detection step includes:
a feature quantity extraction step of extracting a face feature quantity from the search area;
The search area is determined to be a face or a non-face by using the feature quantity extracted by the feature quantity extraction step and a learned normal face feature quantity that has been trained to detect a face. a normal face discrimination step for hierarchically discriminating whether
The feature amount extracted when the non-face is discriminated in any hierarchy of the normal face discriminating step, and the learned specific individual who has undergone learning to detect the face of the specific individual. and a specific individual face determination step of determining whether the search area is the face of the specific individual or the non-face using the facial feature amount of .

According to the above program, the at least one computer hierarchically discriminates whether the search area is a face or a non-face by using the normal face feature quantity in the normal face discrimination step. to detect the face area. Further, even if the non-face is determined in any layer of the normal face determination step, the specific individual face determination step uses the face feature amount of the specific individual to determine the search area. is the specific individual's face or non-face, the face region containing the specific individual's face can be detected. As a result, it is possible to accurately detect the face area regardless of whether it is the normal face or the face of the specific individual. In addition, since the common feature amount extracted from the search area is used in the normal face determination step and the specific individual face determination step, the real-time nature of detection of the face area can be maintained. Therefore, it is possible to construct a device or system capable of accurately sensing the face of each of the specific individual and a normal person other than the specific individual. The program may be a program stored in a storage medium, a program that can be transferred via a communication network, or a program that is executed via a communication network. .

1 is a schematic diagram showing an example of an in-vehicle system including a driver monitoring device according to an embodiment of the present invention; FIG. 1 is a block diagram showing an example hardware configuration of an in-vehicle system including a driver monitoring device according to an embodiment; FIG. 2 is a block diagram showing an example functional configuration of an image processing unit of the driver monitoring device according to the embodiment; FIG. 3 is a block diagram showing a functional configuration example of a face detection unit included in the image processing unit; FIG. FIG. 4 is a schematic diagram for explaining an example of processing operations performed by a face detection unit; FIG. 4 is a schematic diagram for explaining an example of processing operations performed by a face detection unit; 4 is a flowchart showing an example of processing operations performed by an image processing unit of the driver monitoring device according to the embodiment; 7 is a flow chart showing an example of a face detection processing operation and a specific individual determination processing operation performed by an image processing unit of the driver monitoring device according to the embodiment; 4 is a flow chart showing an example of a determination processing operation performed by an image processing unit of the driver monitoring device according to the embodiment;

Embodiments of an image processing device, a monitoring device, a control system, an image processing method, and a program according to the present invention will be described below with reference to the drawings.
INDUSTRIAL APPLICABILITY The image processing apparatus according to the present invention can be widely applied to, for example, apparatuses and systems for monitoring objects such as people using cameras. The image processing device according to the present invention is, for example, a device or system for monitoring a driver (operator) of various moving bodies such as a vehicle, or operates or monitors various facilities such as machines and devices in a factory. It can also be applied to a device or system for monitoring a person who performs a predetermined work or the like.

[Application example]
FIG. 1 is a schematic diagram showing an example of an in-vehicle system including a driver monitoring device according to an embodiment. In this application example, an example in which an image processing device according to the present invention is applied to a driver monitoring device 10 will be described.

The in-vehicle system 1 includes a driver monitoring device 10 that monitors the state of the driver 3 of the vehicle 2 (for example, behavior of the face), and one or more ECUs (Electronic Control Units) that control driving, steering, or braking of the vehicle 2. ) 40 and one or more sensors 41 for detecting the state of each part of the vehicle or the state of the surroundings of the vehicle, which are connected via a communication bus 43 . The in-vehicle system 1 is configured as an in-vehicle network system that communicates according to CAN (Controller Area Network) protocol, for example. In addition, other communication standards than CAN may be adopted as the communication standard of the in-vehicle system 1 . The driver monitoring device 10 is an example of the "monitoring device" of the present invention, and the in-vehicle system 1 is an example of the "control system" of the present invention.

The driver monitoring device 10 includes a camera 11 for capturing an image of the face of the driver 3, an image processing unit 12 for processing an image input from the camera 11, and information based on image processing by the image processing unit 12 via a communication bus 43. and a communication unit 16 that performs processing for outputting to a predetermined ECU 40 via the communication unit 16 . The image processing unit 12 is an example of the "image processing device" of the present invention. The camera 11 is an example of the "imaging section" of the present invention.

The driver monitoring device 10 detects the face of the driver 3 from the image captured by the camera 11, and detects the behavior of the detected face of the driver 3, such as the direction of the face, the direction of the line of sight, or the state of opening and closing the eyes. The driver monitoring device 10 may determine the state of the driver 3, such as looking ahead, looking aside, dozing off, looking backwards, lying down, etc., based on these facial behavior detection results. In addition, the driver monitoring device 10 outputs a signal based on the state determination of the driver 3 to the ECU 40, and the ECU 40 performs attention or warning processing to the driver 3 or operation control (for example, deceleration control) of the vehicle 2 based on the signal. control, or guidance control to the road shoulder, etc.).

In the driver monitoring device 10, the face sensing of the driver 3, in particular, whether the driver 3 is a specific individual or a normal person other than the specific individual, can be performed in real time with high accuracy. One of the purposes is to

In the conventional driver monitoring device, the driver 3 of the vehicle 2 has, for example, a part of facial organs such as eyes, nose, mouth, etc. missing or greatly deformed due to injury, or a large mole or wart on the face. Accuracy of detecting a face from an image captured by a camera when the arrangement of the facial organs deviates from the average position due to body decorations such as tattoos or diseases such as genetic diseases. There was a problem that the

In addition, when the face detection accuracy decreases, post-detection processing such as face direction estimation processing will not be performed properly, so it will not be possible to properly determine the state of the driver 3, such as looking aside or falling asleep. Moreover, there is a problem that various controls that should be executed by the ECU 40 based on the state determination may not be performed appropriately.

In order to solve such a problem, the driver monitoring device 10 according to the embodiment can detect a specific individual, in other words, a common general person regardless of differences in age, gender, race, etc. (individual differences). The following configuration is adopted in order to accurately perform real-time face detection for a specific individual having features different from those of a normal person (also called a normal person).

The image processing unit 12 stores a specific individual's facial feature amount and a normal facial feature amount (in other words, a normal person's face facial features used for detection) are stored.

The image processing unit 12 performs face detection processing to detect a face area while scanning a search area of a predetermined size in the input image of the camera 11 and extracting feature amounts for detecting a face from the search area. conduct. Then, the image processing unit 12 uses the feature amount extracted from the search area and the normal facial feature amount to hierarchically determine whether the search area is a face or a non-face. A face area is detected from the input image by normal face discrimination processing.
Further, when the non-face is determined in any layer of the normal face determination process, the image processing unit 12 extracts the feature amount extracted from the search area and the face feature amount of the specific individual. is used to perform specific individual face determination for determining whether the search area is the face of a specific individual or not.

In the specific individual face determination process, the relationship between the feature amount used in the one layer determined to be non-face in the normal face determination process and the face feature amount of the specific individual corresponding to the one layer is determined. An indicative index, such as a correlation coefficient, may be calculated, and whether the search area is the face or non-face of the specific individual may be determined based on the calculated correlation coefficient.

For example, when the correlation coefficient is greater than a predetermined threshold, it may be determined that the search area is the face of the specific individual, and determination processing may proceed to the next layer in the normal face determination processing. Further, when the correlation coefficient is equal to or less than the predetermined threshold value, it may be determined that the search area is the non-face, and the normal face determination processing for the search area may be terminated. In addition, in the specific individual face determination process, an index other than the correlation coefficient may be used.

As described above, in the driver monitoring device 10, by the normal face determination process, using the normal face feature amount, it is hierarchically determined whether the search area is a face or a non-face, and the face area is determined. is detected. Further, even when the non-face is determined in any layer of the normal face determination process, the specific individual face determination process uses the face feature amount of the specific individual to determine the search area. is the specific individual's face or non-face, the face region containing the specific individual's face is detected.

Further, when the non-face is determined in any layer of the normal face determination process and the non-face is determined by the specific individual face determination process, the search area is a non-face ( face), the process is discontinued, and the process proceeds to the next search area.

With these configurations, it is possible to detect the face region with high accuracy regardless of whether it is the normal face or the face of the specific individual. In addition, since the common feature amount extracted from the search area is used in the normal face determination process and the specific individual face determination process, it is possible to maintain the real-time nature of detection of the face area. .
Therefore, whether the driver 3 is a specific individual or a normal person other than the specific individual, it is possible to accurately detect each face in real time (in other words, by high-speed processing). .

[Hardware configuration example]
FIG. 2 is a block diagram showing an example of the hardware configuration of the in-vehicle system 1 including the driver monitoring device 10 according to the embodiment.

The in-vehicle system 1 includes a driver monitoring device 10 that monitors the state of the driver 3 of the vehicle 2 , one or more ECUs 40 , and one or more sensors 41 , which are connected via a communication bus 43 . One or more actuators 42 are connected to the ECU 40 .

The driver monitoring device 10 includes a camera 11, an image processing unit 12 that processes an image input from the camera 11, and a communication unit 16 that exchanges data and signals with an external ECU 40 or the like. there is

The camera 11 is a device that captures an image including the face of the driver 3 sitting in the driver's seat, and includes, for example, a lens unit, an imaging device unit, a light irradiation unit, an interface unit, and a camera control unit that controls these units. can be configured to include The imaging element section may include an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), a filter, a microlens, and the like. The imaging device section may be a device capable of receiving light in the visible region to form a captured image, or may be a device capable of receiving light in the near-infrared region to form a captured image. The light irradiation unit includes a light emitting element such as an LED (Light Emitting Diode), and may include a near-infrared LED or the like so that the face of the driver 3 can be imaged day or night. The camera 11 captures an image at a predetermined frame rate (for example, several tens of frames per second), and data of the captured image is input to the image processing section 12 . The camera 11 may be an integrated type or an external type.

The image processing unit 12 is configured as an image processing device including one or more CPU (Central Processing Unit) 13 , ROM (Read Only Memory) 14 , and RAM (Random Access Memory) 15 . The ROM 14 includes a program storage unit 141 and a face feature amount storage unit 142 , and the RAM 15 includes an image memory 151 for storing an input image from the camera 11 . The driver monitoring device 10 may be equipped with another storage unit, and the other storage unit may be used as the program storage unit 141 , the facial feature amount storage unit 142 and the image memory 151 . The separate storage unit may be a semiconductor memory or a storage medium readable by a disk drive or the like.

The CPU 13 is an example of a hardware processor, and reads programs stored in the program storage unit 141 of the ROM 14 and data such as facial feature amounts stored in the facial feature amount storage unit 142, interprets and executes them. , the image input from the camera 11 is processed, for example, face image processing such as face detection processing is performed. In addition, the CPU 13 performs processing of outputting the result (for example, processing data, determination signal, control signal, etc.) obtained by the face image processing to the ECU 40 or the like via the communication unit 16 .

The face feature quantity storage unit 142 stores a face feature quantity 142a of a specific individual and a normal face feature quantity 142b as learned face feature quantities that have undergone learning (for example, machine learning) to detect a face from an image. and (see FIGS. 3 and 4) are stored.
Various feature amounts effective for detecting a face from an image can be used as the learned face feature amount. For example, a feature amount (Haar-like feature amount) focused on the light-dark difference (difference in average luminance) of a local region of the face may be used. Alternatively, a feature value (LBP (Local Binary Pattern) feature value) focusing on a combination of luminance distributions in local regions of the face may be used, or a distribution of luminance gradient directions in local regions of the face may be used. A feature amount focused on a combination (HOG (Histogram of Oriented Gradients) feature amount) may be used.

The face feature quantity stored in the face feature quantity storage unit 142 is extracted as a feature quantity effective for face detection using various machine learning techniques, for example. Machine learning is a process of finding patterns inherent in data (learning data) by a computer. For example, AdaBoost may be used as an example of a statistical learning method. AdaBoost selects a large number of classifiers with low discrimination ability (weak classifiers), selects weak classifiers with a small error rate from among these many weak classifiers, adjusts parameters such as weights, and creates a hierarchical structure. It is a learning algorithm that can construct a strong discriminator by Note that the discriminator may also be called a discriminator, a classifier, or a learner.

The strong classifier is configured, for example, to classify one feature quantity effective for face detection by one weak classifier, select a large number of weak classifiers and their combinations by AdaBoost, and use these to perform hierarchical classification. It is good also as what constructed the structure. Note that one weak discriminator may output information such as 1 for a face and 0 for a non-face, for example. Also, as a learning method, a learning method called Real AdaBoost, which can output a real number between 0 and 1 instead of 0 or 1, may be used. Moreover, a neural network having an input layer, an intermediate layer, and an output layer may be used for these learning methods.

A large number of face images captured under various conditions and a large number of non-face images (non-face images) are given as learning data to a learning device equipped with such a learning algorithm. By adjusting parameters for optimization, it is possible to construct a strong discriminator having a hierarchical structure capable of detecting faces with high accuracy. One or more feature amounts used in the weak classifiers of each layer constituting such a strong classifier can be used as learned face feature amounts.

For example, the specific individual's face feature quantity 142a is obtained by individually capturing a specific individual's face image at a predetermined location in advance under various conditions (various conditions such as face orientation, line-of-sight direction, and eye open/closed state). Then, these many captured images are input to the learning device as training data, and adjusted by learning processing, and are parameters indicating the facial features of a specific individual. The face feature amount 142a of the specific individual may be, for example, a combination pattern of light and dark differences in local regions of the face obtained by learning processing. The face feature quantity 142a of the specific individual stored in the face feature quantity storage unit 142 may be the face feature quantity of only one specific individual, or may correspond to the case where a plurality of specific individuals drive the vehicle 2. , facial features of a plurality of specific individuals may be stored.

The normal face feature amount 142b is obtained by using images of normal people's faces captured under various conditions (various conditions such as face orientation, line of sight direction, and eye open/closed state) as teacher data. , and adjusted by the learning process, representing the features of a normal human face. The normal face feature quantity 142b may be, for example, a combination pattern of light and dark differences in local areas of the face obtained by learning processing. Information registered in a predetermined facial feature amount database may be used as the normal facial feature amount 142b.

The learned facial features stored in the facial feature storage unit 142 are stored in the facial feature storage unit 142 by, for example, fetching them from a cloud server or the like via a communication network such as the Internet or a mobile phone network. It may be configured to be

The ECU 40 is composed of a computer device including one or more processors, memories, communication modules, and the like. A processor installed in the ECU 40 reads, interprets, and executes the program stored in the memory, thereby performing predetermined control on the actuator 42 and the like.

The ECU 40 includes, for example, at least one of a driving system ECU, a driving support system ECU, a body system ECU, and an information system ECU.

The travel system ECU includes, for example, a drive system ECU, a chassis system ECU, and the like. The drive system ECU includes, for example, a control unit related to a "running" function such as engine control, motor control, fuel cell control, EV (Electric Vehicle) control, or transmission control. The chassis system ECU includes, for example, a control unit for "stop, turn" functions such as brake control or steering control.

The driving support system ECU has, for example, an automatic braking support function, a lane keeping support function (also referred to as LKA/Lane Keep Assist), a constant speed driving/vehicle distance support function (also referred to as ACC/Adaptive Cruise Control), and a forward collision warning function. , Lane departure warning function, blind spot monitoring function, traffic sign recognition function, etc. Functions that automatically improve safety or realize comfortable driving in cooperation with driving system ECU etc. (driving support function or automatic driving function) may be configured to include at least one or more control units for

The driving support system ECU includes, for example, level 1 (driver support), level 2 (partially automated driving), and level 3 (conditional automated driving) in the automated driving levels presented by the Society of Automotive Engineers (SAE). ) may be equipped with at least one of the functions of Furthermore, the function of level 4 (highly automated driving) or level 5 (fully automated driving) of the automatic driving level may be equipped, or only the functions of levels 1 and 2 or only levels 2 and 3 are equipped. good too. Also, the in-vehicle system 1 may be configured as an automatic driving system.

The body system ECU may include at least one or more control units related to vehicle body functions such as door locks, smart keys, power windows, air conditioners, lights, meter panels, and blinkers.

The information system ECU may include, for example, an infotainment device, a telematics device, or an ITS (Intelligent Transport Systems)-related device. The infotainment device may include, for example, an HMI (Human Machine Interface) device that functions as a user interface, a car navigation device, an audio device, and the like. The telematics device may include a communication unit or the like for communicating with the outside. The ITS-related device may include an ETC (Electronic Toll Collection System), a communication unit for performing road-to-vehicle communication with a roadside device such as an ITS spot, or vehicle-to-vehicle communication.

The sensor 41 may include various in-vehicle sensors that acquire sensing data necessary for the ECU 40 to control the operation of the actuator 42 . For example, in addition to vehicle speed sensors, shift position sensors, accelerator opening sensors, brake pedal sensors, steering sensors, etc., peripheral monitoring such as cameras for imaging outside the vehicle, radar such as millimeter waves, LIDER, and ultrasonic sensors Sensors and the like may also be included.

The actuator 42 is a device that executes operations related to running, steering, braking, etc. of the vehicle 2 based on control signals from the ECU 40, and includes, for example, an engine, a motor, a transmission, a hydraulic or electric cylinder, and the like.

[Example of functional configuration]
FIG. 3 is a block diagram showing a functional configuration example of the image processing section 12 of the driver monitoring device 10 according to the embodiment.
The image processing unit 12 includes an image input unit 21, a face detection unit 22, a specific individual determination unit 25, a first face image processing unit 26, a second face image processing unit 30, an output unit 34, and a face feature amount storage unit 142. is composed of

The image input unit 21 performs processing for capturing an image including the face of the driver 3 captured by the camera 11 .

The face detection unit 22 detects a face area while scanning a search area of a predetermined size in an input image and extracting facial features from the search area. The face detection unit 22 includes a first feature amount extraction unit 221 , a normal face discriminator 222 and a specific individual face determination unit 223 . The face detection section 22 may further include a face area integration section 224 and a second feature amount extraction section 225 .

FIG. 4 is a block diagram showing a functional configuration example of the face detection section 22. As shown in FIG.
5 and 6 are schematic diagrams for explaining an example of the processing operation performed by the face detection unit 22. FIG.

In the present embodiment, the face detection unit 22 uses, for example, Haar-like features as image features, and a normal face discriminator 222 with a hierarchical structure constructed using an AdaBoost learning algorithm, and a normal face discriminator 222 222 is configured to use a specific individual face determination unit 223 added thereto.

A Haar-like feature is also called a rectangular feature. For example, the difference in average brightness between two rectangular regions is used as a feature quantity. , to the side of the eye), which utilizes the feature that the brightness is high. Haar-like features may use rectangle features that are a combination of two, three, or four rectangles. Feature amounts effective (highly important) for face detection and their combinations are selected using a learning algorithm and stored in the face feature amount storage unit 142 . The facial feature amount storage unit 142 stores normal facial feature amounts 142b used in processing by the normal face discriminator 222 and specific individual facial feature amounts 142a used in processing by the specific individual face determination unit 223. It is

As shown in FIG. 4, the normal face discriminator 222 includes first discriminators 222a to Nth discriminators 222n, and has a hierarchical structure (also referred to as a cascade structure) in which a plurality of these discriminators are connected. Each of these discriminators uses one or more feature values 221a effective for face detection, which are extracted from a search region 210 of a predetermined size cut out from an image, to determine whether the search region 210 is a face or a non-face (face). other than). Discriminators at the beginning of the hierarchical structure, such as the first discriminator 222a, use the feature quantity 221a that roughly captures the face, such as whether or not the face has eyes. Further, in classifiers deeper in the hierarchical structure such as the N-th classifier 222n, for example, whether there are eyes, a nose, a mouth, whether the face is frontal, whether the face is oblique, or whether the face is in profile, etc. , the feature amount 221a that captures the details of the face is used.

The first feature amount extraction unit 221 extracts from the search area 210 one or more feature amounts 221 a that are set to be determined by each classifier constituting the normal face classifier 222 .

The normal face classifier 222 uses the feature quantity 221a extracted from the search region 210 by the first feature quantity extraction unit 221 and the normal face feature quantity 142b to determine whether the search region 210 is a face or not. The presence is determined hierarchically in order from the first discriminator 222a to the Nth discriminator 222n.

As shown in FIG. 4, the specific individual face determination unit 223 includes a first determination unit 223a to an N-th determination unit 223n. Then, using the feature amount 221a extracted from the search area 210 and the face feature amount 142a of the specific individual, it is determined whether the search area 210 is the face of the specific individual or not.

For example, when the second discriminator 222b determines that the search area 210 is non-face, the second determination unit 223b extracts the feature amount 221a (used by the second discriminator 222b) from the search area 210. , and the face feature amount 142a of the specific individual corresponding to the hierarchy, it is determined whether the search area 210 is the face of the specific individual or not.

When the second determination unit 223b determines that the search area 210 is the face of a specific individual, the process proceeds to determination processing in the third discriminator 222c. The discrimination process is terminated by the discriminator 222b, and the process proceeds to face detection process for the next search area.

Then, the determination process proceeds, and when the N-th discriminator 222n determines that the search area 210 is a face, or when the N-th determination unit 223n determines that the search area 210 is the face of a specific individual, The search area 210 is stored as a face area candidate.

In order to detect faces of various sizes, the face detection unit 22 generates reduced images 20a and 20b by reducing an input image 20 by a plurality of magnifications, as shown in FIG. A search area 210 of a predetermined size may be cut out from the reduced images 20a and 20b, and a normal face discriminator 222 may be used to determine whether the search area 210 is a face or a non-face. Then, by scanning the search area 210 in the input image 20 and the reduced images 20a, 20b, faces of various sizes and their positions in the image 20 may be detected. Note that the search area 210 may have any shape other than a rectangle.

Further, the face detection unit 22 may be configured to detect faces facing (rotated) in various directions and faces tilted at various angles. For example, the first feature quantity extraction unit 221 extracts a feature quantity 221a effective for determining the orientation and tilt of the face from the search area 210, and the normal face discriminator 222 extracts the orientation and tilt of the face. It may be configured such that it can be determined whether the search area 210 is a face or a non-face using a learned discriminator that has learned about each feature amount.

For example, as shown in FIG. 6, in order to detect a frontal face, an oblique face, and a side face, respectively, the normal face classifier 222 detects frontal (0 degrees), left oblique (45 degrees), and left lateral (90 degrees) faces. ) may be provided with a discriminator that has learned each feature amount. In this case, learning may be performed so that one discriminator can cover a predetermined angle (for example, 22.5 degrees) or more. In addition, the right diagonal (45 degrees) can be determined by horizontally inverting the left diagonal (45 degrees), and the right horizontal (90 degrees) can be determined by horizontally inverting the left horizontal (90 degrees). good. Further, as shown in FIG. 6, the normal face discriminator 222 may include discriminators that have learned each feature amount for each predetermined tilt so that the tilt of the face can be detected.

The face area integration unit 224 performs processing to integrate one or more face area candidates determined to be faces by the normal face determination unit 222 . The method of integrating one or more face area candidates is not particularly limited. For example, integration may be performed based on an average value of area centers of one or more face area candidates and an average value of area sizes.

The second feature quantity extraction unit 225 performs a process of extracting a face feature quantity from the face regions integrated by the face region integration unit 224 .

The specific individual determination unit 25 uses the feature amount of the face area detected by the face detection unit 22 and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142 to determine the face in the detected face area. is the face of a specific individual or the face of a normal person other than the specific individual.

The specific individual determining unit 25 calculates a correlation coefficient as an index indicating the relationship between the feature quantity extracted from the face region and the face feature quantity 142a of the specific individual, for example, as an index indicating correlation, and calculates the calculated correlation coefficient. , it may be determined whether the face in the face area is the face of a specific individual. If the correlation coefficient is greater than a predetermined threshold, the face in the detected face region is determined to be the face of a specific individual. may be determined not to be the face of

Further, the specific individual determination unit 25 may determine whether or not the face in the detected face area is the face of a specific individual based on the determination result for one frame of the input image from the camera 11, It may be determined whether or not the face in the detected face area is the face of a specific individual, based on the determination results for a plurality of frames of the input image from the camera 11 .

The first face image processing unit 26 performs face image processing for a specific individual when the specific individual determining unit 25 determines that the face is that of a specific individual. The first face image processing unit 26 includes a specific individual's face orientation estimation unit 27, a specific individual's eye open/closed detection unit 28, and a specific individual's gaze direction estimation unit 29. A configuration for estimating or detecting facial behavior may also be included. Further, the first face image processing unit 26 may perform any one of face image processing for a specific individual using the face feature amount 142a of the specific individual. Further, a learned feature amount that has undergone learning for performing face image processing for a specific individual is stored in the face feature amount storage unit 142, and the learned feature amount is used to perform facial image processing for a specific individual. Any processing of face image processing may be performed.

The specific individual's face direction estimating unit 27 performs processing for estimating the face direction of the specific individual. The specific individual face direction estimation unit 27 detects the positions and shapes of facial features such as eyes, nose, mouth, and eyebrows from the face area detected by the face detection unit 22, and detects the positions and shapes of the detected facial features. Based on this, the process of estimating the orientation of the face is performed.

A method for detecting facial features from a facial region in an image is not particularly limited, but it is preferable to employ a method that can detect facial features at high speed and with high accuracy. For example, a method of creating a three-dimensional face shape model, fitting it to a face region on a two-dimensional image, and detecting the position and shape of each organ of the face can be adopted. As a technique for fitting a 3D face shape model to a human face in an image, for example, the technique described in Japanese Patent Application Laid-Open No. 2007-249280 can be applied, but it is not limited to this.

Further, the specific individual's face orientation estimation unit 27 uses, as estimation data of the specific individual's face orientation, for example, the pitch angle of vertical rotation (around the X axis), which is included in the parameters of the three-dimensional face shape model, A yaw angle for left-right rotation (around the Y-axis) and a roll angle for overall rotation (around the Z-axis) may be output.

The specific individual's eye open/close detection unit 28 performs processing for detecting the open/closed state of the eye of the specific individual. The specific individual's eye open/close detection unit 28, for example, based on the position and shape of the facial organs obtained by the specific individual's face direction estimation unit 27, particularly the position and shape of the eye feature points (eyelid, pupil), Detects the open/closed state, for example, whether the eyes are open or closed. The open/closed state of the eyes can be determined by, for example, using a learner in advance for the feature values of eye images (eyelid position, shape of pupil (black eye), size of white and black eye portions, etc.) in various eye open/closed states. It may also be detected by learning using and evaluating the degree of similarity with these learned feature amount data.

The specific individual's line-of-sight direction estimating unit 29 performs processing for estimating the direction of the specific individual's line of sight. The sight line direction estimating unit 29 of the specific individual, for example, based on the orientation of the face of the driver 3 and the position and shape of the facial organs of the driver 3, especially the position and shape of the characteristic points of the eyes (outer corners, inner corners, pupils), Estimate the direction of the line of sight. The line-of-sight direction is the direction in which the driver 3 is looking, and is determined by, for example, a combination of the direction of the face and the direction of the eyes.

In addition, the direction of the line of sight is, for example, the feature amount of the image of the eyes (the relative positions of the outer corner, the inner corner, and the pupil, or the relative positions of the white and black portions of the eye, shading, light and shade, etc.) in various combinations of face orientation and eye orientation. texture, etc.) may be learned in advance using a learning device, and the degree of similarity with the learned feature amount data may be evaluated for detection. In addition, the sight line direction estimation unit 29 of the specific individual estimates the size and center position of the eyeball from the size and orientation of the face, the position of the eyes, etc. using the fitting result of the three-dimensional face shape model, etc. , and a vector connecting the center of the eyeball and the center of the pupil may be detected as the line-of-sight direction.

When the specific individual determination unit 25 determines that the face is not that of a specific individual, the second face image processing unit 30 performs normal face image processing. The second face image processing unit 30 includes a normal face orientation estimation unit 31, a normal eye open/close detection unit 32, and a normal gaze direction estimation unit 33. It may also include an estimating or detecting configuration. Further, the second face image processing section 30 may perform any of normal face image processing using the normal face feature amount 142b. Also, a learned feature amount that has undergone learning for normal face image processing is stored in the face feature amount storage unit 142, and normal face image processing is performed using the learned feature amount. Either processing may be performed. The processing performed by the normal face orientation estimation unit 31, the normal eye open/closed detection unit 32, and the normal gaze direction estimation unit 33 is performed by the specific individual's face orientation estimation unit 27 and the specific individual's eye open/closed detection. Since the unit 28 is basically the same as the specific individual's line-of-sight direction estimation unit 29, the description thereof will be omitted here.

The output unit 34 performs processing for outputting information based on image processing by the image processing unit 12 to the ECU 40 or the like. The information based on the image processing may be, for example, information related to the facial behavior of the driver 3, such as the orientation of the face, the direction of the line of sight, or the open/closed state of the eyes, or the driver 3 determined based on the detection result of the facial behavior. (for example, states such as looking ahead, looking aside, dozing off, looking backward, lying down, etc.). Information based on image processing may be a predetermined control signal (a control signal for performing attention or warning processing, or a control signal for controlling the operation of the vehicle 2, etc.) based on the state determination of the driver 3.

[Processing operation example]
FIG. 7 is a flowchart showing an example of processing operations performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. The camera 11 captures, for example, several tens of frames per second, and this process is performed for each frame or for each frame at regular intervals.

First, in step S1, the CPU 13 operates as the image input unit 21, reads an image captured by the camera 11 (an image including the face of the driver 3), and proceeds to step S2.

In step S2, the CPU 13 operates as the face detection unit 22, performs face detection processing for detecting a face region while scanning the search region of the input image, and proceeds to step S3. A specific example of the face detection process in step S2 will be described later.

In step S3, the CPU 13 operates as the specific individual determining unit 25, and uses the feature amount of the face area detected in step S2 and the specific individual's facial feature amount 142a read from the facial feature amount storage unit 142, A process of determining whether or not the face in the detected face area is the face of a specific individual is performed, and the process proceeds to step S4.

In step S4, the CPU 13 determines whether or not the result of the determination processing in step S3 is the face of a specific individual.

In step S5, the CPU 13 operates as the face direction estimation unit 27 of the specific individual, and detects, for example, the positions and shapes of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S2. The orientation of the face is estimated based on the position and shape of the facial features, and the process proceeds to step S6.

In step S6, the CPU 13 operates as the eye opening/closing detection unit 28 of the specific individual, and, for example, based on the position and shape of the facial features obtained in step S5, particularly the position and shape of the characteristic points of the eyes (eyelids, pupils). , the open/closed state of the eyes, for example, whether the eyes are open or closed, is detected, and the process proceeds to step S7.

In step S7, the CPU 13 operates as the gaze direction estimating unit 29 of the specific individual, for example, the direction of the face obtained in step S5, the position and shape of the facial organs, especially the characteristic points of the eyes (outer corners, inner corners, pupils). Based on the position and shape, the direction of the line of sight is estimated, and then the process ends.

On the other hand, if the CPU 13 determines in step S4 that the face is not that of a specific individual, in other words, that it is a normal face, the process proceeds to step S8.
In step S8, the CPU 13 operates as a normal face direction estimation unit 31, for example, detects the positions and shapes of facial features such as eyes, nose, mouth, and eyebrows from the face area detected in step S2, and detects the detected face. The orientation of the face is estimated based on the position and shape of the organ, and the process proceeds to step S9.

In step S9, the CPU 13 operates as a normal eye open/close detection unit 32. For example, based on the position and shape of the facial features obtained in step S8, particularly the position and shape of the characteristic points of the eyes (eyelids, pupils), The open/closed state of the eyes, for example, whether the eyes are open or closed is detected, and the process proceeds to step S10.

In step S10, the CPU 13 operates as a normal gaze direction estimating unit 33, for example, the orientation of the face, the positions and shapes of facial organs, particularly the positions of eye feature points (outer corners, inner corners, pupils) obtained in step S8. Based on the shape and shape, the direction of the line of sight is estimated, and then the process ends.

FIG. 8 is a flowchart showing an example of processing operations performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. This processing operation is an example of the face detection processing operation of step S2 and the specific individual determination processing operation of step S3 shown in FIG. 7, and is an example of processing operation for one input image (one frame).

First, in step S21, the CPU 13 starts loop processing L1 for detecting the face size (magnitude), and in the next step S22, starts loop processing L2 for detecting the rotation angle (orientation or inclination) of the face. , in the next step S23, a loop process L3 for detecting the position of the face is started, and the process proceeds to step S24.

The loop process L1 is repeated according to the number of reduced images (eg, 20a, 20b shown in FIG. 5) generated to detect faces of various sizes. The loop processing L2 is repeated according to the setting of the discriminator that discriminates the rotation angle of the face (for example, front face, oblique face, side face, tilt shown in FIG. 6). The loop process L3 is repeated as many times as the search area 210 is scanned to detect the position of the face.

In step S24, the CPU 13 operates as the first feature amount extraction unit 221, and performs processing for extracting the facial feature amount 221a from the search area 210 under each condition of the loop processes L1, L2, and L3.

In step S25, the CPU 13 operates as the normal face discriminator 222 and the specific individual face determination unit 223, and hierarchically determines whether the search area 210 is a face (is a face) or a non-face (other than a face). , and if it is determined that the search area 210 is a non-face in any hierarchy, a process of determining whether the search area 210 is the face of a specific individual or a non-face is performed.

For example, the CPU 13 scans the search area 210 for all the reduced images, and after completing face detection (in other words, detection of face area candidates) in all the search areas 210, performs loop processing in step S26. After completing L1, loop processing L2 is completed in step S27, loop processing L3 is completed in step S28, and then the process proceeds to step S29.

In step S29, the CPU 13 operates as the face area integration unit 224, performs processing to integrate one or more face area candidates determined to be faces in the processes of steps S21 to S28, and proceeds to step S30. . The method of integrating one or more face area candidates is not particularly limited. For example, integration may be performed based on an average value of area centers of one or more face area candidates and an average value of area sizes.

In step S30, the CPU 13 operates as the second feature amount extraction unit 225, performs processing for extracting facial feature amounts from the face area integrated in step S29, and proceeds to step S31.

In step S31, the CPU 13 operates as the specific individual determination unit 25, and in step S30, the feature quantity extracted from the integrated face region and the face feature quantity 142a of the specific individual read from the face feature quantity storage unit 142. , and the process proceeds to step S32.

In step S32, the CPU 13 determines whether the calculated correlation coefficient is greater than a predetermined threshold value for determining whether or not the face is of a specific individual. If it is determined that the feature quantity extracted from the face region and the face feature quantity 142a of the specific individual are highly correlated (in other words, the degree of similarity is high), the process proceeds to step S33.
In step S33, the CPU 13 determines that the face detected in the face area is the face of a specific individual, and then terminates the process.

On the other hand, in step S32, if the correlation coefficient is equal to or less than a predetermined threshold, in other words, the correlation between the feature quantity extracted from the face region and the face feature quantity 142a of the specific individual is low (in other words, the similarity is low), the process proceeds to step S34.
In step S34, the CPU 13 determines that the face is not that of a specific individual, in other words, that it is a normal face, and then terminates the process.

FIG. 9 is a flow chart showing an example of the face detection processing operation performed by the CPU 13 of the image processing section 12 in the driver monitoring device 10 according to the embodiment. This processing operation is an example of the determination processing operation of step S25 shown in FIG.

First, in step S41, the CPU 13 reads the feature amount 221a extracted from the search area 210 in step S24 of FIG. 8, and advances the process to step S42.
In step S42, the CPU 13 sets 0 to a discriminator counter (i) that counts in which layer the normal face discriminator 222 is located, and n, which indicates the number of hierarchical structures of the normal face discriminator 222, is set to , 1 is set as an initial value, and the process proceeds to step S43.

In step S43, the CPU 13 operates as the normal face discriminator 222, and performs processing for discriminating whether the search area 210 is a face or a non-face by the n-th discriminator.

In step S44, the CPU 13 determines whether the search area 210 is a face or a non-face, and if determined to be a face, proceeds to step S45.
In step S45, the CPU 13 adds 1 to the discriminator counter (i), and proceeds to step S46.

In step S46, the CPU 13 determines whether or not the discriminator counter (i) is less than N, which indicates the number of discriminators. , in step S47, 1 is added to n in order to proceed to the processing by the discriminator of the next layer, and then the processing returns to step S43 to repeat the processing.

On the other hand, in step S46, if the CPU 13 determines that the discriminator counter (i) is not less than the discriminator number N, in other words, if the discriminator counter (i) has reached the discriminator number N, the process proceeds to step S48. proceed.

In step S48, the CPU 13 determines that the search area 210 is a candidate for the face area, stores the information of the search area as a candidate for the face area, finishes the face detection process for the search area, and then selects the next search area. Repeat the face detection process for .

On the other hand, if the CPU 13 determines in step S44 that the search area 210 is non-face, the process proceeds to step S49.
In step S49, the CPU 13 operates as the specific individual face determination unit 223, and uses the feature amount (the feature amount extracted from the search area 210) used in the n-th discriminator and the specific individual corresponding to the hierarchy of the n-th discriminator. , and the processing proceeds to step S50.

In step S50, the CPU 13 determines whether the calculated correlation coefficient is greater than a predetermined threshold value for determining whether or not the face is of a specific individual, and determines whether the correlation coefficient is greater than the predetermined threshold value. (That is, if the correlation is high) (in other words, if it is determined that the search area 210 is the face of a specific individual), the process proceeds to step S45, and the discrimination process by the normal face discriminator 222 proceeds. .

On the other hand, in step S50, if the CPU 13 determines that the correlation coefficient is equal to or less than a predetermined threshold value (that is, the correlation is low) (in other words, if it determines that the search area 210 is non-face), step The process proceeds to S51.

In step S51, the CPU 13 determines that the search area is a non-face (other than a face), and in the next step S52, terminates the discrimination processing after the n-th discriminator by the normal face discriminator 222, and then the search is performed. After finishing the face detection processing for the region, the face detection processing for the next search region is repeated.

[Action/effect]
According to the driver monitoring device 10 according to the above-described embodiment, the facial feature amount storage unit 142 stores the facial feature amount 142a of the specific individual and the normal facial feature amount 142b as the learned facial feature amount, The normal face discriminator 222 uses the normal face feature quantity 142b to hierarchically discriminate whether the search region 210 cut out from the image is a face or a non-face, thereby detecting a face region. be.

In addition, even if the normal face discriminator 222 determines that it is a non-face at any level, the specific individual face determination unit 223 uses the specific individual's face feature amount 142a to determine whether the search area 210 is A face area containing the specific individual's face is detected by determining whether it is the specific individual's face or not.

As a result, it is possible to accurately detect a face area from an image, whether it is a normal face or a specific individual's face. In addition, since the normal face discriminator 222 and the specific individual face determination unit 223 use the common feature amount extracted from the search area 210, there is no need for a separate process of extracting the feature amount, so that the face area can be detected. Such real-time property can be maintained.

Therefore, regardless of whether the driver 3 is a specific individual or a normal person other than the specific individual, each face can be accurately detected in real time (in other words, by high-speed processing).

In addition, the feature amount used in one layer of the normal face discriminator 222, which is determined to be a non-face by the specific individual face determination unit 223, and the face feature amount 142a of the specific individual corresponding to the one layer. Based on the correlation, it is possible to efficiently determine whether the search area is the face of a specific individual or not. Therefore, even when the normal face discriminator 222 discriminates the face as a non-face, it is possible to accurately determine the case as the face of a specific individual.

Further, according to the driver monitoring device 10, when the specific individual face determination unit 223 determines that the face is that of a specific individual, the determination proceeds to the next layer of the normal face discriminator 222, and the determination process can be performed quickly. Continued. On the other hand, when the specific individual face determining unit 223 determines that the face is a non-face, the determination by the normal face determining unit 222 is terminated. Therefore, while maintaining the efficiency of the normal face discriminator 222, it is possible to perform the process of discriminating the face of a specific individual.

Further, according to the driver monitoring device 10, the facial area integration unit 224 integrates one or more facial area candidates determined to be a face through the normal face discriminator 222, and the specific individual determination unit 25 , using the feature quantity extracted from the integrated face region and the face feature quantity 142a of the specific individual, it is determined whether or not the face in the face region is the face of the specific individual. Therefore, based on the face area integrated by the face area integration unit 224, it is possible to accurately determine whether the face is that of a specific individual or that of an ordinary person.

The in-vehicle system 1 also includes a driver monitoring device 10 and one or more ECUs 40 that execute predetermined processing based on the monitoring results output from the driver monitoring device 10 . Therefore, it is possible to cause the ECU 40 to appropriately perform predetermined control based on the monitoring result. This makes it possible to build a highly safe vehicle-mounted system that allows even specific individuals to drive with peace of mind.

[Modification]
Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various modifications and changes can be made without departing from the scope of the invention.
In the above embodiment, the case where the image processing device according to the present invention is applied to the driver monitoring device 10 has been described, but application examples are not limited to this. For example, in devices and systems that monitor people who operate and monitor various facilities such as machines and devices in a factory, and perform predetermined work, the above-mentioned specific individuals are included in the monitoring targets. In this case, the image processing apparatus according to the present invention can be applied.

[Appendix]
Embodiments of the present invention can also be described in the following appendices, but are not limited thereto.
(Appendix 1)
An image processing device (12) for processing an image input from an imaging unit (11),
A facial feature amount storage unit that stores a specific individual's facial feature amount (142a) and a normal facial feature amount (142b) as learned facial feature amounts that have been trained to detect a face from the image. (142) and
a face detection unit (22) for detecting a face area while scanning a search area of the image;
The face detection unit (22)
a first feature quantity extraction unit (221) for extracting a face feature quantity from the search area;
A hierarchical normal face discriminator for discriminating whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount (142b). (222) and
When the normal face discriminator (222) determines that the face is non-face at any level, using the feature quantity extracted from the search area and the face feature quantity of the specific individual, An image processing apparatus, comprising: a specific individual face determination unit (223) that determines whether the search area is the face of the specific individual or the non-face.

(Appendix 2)
An image processing method for processing an image input from an imaging unit (11),
including a face detection step (S2) of detecting a face region while scanning a search region of the image;
The face detection step (S2) is
a feature quantity extraction step (S24) for extracting a face feature quantity from the search area;
The search area is determined to be a face using the feature amount extracted by the feature amount extraction step (S24) and a learned normal face feature amount (142b) that has been trained to detect a face. a normal face discrimination step (S43, S44) for hierarchically discriminating whether or not it is a non-face;
Learning for detecting the extracted feature quantity and the face of a specific individual when the normal face determination step (S43, S44) determines that the face is non-face at any one of the hierarchies. a specific individual face determination step (S49, S50) for determining whether the search area is the face of the specific individual or the non-face using the facial feature amount (142a) of the specific individual that has already been processed; An image processing method comprising:

1 in-vehicle system 2 vehicle 3 driver 10 driver monitoring device 11 camera 12 image processing unit 13 CPU
14 ROMs
141 program storage unit 142 facial feature amount storage unit 142a specific individual facial feature amount 142b normal facial feature amount 15 RAM
151 image memory 16 communication unit 20 image (input image)
20a, 20b Reduced image 21 Image input unit 22 Face detection unit 210 Search area 221 First feature amount extraction unit 221a Feature amount 222 Normal face discriminator 223 Specific individual face determination unit 224 Face area integration unit 225 Second feature amount extraction unit 25 Specific individual determination unit 26 First face image processing unit 27 Specific individual face direction estimation unit 28 Specific individual eye open/close detection unit 29 Specific individual gaze direction estimation unit 30 Second face image processing unit 31 Normal face direction estimation unit 32 Normal eye open/close detection unit 33 Normal line-of-sight direction estimation unit 34 Output unit 40 ECU
41 sensor 42 actuator 43 communication bus

Claims (10)

An image processing device that processes an image input from an imaging unit,
a facial feature amount storage unit that stores a specific individual's facial feature amount and a normal facial feature amount as learned facial feature amounts that have been trained to detect a face from the image;
a face detection unit that detects a face area while scanning a search area of the image;
The face detection unit
a first feature quantity extraction unit that extracts a face feature quantity from the search area;
a hierarchical normal face discriminator that discriminates whether the search area is a face or a non-face by using the feature amount extracted from the search area and the normal face feature amount;
When the face is determined to be a non-face in any hierarchy of the normal face classifier, the search region is determined using the feature quantity extracted from the search region and the face feature quantity of the specific individual. and a specific individual face determination unit that determines whether the is the face of the specific individual or the non-face of the specific individual. The specific individual face determination unit
calculating an index indicating a correlation between the feature amount used in one layer of the normal face discriminator determined as the non-face and the face feature amount of the specific individual corresponding to the one layer;
2. The image processing apparatus according to claim 1, wherein, based on the calculated index, it is determined whether the search area is the face of the specific individual or the non-face. The specific individual face determination unit
determining that the search area is the face of the specific individual when the index is greater than a predetermined threshold;
3. The image processing apparatus according to claim 2, wherein the search area is determined to be the non-face when the index is equal to or less than the predetermined threshold. a determination progressing unit that advances determination to the next layer of the normal face classifier when the specific individual face determining unit determines that the face is that of the specific individual;
4. The method according to any one of claims 1 to 3, further comprising a determination terminating unit that terminates determination by the normal face discriminator when the specific individual face determining unit determines that the face is a non-face. 10. The image processing device according to claim 1. The face detection unit is
a face area integration unit that integrates one or more candidates for the face area determined to be the face by the normal face determiner;
a second feature quantity extraction unit that extracts a face feature quantity from the integrated face region;
a specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual, using the feature quantity extracted from the integrated face region and the face feature quantity of the specific individual; The image processing apparatus according to any one of claims 1 to 4, characterized by comprising: An image processing device according to any one of claims 1 to 5;
an imaging unit that captures an image to be input to the image processing device;
and an output unit for outputting information based on image processing by the image processing device. a monitoring device according to claim 6;
A control system comprising one or more control devices communicably connected to the monitoring device and executing predetermined processing based on the information output from the monitoring device. the monitoring device is a device for monitoring a driver of a vehicle;
8. The control system of claim 7, wherein said controller includes an electronic control unit mounted on said vehicle. An image processing method for processing an image input from an imaging unit,
A face detection step of detecting a face region while scanning a search region of the image;
The face detection step includes:
a feature quantity extraction step of extracting a face feature quantity from the search area;
The search area is determined to be a face or a non-face by using the feature quantity extracted by the feature quantity extraction step and a learned normal face feature quantity that has been trained to detect a face. a normal face discrimination step for hierarchically discriminating whether
The feature amount extracted when the non-face is discriminated in any hierarchy of the normal face discriminating step, and the learned specific individual who has undergone learning to detect the face of the specific individual. and a specific individual face determination step of determining whether the search area is the face of the specific individual or the non-face using the facial feature amount of . A program for causing at least one or more computers to process an image input from an imaging unit,
to said at least one computer;
A face detection step of detecting a face region while scanning a search region of the image;
The face detection step includes:
a feature quantity extraction step of extracting a face feature quantity from the search area;
The search area is determined to be a face or a non-face by using the feature quantity extracted by the feature quantity extraction step and a learned normal face feature quantity that has been trained to detect a face. a normal face discrimination step for hierarchically discriminating whether
The feature amount extracted when the non-face is discriminated in any hierarchy of the normal face discriminating step, and the learned specific individual who has undergone learning to detect the face of the specific individual. and a specific individual face determination step of determining whether the search area is the face of the specific individual or the non-face using the facial feature amount of .

Images