WO2014061175A1

WO2014061175A1 - State monitoring device

Info

Publication number: WO2014061175A1
Application number: PCT/JP2013/003046
Authority: WO
Inventors: 泰斗渡邉
Original assignee: 株式会社デンソー
Priority date: 2012-10-15
Filing date: 2013-05-13
Publication date: 2014-04-24
Also published as: JP2014082585A

Abstract

This state monitoring device (100) is mounted in a vehicle (1) and monitors the state of a driver using a facial image (51) of the driver. The state monitoring device emits illumination light from a projection unit (15) towards the face surface of the driver presumed to be positioned in a stipulated region (PA). The state monitoring device acquires a first face image (51) captured by receiving incoming light coming in from the stipulated region, and a second face image (52) captured by receiving attenuated light resulting from the light from the red region of the incoming light having been attenuated. Here, when a plurality of bright sections (55, 56) that are candidates for the eyes of the driver have been captured in the first face image, the bright sections not captured in the second face image among the plurality of bright sections are determined to be eyes, and on the basis of the locations of the determined eyes, face recognition is performed.

Description

Condition monitoring device

Cross-reference of related applications

This disclosure is based on Japanese Patent Application No. 2012-228178 filed on October 15, 2012, the contents of which are incorporated herein by reference.

The present disclosure relates to a technique for monitoring the state of a driver using a face image that is mounted on a vehicle and that captures the face of the driver who controls the vehicle.

Conventionally, a state monitoring device mounted on a vehicle performs face recognition using a face image obtained by photographing a driver's face in order to monitor the state of the driver. In order to perform such face recognition, the state monitoring device has a configuration in which light is emitted toward a prescribed region that is defined in advance as a region where the operator's face is located, and light that is incident from the prescribed region is received. And a configuration for recognizing the face based on the position of the eye in the captured face image.

The image processing apparatus of Patent Document 1, which is a kind of configuration for recognizing a face as described above, performs face recognition using red eyes. More specifically, in a face image shot with flash emission, a human eye can appear as a red bright part. Such a red eye phenomenon is a phenomenon that occurs mainly because light in the red band of the flash light is reflected by the blood vessels of the retina. This red eye acquires high contrast with respect to the surroundings of the red eye. Therefore, the image processing apparatus can accurately specify the position of the eye in the face image based on the position of the red bright part.

JP 2003-30647 A

Now, the inventor of the present disclosure has attempted to employ the above-described face recognition using red eyes in a state monitoring device. Then, the following was found. That is, when a pilot wearing a spectacle is photographed while illuminating a specified area with a light emitting unit such as a flash in a state with little ambient light, the photographed face image includes not only the driver's red eyes. The light emitting part reflected in the glasses is photographed as a bright part. As described above, if a plurality of bright portions that are candidate eyes are photographed around the eyes of the driver, the specification of the position of the eyes in the face image may be incorrect. Then, the accuracy of face recognition performed based on the position of the eyes may not be ensured.

An object of the present disclosure is to provide a state monitoring device that can ensure the accuracy of face recognition performed based on the position of the eye.

In order to achieve the above object, as a first aspect of the present disclosure, a state monitoring device that is mounted on a vehicle and monitors the state of the driver using a face image obtained by photographing the face of the driver who controls the vehicle is provided. Provided as follows. A light emitting unit that emits light in a red to near-infrared band toward a prescribed region that is defined in advance as a region where the face is located, and a first face that is photographed by receiving incident light incident from the prescribed region A first image acquisition unit that acquires an image as a face image, and a second face image that is captured by receiving attenuated light obtained by attenuating red band light from incident light is separated from the first face image. When a plurality of bright parts that are candidates for the pilot's eyes and a plurality of bright parts that are candidates for the driver's eyes are photographed in the first face image, the second face image is photographed in the plurality of bright parts. It is characterized by comprising an eye determination unit that determines that a bright part that is not an eye is an eye, and a face recognition unit that recognizes a face based on the position of the eye determined by the eye determination unit.

According to a second aspect of the present disclosure, there is provided a state monitoring method for causing a computer to execute a process for monitoring a state of a driver mounted on a vehicle and using a face image obtained by photographing a face of the driver who controls the vehicle. The first step is taken by emitting a light emitting step for emitting light in a band from red to near-infrared toward a prescribed region defined in advance as a region where the face is located, and receiving incident light incident from the prescribed region. A first image acquisition step of acquiring a face image as a face image, and a second face image captured by receiving attenuated light obtained by attenuating light in the red band from incident light is referred to as a first face image. Second image acquisition step to acquire as another face image, and when a plurality of bright parts that are candidates for the pilot's eyes are photographed in the first face image, the second face image is photographed among the plurality of bright parts The bright part that is not Comprising the eye determination step of constant, based on the position of the eye is determined by eye determination step, and recognizing the face recognition step the face, the.

As a third aspect of the present disclosure, the storage medium is a non-transition storage medium, and the medium includes instructions that are read and executed by a computer, and the instructions are the state monitoring method according to the second aspect described above. And the method is computer mounted.

According to these above viewpoints, in the second face image taken by receiving the attenuated light, the red band light is attenuated from the incident light, so the pilot's eyes appear as a bright part. hard. Therefore, when a plurality of bright parts that are candidates for the eyes of the pilot are photographed in the first face image, by determining those of these bright parts that are not photographed in the second face image as eyes, Even when the light emitting part is reflected on the driver's glasses, the position of the eye in the face image can be accurately specified. Therefore, the accuracy of face recognition performed based on the eye position can be ensured.

The above object and other objects, features, and advantages of the present disclosure will become clearer by the following detailed description with reference to the accompanying drawings.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings.
It is a figure for demonstrating arrangement | positioning in the vehicle of the state monitoring apparatus by 1st embodiment of this indication, It is a block diagram for demonstrating the electrical structure of the state monitoring apparatus by 1st embodiment, It is a perspective view for explaining the mechanical configuration of the state monitoring device, It is a figure which shows an example of a 1st face image typically, It is a schematic diagram for explaining the configuration of the image sensor, It is a figure which shows an example of a 2nd face image typically, It is a flowchart which shows the process implemented by the image recognition part, It is a block diagram for demonstrating the electrical structure of the state monitoring apparatus by 2nd embodiment, It is a block diagram for demonstrating the electrical structure of the state monitoring apparatus by 3rd embodiment, It is a schematic diagram which shows the modification of FIG.

Hereinafter, a plurality of embodiments of the present disclosure will be described with reference to the drawings. In addition, the overlapping description may be abbreviate | omitted by attaching | subjecting the same code | symbol to the corresponding component in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other part of the configuration. In addition, not only combinations of configurations explicitly described in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined even if they are not explicitly specified unless there is a problem with the combination. . Combinations that are not explicitly described between the configurations described in the plurality of embodiments are also disclosed by the following description.

As shown in FIG. 1, the state monitoring device 100 according to the first embodiment of the present disclosure is mounted on a vehicle 1 as a moving body, and indicates the state of a driver (hereinafter also referred to as a driver) that drives or controls the vehicle. It is a driver status monitor to be monitored. As shown in FIG. 2, the state monitoring apparatus 100 includes an imaging unit 10, a light emitting unit 15, a control circuit 20, and a housing 60 (see FIG. 3) that houses these configurations. The state monitoring device 100 is connected to an actuation unit 90 and a vehicle control device 96 mounted on the vehicle.

The imaging unit 10 shown in FIGS. 1 and 2 is a device that generates a face image 51 (see also FIG. 4) obtained by photographing a driver's face in the state monitoring device 100 installed on the upper surface of the steering column 81. The state monitoring device 100 installed on the upper surface of the steering column 81 is a device that generates a face image 51 (see also FIG. 4) obtained by photographing the driver's face. The image capturing unit 10 captures a predetermined area PA defined in advance in the vehicle 1. This prescribed area PA includes an area where the face of the driver who is seated in the driver's seat is assumed to be located. Specifically, such a defined area PA is defined based on the eyelips assumed from the eye range of each eye of the driver, and is defined to include, for example, the 99th percentile of the eyelips.

The imaging unit 10 is a so-called near-infrared camera, and is configured by combining the imaging element 11 with an optical lens, an optical filter, and the like. The imaging element 11 generates an electrical signal corresponding to the intensity of received light by a plurality of pixels arranged along the imaging surface. The image sensor 11 is arranged in a posture in which the imaging surface is directed to the defined area PA. The image sensor 11 is in an exposure state based on a control signal from the control circuit 20 and receives incident light incident from the defined area PA. As a result, a monochrome face image 51 drawn with shades of white and black is generated. The face images 51 thus photographed are sequentially output from the imaging unit 10 to the control circuit 20.

The light projecting unit 15 has a plurality of light emitting diodes 16. Each light emitting diode 16 is disposed so as to sandwich the imaging unit 10 (see FIG. 3), and emits illumination light in a band from red to near infrared toward the defined area PA. The on state and the off state of light emission of the light emitting diode 16 are controlled by a current supplied from the control circuit 20.

The control circuit 20 is connected to the imaging unit 10, the light projecting unit 15, the actuation unit 90, and the like, and is a circuit that controls the operation of these components. The control circuit 20 is mainly configured by a microcomputer including a processor that performs various arithmetic processes, a RAM that functions as a work area for the arithmetic processes, and a flash memory that stores programs used for the arithmetic processes. . In addition, the control circuit 20 includes a power supply circuit that supplies power to the imaging unit 10, the light projecting unit 15, and the like.

The control circuit 20 executes a plurality of functions such as a light emission control unit 21, an imaging control unit 23, an image recognition unit 24, a state determination unit 31, and a warning control unit 33 by executing a state monitoring program stored in advance by a processor. With blocks. This functional block is also referred to as a functional section.

The light emission control unit 21 is a functional block related to the light emission control of the light projecting unit 15. The light emission control unit 21 causes the light emitting diode 16 to emit light by applying a predetermined current to the light emitting diode 16. Based on the control value calculated by the image recognition unit 24, the light emission control unit 21 causes the light projecting unit 15 to emit illumination light in accordance with the timing at which the imaging unit 10 is in the exposure state.

The imaging control unit 23 is a functional block related to imaging control of the imaging unit 10. The imaging control unit 23 controls the exposure start timing, gain, exposure time, and the like in the imaging unit 10 based on the control value calculated by the image recognition unit 24.

The image recognition unit 24 is a functional block related to image processing of the face image 51 and the like. The image recognition unit 24 sets an imaging condition in the imaging unit 10 and a light emission condition in the light projecting unit 15 in order to acquire a face image 51 from which the driver's face can be extracted. The image acquisition unit 21 controls the imaging control unit 23 and the light emission control unit 21 to control the imaging unit 10 and the light projecting unit 15 in order to cause the imaging unit 10 and the light projecting unit 15 to perform operations in accordance with the set imaging conditions and light emission conditions. Calculate the value. The image recognition unit 24 acquires the face image 51 thus photographed from the imaging unit 10. The image recognizing unit 24 performs image processing on the acquired face image 51 to obtain values related to the driver's face orientation and the degree of eye opening (hereinafter referred to as “eye open degree”) and the degree of sleepiness of the driver. Calculate related values.

The state determination unit 31 compares the value calculated by the image acquisition unit 21 with a preset threshold value. By this comparison processing, the state determination unit 31 estimates whether, for example, a sign of driving aside or a sign of dozing operation is detected. And the state determination part 31 which detected the above-mentioned sign determines with the state which should alert a driver | operator having arisen.

The warning control unit 33 is connected to the actuation unit 90. The warning control unit 33 outputs a control signal to the actuation unit 90 when the state determination unit 31 determines that a situation that should warn the driver is occurring. Thus, the warning control unit 33 issues a warning to the driver by operating the actuation unit 90.

The housing 60 includes a main body member 63, a front cover member 66, a rear cover member (not shown), and the like as shown in FIG.

The main body member 63 holds the sub-board 62 on which the light projecting unit 15 and the imaging unit 10 are mounted. A main board 61 on which the control circuit 20 is formed is attached to the sub board 62 in a posture orthogonal to the sub board 62. The body member 63 is provided with an insertion hole 64 and a light distribution portion 65. The insertion hole 64 is provided in the central portion of the main body member 63 in the horizontal direction, and allows the imaging unit 10 mounted on the sub-board 62 to be inserted. The insertion hole 64 cooperates with a light blocking hole provided in the sub-substrate 66 to exhibit a light blocking function between the light projecting unit 15 and the imaging unit 10, thereby Light leakage to the imaging unit 10 is prevented. The light distribution unit 65 is disposed so as to sandwich the insertion hole 64 in the horizontal direction, and faces the light projecting unit 15 mounted on the sub-board 62. The light distribution unit 65 distributes light to the defined area PA (see FIG. 1) while transmitting the light emitted from the light projecting unit 15.

The front cover member 66 is provided with a visible light filter 67. The visible light filter 67 mainly transmits light in the red to near-infrared band used for generating the face image 51 (see FIG. 4) and shields light in the visible light band that is unnecessary for generating the face image 51. To do. The visible light filter 67 covers an opening 68 formed at a position facing the light distribution portion 65 in the front cover member 66. The rear cover member is disposed on the opposite side of the front cover member 66 with the main body member 63 interposed therebetween. The rear cover member covers the

substrates

61 and 62 to protect them from dust and dirt in the atmosphere.

2 includes, for example, a speaker 91, a seat vibration device 93, an air conditioner 95, and the like mounted on the vehicle 1 (see FIG. 1). The speaker 91 alerts the driver by reproducing audio data based on a control signal from the warning control unit 33. The seat vibration device 93 is installed inside the seat surface of the driver's seat or the like, and alerts the driver by vibrating the driver's seat based on a control signal from the warning control unit 33. The air conditioner 95 alerts the driver by an operation such as introducing outside air into the vehicle 1 based on a control signal from the warning control unit 33.

Next, the face image 51 photographed by the state monitoring device 100 will be described in more detail.

As shown in FIG. 4, when the driver's face is photographed while irradiating the prescribed area PA with illumination light under a low ambient light condition, both eyes of the driver appear as a bright portion 55. Such a so-called red-eye phenomenon occurs because light in the red band included in ambient light and illumination light is reflected by capillaries of the retina. This red eye acquires high contrast with respect to the surroundings of the red eye. Therefore, the image recognition unit 24 (see FIG. 2) can accurately specify the position of the eye in the face image 51 based on the position of the bright part 55 that appears in the face image 51.

However, it is assumed that a driver wearing spectacles was photographed while irradiating illumination light to the prescribed area PA in a state with little ambient light. In this case, not only the driver's so-called red eye but also the light-emitting diode 16 (see FIG. 2) reflected in the glasses is photographed as the bright portion 56 in the photographed face image 51. As described above, if a plurality of

bright portions

55 and 56 that are candidate eyes are photographed around both eyes of the driver, the specification of the eye position in the face image 51 may be erroneous. As a result, the accuracy of face recognition performed based on the position of the eyes may not be ensured.

In order to improve the accuracy of face recognition, the configuration of the state monitoring apparatus 100 in FIG. 2 and the processing performed by the control circuit 20 will be described in detail below.

As shown in FIG. 5, the imaging element 11 is formed with a covering region 77 and a non-covering region 76. The covering region 77 is a region covered with the color filter 78 on the imaging surface. The color filter 78 allows light in the near infrared band to pass through while attenuating light in the red band. Therefore, the covering region 77 receives the attenuated light that is incident from the defined region PA (see FIG. 1) and passes through the color filter 78 and in which the red band light is attenuated. The uncovered area 76 is located outside the covered area 77 and is not covered with the color filter 78. Therefore, the uncovered area 76 receives incident light incident from the defined area PA.

The image sensor 11 is a so-called VGA size element in which, for example, 640 × 480 pixels are arranged in each of the horizontal direction H and the vertical direction V. The image sensor 11 has a first pixel 70 and a second pixel 73 as a plurality of pixels. The first pixel 70 is a pixel that is not covered with the color filter 78. Therefore, the entire area of the first pixel 70 becomes the uncovered area 76. The second pixel 73 is a pixel covered with the color filter 78. Therefore, the entire area of the second pixel 73 becomes the covering region 77. The number of second pixels 73 provided on the imaging surface is smaller than the number of first pixels 70. Therefore, on the imaging surface of the imaging device 11, the area of the covered region 77 is narrower than the area of the non-covered region 76.

With the configuration of the imaging device 11 described above, based on the output from the first pixel 70, the imaging unit 10 shown in FIG. 2 receives a face image (hereinafter referred to as “first” for convenience) taken by receiving incident light. 51 ”(described as“ face image ”). The imaging unit 10 also captures a face image captured by receiving attenuated light based on the output from the second pixel 73 (see FIG. 5) (hereinafter referred to as “second face image” for convenience). 52 is generated. According to the above, the number of pixels of the first face image 51 is larger than the number of pixels of the second face image 52. Furthermore, the imaging unit 10 can acquire the output from the second pixel 73 while acquiring the output from the first pixel 70 (see FIG. 5). Therefore, the first face image 51 and the second face image 52 are images taken substantially simultaneously.

In order to realize highly accurate face recognition using the first face image 51 and the second face image 52, the image recognition unit 24 includes a first image acquisition block 25, a second image acquisition block 26, It has a dark place determination block 28 and an eye determination block 27.

The first image acquisition block 25 acquires the first face image 51 from the imaging unit 10. On the other hand, the second image acquisition block 26 acquires the second face image 52 from the imaging unit 10 as a face image different from the first face image 51.

The dark place determination block 28 determines whether or not the room of the vehicle 1 (see FIG. 1) is a dark place based on the information acquired from the vehicle control device 96. The vehicle control device 96 is a device that controls various devices mounted on the vehicle 1. The vehicle control device 96 can control the operation of the headlamp of the vehicle 1 as one of its functions. The dark place determination block 28 determines that the interior of the vehicle 1 is a dark place based on the state information of the headlamp of the vehicle 1 when the headlamp is in a lighting state. On the other hand, the dark place determination block 28 determines that the interior of the vehicle 1 is not a dark place when the headlamp is turned off. It should be noted that the switching between the lighting state and the unlighting state of the headlamp by the vehicle control device 96 may be performed based on the detection result of the external light sensor mounted on the vehicle 1 (see FIG. 1). It may be performed based on the operation state of the changeover switch provided in FIG.

The eye determination block 27 includes a plurality of

bright portions

55 and 56 that are candidates for the eyes when a plurality of

bright portions

55 and 56 that are candidates for the eyes of the driver shown in FIG. From this, it is determined which bright part corresponds to the actual eye. Specifically, as shown in FIG. 6, in the second face image 52 photographed by receiving the attenuated light, the red band light is attenuated from the incident light. It is hard to become eyes. That is, in the second face image 52, the driver's eyes are not easily captured as the bright portion 55 as shown in FIG. Therefore, among the

bright portions

55 and 56 that are candidates for the eyes photographed in the first face image 51, those not photographed in the second face image 52 in FIG. 5 are the bright portions 55 by the actual eyes. The probability is high.

Based on the principle described above, the eye determination block 27 shown in FIG. 2 performs a process of comparing the first face image 51 and the second face image 52, thereby a plurality of bright portions 55 appearing in the first face image 51. , 56 (see FIG. 4), the bright portion 55 that has not been photographed in the second face image 52 is determined as an eye. The determination of specifying the eye position is performed on the condition that the dark place determination block 28 determines that the room is dark.

Next, the process performed by the image recognition unit 24 in order to realize the determination for specifying the eye position described so far will be described in detail with reference to FIG. The process shown in FIG. 7 is started by the image recognition unit 24 when the ignition of the vehicle 1 (see FIG. 1) is turned on.

In S101, the light emission control unit 21 outputs a control signal for instructing the light projecting unit 15 to emit light, and the imaging control unit 23 outputs a control signal for supporting imaging to the imaging unit 10, and the process proceeds to S102. Based on the control signal output in S101, the light projecting unit 15 emits illumination light toward the defined area PA. And the imaging part 10 image | photographs the 1st face image 51 and the 2nd face image 52 by receiving the light from regulation area | region PA containing illumination light.

Here, the flowchart described in this application or the process of the flowchart is configured by a plurality of sections (or referred to as steps), and each section is expressed as S101, for example. Further, each section can be divided into a plurality of subsections, while a plurality of sections can be combined into one section. Further, each section configured in this manner can be referred to as a device, module, or means.

In S102, the first face image 51 and the second face image 52 based on the control signal output in S101 are acquired by the first image acquisition block 25 and the second image acquisition block 26, and the process proceeds to S103.

In S103, based on the headlight status information acquired from the vehicle control device 96, it is determined whether or not the interior of the vehicle 1 is in a dark place. If a positive determination is made in S103, the process proceeds to S105. On the other hand, if a negative determination is made in S103, the process proceeds to S104. In S104, face recognition processing without using red eyes is performed, and the process proceeds to S110. On the other hand, in S105, which is carried out on the condition that an affirmative determination is made in S103, the first face image 51 acquired in S102 is subjected to image processing, so that the driver's eye candidate is determined from the image 51. The

bright parts

55 and 56 are extracted, and the process proceeds to S106.

In S106, as a result of the image processing performed in S105, it is determined whether or not a plurality of

bright portions

55 and 56 that are candidates for the eye are photographed in a range preliminarily assumed as the vicinity region of the eye. If a negative determination is made in S106, the process proceeds to S107. In S107, face recognition is performed using the bright part 55, which is the red eye extracted in S105, and the process proceeds to S110. On the other hand, if a positive determination is made in S106, the process proceeds to S108.

In S108, by comparing the first face image 51 and the second face image 52, the second face image 52 is not photographed among the plurality of

bright portions

55 and 56 photographed in the first face image 51. The light part 55 is determined to be the eyes of the driver, and the process proceeds to S109. In S109, face recognition is performed using the bright portion 55 that is the red eye determined to be an eye in S108, and the process proceeds to S110.

In S110, the result of face recognition by any of S104, S107, and S109 is output to the state determination unit 31, and the process proceeds to S111. Based on the face recognition result output in S110, the state determination unit 31 determines whether or not there is a state in which the driver should be warned, such as a sign of driving aside or a sign of dozing. .

In S111, it is determined whether or not the ignition ON state of the vehicle 1 is continued. If a negative determination is made in S111 because the ignition is turned off, the process ends. On the other hand, if a positive determination is made in S111, the process returns to S101.

According to the first embodiment described so far, in the second face image 52 photographed by receiving the attenuated light, the red band light is attenuated from the incident light. It is hard to be seen as the bright part 55. Therefore, among the

bright portions

55 and 56 photographed in the first face image 51, the bright portion 55 that is not photographed in the second face image 52 is determined as the eye. Thereby, even when the light emitting diode 16 is reflected in the driver's glasses, the position of the eye in the first face image 51 can be accurately specified. Therefore, the accuracy of face recognition performed based on the eye position can be ensured.

In addition, in the image sensor 11 used in the first embodiment, the second pixel 73 which is a part of the image sensor 11 is covered with a color filter 78 as a covering region 77. Therefore, the imaging unit 10 can obtain both an output for generating the second face image 52 and an output for generating the first face image 51 from one image sensor 11. Therefore, a configuration in which a part of the image sensor 11 is covered with the color filter 78 is particularly suitable for the state monitoring apparatus 100 that identifies the eye position by comparing the first face image 51 and the second face image 52.

Further, the second face image 52 in the first embodiment is a face image that is mainly necessary for specifying the position of the eye. Therefore, the second face image 52 may not be as clear as the first face image 51. Therefore, the number of second pixels 73 in the image sensor 11 is smaller than the number of first pixels 70. Thereby, since the area of the covering region 77 is smaller than the area of the non-covering region 76, the first face image 51 based on the output from the non-covering region 76 maintains a high resolution and effectively uses the illumination light. It can be a clear image. Therefore, the accuracy of face recognition by the image recognition unit 24 using the first face image 51 can be reliably ensured.

Furthermore, according to the first embodiment, the imaging unit 10 can generate the first face image 51 and the second face image 52 based on outputs acquired at substantially the same timing. Therefore, the difference in photographing timing between the first face image 51 and the second face image 52 can be substantially eliminated. As described above, it is possible to avoid a situation in which the position of the photographed driver is shifted between the first face image 51 and the second face image 52. Therefore, the accuracy of the eye position specified by the comparison between the first face image 51 and the second face image 52, and hence the accuracy of face recognition, can be further ensured.

In addition, in the first embodiment, the eye position is identified by comparing the first face image 51 and the second face image 52 on the condition that the interior of the vehicle 1 is dark. . As described above, the bright portion 55 due to red eyes and the bright portion 56 due to the reflection of the light emitting diodes 16 on the glasses are both reflected in the first face image 51 mainly under a condition with little ambient light. Therefore, according to the condition that the position of the eye is specified on the condition that the room is in a dark place, the processing load in the state monitoring device 100 is maintained while maintaining high face recognition accuracy. Can be reduced.

In the first embodiment, the vehicle 1 is also referred to as a moving body. The imaging unit 10 is also referred to as an imaging device or imaging means. The light projecting unit 15 is also referred to as a light emitting unit, a light emitting device, or a light emitting means. The image recognition unit 24 is also referred to as a face recognition unit, a face recognition device, or a face recognition means. The first image acquisition block 25 is also referred to as a first image acquisition unit, a first image acquisition device, or a first image acquisition means. The second image acquisition block 26 is also referred to as a second image acquisition unit, a second image acquisition device, or a unit second image acquisition means. The eye determination block 27 is also referred to as an eye determination unit, an eye determination device, or an eye determination means. The dark place determination block 28 is also referred to as a dark place determination unit, a dark place determination device, or a dark place determination means. The color filter 78 is also referred to as an attenuation filter. S101 is also referred to as a light emission section or a light emission step. S102 is also referred to as a first image acquisition section or first image acquisition step and a second image acquisition section or second image acquisition step. S108 is also referred to as an eye determination section or an eye determination step. S109 is also referred to as a face recognition section or a face recognition step.

(Second embodiment)
The second embodiment of the present disclosure shown in FIG. 8 is a modification of the first embodiment. The state monitoring apparatus 200 according to the second embodiment includes a first imaging unit 110 and a second imaging unit 210 instead of the imaging unit 10 (see FIG. 2) of the first embodiment. Hereinafter, based on FIG. 8, the structure of the state monitoring apparatus 200 for acquiring the 1st face image 51 and the 2nd face image 52 is demonstrated in detail.

The first imaging unit 110 and the second imaging unit 210 are both near-infrared cameras and have a configuration corresponding to the imaging unit 10 of the first embodiment. The first image pickup unit 110 includes a first image pickup element 111 corresponding to the image pickup element 11 (see FIG. 5) of the first embodiment, and the image pickup surface of the element 111 is set to a specified area PA (see FIG. 1). It is arranged in a facing posture. The first image sensor 111 receives incident light incident from the defined area PA. With the above configuration, the first imaging unit 110 generates the first face image 51 based on the output from the first imaging element 111 and sequentially outputs it to the image recognition unit 24.

The second imaging unit 210 includes a second imaging element 211 corresponding to the imaging element 11 (see FIG. 5), and a color filter 278 that attenuates red band light. The second imaging unit 210 is arranged in a posture in which the imaging surface of the second imaging element 211 is directed to the defined area PA (see FIG. 1). The second imaging element 211 receives the attenuated light by being covered with the color filter 278. With the above configuration, the second imaging unit 210 generates the second face image 52 based on the output from the second imaging element 211 and sequentially outputs it to the image recognition unit 24.

In the above configuration, the entire area of the first image sensor 111 becomes the uncovered area 76, and the entire area of the second image sensor 211 becomes the covered area 77. In the second embodiment, the pixel pitches and the number of pixels of the

image sensors

111 and 211 are the same. Therefore, the area of the covering region 77 is substantially equal to the area of the non-covering region 76.

The imaging control unit 23 outputs a control signal to each of the first imaging unit 110 and the second imaging unit 210. The imaging control unit 23 sets both the first imaging element 111 and the second imaging element 211 to the exposure state in accordance with the timing when the light emission control unit 21 sets the light emitting diode 16 of the light projecting unit 15 to the light emitting state. Thus, the imaging control unit 23 synchronizes the imaging timings of the first imaging element 111 and the second imaging element 211, so that the first face image 51 and the second face image 52 are captured at substantially the same timing. Is done.

With the above configuration, the first image acquisition block 25 acquires the first face image 51 photographed using the first imaging element 111 from the first imaging unit 110. On the other hand, the second image acquisition block 26 acquires the second face image 52 captured using the second image sensor 211 from the second imaging unit 210. Thus, the imaging part for imaging | photography the 1st face image 51 and the 2nd face image 52 may be provided separately.

In the second embodiment described so far, the second image 52 that is captured using attenuated light is generated by the second imaging unit 210. Therefore, since the eye position can be specified by comparing the first face image 51 and the second face image 52, the accuracy of the face recognition performed based on the eye position is ensured.

In addition, in the second embodiment, the second imaging unit 210 is provided as a configuration different from the first imaging unit 110. Therefore, the freedom degree of the image pick-up element employ | adopted as each image pick-up element 111,211 can be ensured. With the above configuration, it becomes easier to acquire the first face image 51 and the second face image 52 at a resolution suitable for specifying the eye position.

In the second embodiment, the first imaging unit 110 is also referred to as a first imaging device or a first imaging means. The second imaging unit 210 is also referred to as a second imaging device or a second imaging means. The color filter 278 is also referred to as an attenuation filter.

(Third embodiment)
The third embodiment of the present disclosure shown in FIG. 9 is another modification of the first embodiment. The state monitoring apparatus 300 according to the third embodiment includes an imaging unit 310 instead of the imaging unit 10 (see FIG. 2) of the first embodiment. Hereinafter, based on FIG. 9, the structure of the state monitoring apparatus 300 for acquiring the 1st face image 51 and the 2nd face image 52 is demonstrated in detail.

The imaging unit 310 includes a color filter 378 and a switching mechanism 313 in addition to the imaging element 11. The color filter 378 is configured to attenuate red band light and pass near infrared band light. The color filter 378 is formed in a size that can cover the image sensor 11. The switching mechanism 313 is a mechanism for moving the color filter 378.

The imaging unit 310 described above is provided with an attenuation imaging mode and a non-attenuation imaging mode that can be switched to each other as the imaging mode. In the attenuation shooting mode, the imaging unit 310 moves the color filter 378 to a position covering the imaging surface of the imaging element 11 by the operation of the switching mechanism 313. Thus, by covering the image sensor 11 with the color filter 378, the image sensor 11 receives attenuated light. Therefore, the imaging unit 310 can generate the second face image 52 based on the output from the imaging device 11.

On the other hand, in the non-attenuating shooting mode, the imaging unit 310 retracts the color filter 378 from the imaging surface of the imaging device 11 by the operation of the switching mechanism 313. Therefore, the image sensor 11 receives incident light. As described above, the imaging unit 310 can generate the first face image 51 based on the output from the imaging element 11.

The imaging control unit 23 outputs a control signal to the imaging unit 310 so that the attenuation imaging mode and the non-attenuation imaging mode are alternately repeated. Thereby, in the imaging part 310, the 1st face image 51 and the 2nd face image 52 are image | photographed alternately by a slight time difference. Therefore, the image recognition unit 24 acquires the first face image 51 captured in the non-attenuation shooting mode by the first image acquisition block 25 and the second face image 52 captured in the attenuation shooting mode. Acquisition by the image acquisition block 26 is performed alternately. Note that the light emission control unit 21 emits illumination light from the light projecting unit 15 in accordance with the timing at which the image sensor 11 is exposed in each shooting mode.

Also in the third embodiment described so far, the second face image 52 photographed using the attenuated light is generated in the attenuation photographing mode of the imaging unit 310. Accordingly, the eye position can be specified by comparing the first face image 51 and the second face image 52, and thus the accuracy of the face recognition performed based on the eye position is ensured.

In the third embodiment, the imaging unit 310 is also referred to as an imaging device or imaging means. The color filter 378 is also referred to as an attenuation filter.

(Other embodiments)
Although a plurality of embodiments according to the present disclosure have been described above, the present disclosure is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure. can do.

In yet another modification 1 of the first embodiment, an image sensor 411 shown in FIG. 10 is used instead of the image sensor 11 (see FIG. 5). A second pixel 473 included in the image sensor 411 is provided with a sub-pixel 474 covered with a color filter 78. The imaging unit 410 having the above configuration generates the first face image 51 (see FIG. 4) based on the output from the first pixel 70 and the output from the region excluding the sub pixel 474 in the second pixel 473. . In addition, the imaging unit 410 generates the second face image 52 (see FIG. 4) based on the output from the sub-pixel 474. As described above, in the embodiment in which the covering region 77 is provided as the sub-pixel 474, the second face image 52 is photographed together with the first face image 51 while maintaining the sensitivity of the image sensor 411 with respect to near infrared rays, as in the first embodiment. can do.

In the modified example of the second embodiment, the first imaging element and the second imaging element are provided in one imaging unit. The imaging unit is further provided with a splitter that divides incident light from the defined area PA into each imaging element and a color filter positioned between the splitter and the second imaging element. As described above, the first image sensor that forms the uncovered area and the second image sensor that forms the covered area may be provided together in one image pickup unit.

In the modification of the above embodiment, the dark place determination block is omitted. As a result, regardless of whether or not the interior of the vehicle is in a dark place, the eye determination block allows the bright portions that are candidates for a plurality of eyes to be captured in the first face image 51. From the inside, the determination which identifies the bright part corresponding to eyes is implemented.

In the modification of the first embodiment, the number of first pixels and the number of second pixels are set to be approximately the same. As a result, the area of the covering region is approximately the same as the area of the non-covering region. In still another modification, the number of pixels of the second pixel is larger than the number of pixels of the first pixel. Thereby, the area of the covering region is narrower than the area of the non-covering region.

In the modification of the third embodiment described above, the installation and evacuation of the color filter by the switching mechanism may be performed electrically. For example, the color filter may be configured to block light in the red band by applying a voltage. In such a form, the configuration for applying a voltage to the color filter corresponds to the switching mechanism.

In the modification of the above embodiment, the ratio and layout of the pixels provided with the color filter can be changed as appropriate. For example, the pixels having the color filter may be provided so as to be narrowed down to a range that captures a region near the eye in the image sensor.

In the modified example of the first embodiment, the timing at which the output from the non-covering area is acquired and the timing at which the output from the covering area is acquired in the imaging unit are different. In the modification of the second embodiment, the timing at which the output from the first imaging element is acquired in the first imaging unit is different from the timing at which the output from the second imaging element is acquired in the second imaging unit. . As described above, a shift in shooting timing between the first face image and the second face image is acceptable.

Image sensors such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor) can be appropriately employed for each image sensor in the above embodiment. Further, the frequency band of light detected by the image sensor is not limited to the near infrared band, and may include the visible light band in addition to the near infrared band. Furthermore, it is desirable that the light emitting diodes be appropriately changed in frequency band, number, arrangement, and the like of emitted light so as to correspond to the specifications of the image sensor.

In the above-described embodiment, the installation position of the imaging unit and the state monitoring device, which are the upper surface of the steering column 81, may be changed as appropriate as long as the specified area PA can be imaged. The state monitoring device may be installed on the upper surface of the instrument panel, for example, or may be attached to a ceiling portion near the sun visor. Furthermore, the imaging unit may be provided separately from the main body of the state monitoring device and at a position suitable for photographing the defined area PA.

The method for determining the prescribed area PA in the above embodiment may be changed as appropriate. For example, the defined area PA may be defined to include the 95th percentile of Ilips. Furthermore, the method for determining the prescribed area PA is not limited to the method for determining from the iris. For example, the prescribed area PA may be determined experimentally by actually sitting a plurality of drivers of different races, genders, ages, etc. on the driver's seat. Such a defined area PA is desirably defined in consideration of the movement of the face accompanying the driving operation.

In the above embodiment, the plurality of functions provided by the control circuit 20 that has executed the state monitoring program may be provided by hardware and software different from the above-described control device, or a combination thereof. For example, functions corresponding to each functional block and sub-functional block may be provided by an analog circuit that performs a predetermined function without depending on a program.

In the above embodiment, an example in which the present disclosure is applied to a state monitoring device that is mounted on a vehicle and monitors the state of the driver of the vehicle has been described. However, the present disclosure monitors not only the so-called driver status monitor for automobiles as vehicles, but also the state of the driver by various moving bodies (transport equipment) such as motorcycles, tricycles, ships, and aircraft as vehicles. The present invention can be applied to a state monitoring device.

Although the present disclosure has been described based on the embodiments, it is understood that the present disclosure is not limited to the embodiments and structures. The present disclosure includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more or less, are within the scope and spirit of the present disclosure.

Claims

A state monitoring device that is mounted on a vehicle (1) and monitors the state of the driver using face images (51, 52) obtained by photographing the face of the driver who controls the vehicle,
A light emitting section (15, S101) for emitting light in a band from red to near infrared toward a predetermined area (PA) that is defined in advance as an area where the face is located;
A first image acquisition unit (25, S102) for acquiring, as the face image, a first face image (51) captured by receiving incident light incident from the prescribed region;
A second image obtained by receiving a second face image (52) taken by receiving attenuated light obtained by attenuating red band light from the incident light as the face image different from the first face image. An acquisition unit (26, S102);
When a plurality of bright portions (55, 56) that are candidates for the pilot's eyes are photographed in the first face image, a bright portion that is not photographed in the second face image among the plurality of bright portions. An eye determination unit (27, S108) for determining (55) as the eye;
A state monitoring apparatus comprising: a face recognition unit (24, S109) that recognizes the face based on the eye position determined by the eye determination unit.
An attenuation filter (78) that attenuates light in the red band, a covering region (77) that receives the attenuated light by being covered by the attenuation filter, and a non-light that receives the incident light at a position outside the covering region. An image sensor (11) that forms a covered region (76), and generates the first face image based on an output from the non-covered region and the second face image based on an output from the covered region. An imaging unit (10),
The first image acquisition unit acquires the first face image captured using the uncovered region from the imaging unit,
The state monitoring device according to claim 1, wherein the second image acquisition unit acquires the second face image captured using the covering region from the imaging unit.
The state monitoring device according to claim 2, wherein an area of the covered region is smaller than an area of the non-covered region in the imaging element.
The state monitoring device according to claim 2 or 3, wherein the imaging unit acquires an output from the covered region while acquiring an output from the non-covered region.
A first imaging unit (110) that includes the first imaging element (111) that receives the incident light, and that generates the first face image based on an output from the first imaging element;
The second face image has a second image sensor (211) that receives the attenuated light by being covered by an attenuation filter (278) that attenuates light in the red band, and is based on an output from the second image sensor. A second imaging unit (210) for generating
The first image acquisition unit acquires the first face image captured using the first imaging element from the first imaging unit,
The state monitoring apparatus according to claim 1, wherein the second image acquisition unit acquires the second face image captured using the second imaging element from the second imaging unit.
An image sensor (311) that receives the incident light and an attenuation filter (378) that attenuates red band light. The image sensor that receives the attenuated light by covering the image sensor with the attenuation filter. Based on the output from the image sensor that receives the incident light by retracting the attenuation filter from the imaging surface of the image sensor and generating the second face image based on the output of An imaging unit (310) capable of switching between a non-attenuating shooting mode for generating the first face image;
The first image acquisition unit acquires the first face image shot in the non-attenuating shooting mode from the imaging unit,
The state monitoring device according to claim 1, wherein the second image acquisition unit acquires the second face image captured in the attenuation shooting mode from the imaging unit.
A dark place determination unit (28, S103) for determining whether the interior of the vehicle is a dark place,
The eye determination unit is configured in a case where a plurality of bright portions that are candidates for the eyes of the pilot are photographed in the first face image as a condition that the room is determined to be dark in the dark place determination unit. The state monitoring device according to any one of claims 1 to 6, wherein a bright portion that is not photographed in the second face image among the plurality of bright portions is determined to be the eye.
A state monitoring method that is mounted on a vehicle (1) and causes a computer to execute processing for monitoring the state of the operator using face images (51, 52) obtained by photographing the face of the operator who controls the vehicle. ,
A light emitting step (S101) for emitting light in a band from red to near infrared toward a defined area (PA) defined in advance as an area where the face is located;
A first image acquisition step (S102) for acquiring, as the face image, a first face image (51) captured by receiving incident light incident from the prescribed region;
A second image obtained by receiving a second face image (52) taken by receiving attenuated light obtained by attenuating red band light from the incident light as the face image different from the first face image. An acquisition step (S102);
When a plurality of bright portions (55, 56) that are candidates for the pilot's eyes are photographed in the first face image, a bright portion that is not photographed in the second face image among the plurality of bright portions. (55) an eye determination step (S108) for determining that the eye is the eye;
A state monitoring method comprising: a face recognition step (S109) for recognizing the face based on the eye position determined in the eye determination step.
A non-transitory storage medium, the medium comprising instructions read and executed by a computer;
The instruction includes the state monitoring method according to claim 8, wherein the method is a storage medium mounted on a computer.