JP2014082585A - State monitor device and state monitor program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a state monitor technique capable of securing accuracy in face recognition to be performed based on the eye positions of a driver.SOLUTION: A state monitor device 100 is mounted on a vehicle 1 and monitors the state of a driver by using a face image 51 of the driver. The state monitor device 100 emits illumination light from a light projection unit 15 toward the face of a driver assumed to be positioned in a provision area PA. The state monitor device 100 acquires a first face image 51 to be captured by receiving incident light made incident from the provision area PA and a second face image 52 captured by receiving attenuation light obtained by attenuating red band light from the incident light. In this case, when multiple bright parts 55, 56 to be the candidates of the eyes of a driver are captured in the first face image 51, the bright parts 55 that are not captured in the second face image 52 between the bright parts are determined to be the eyes. Processing for face recognition is performed based on the eye positions identified by the above processing.

Description

  The present invention relates to a technique for monitoring the state of a pilot mounted on a mobile body using a face image obtained by photographing the face of the pilot operating the mobile body.

  2. Description of the Related Art Conventionally, a state monitoring apparatus mounted on a moving body such as a vehicle performs face recognition using a face image obtained by photographing a driver's face in order to monitor the state of the driver. In order to perform such face recognition, the state monitoring device has a configuration in which light is emitted toward a prescribed region that is defined in advance as a region where the operator's face is located, and light that is incident from the prescribed region is received. And a configuration for recognizing the face based on the position of the eye in the captured face image.

  The image processing apparatus disclosed in Patent Document 1, which is a type of configuration for recognizing a face as described above, performs face recognition using red eyes. More specifically, in a face image shot with flash emission, a human eye can appear as a red bright part. Such a red eye phenomenon is a phenomenon that occurs mainly because light in the red band of the flash light is reflected by the blood vessels of the retina. This red eye acquires high contrast with respect to the surroundings of the red eye. Therefore, the image processing apparatus can accurately specify the position of the eye in the face image based on the position of the red bright part.

JP 2003-30647 A

  Now, the inventor of the present invention tried to adopt the above-described face recognition using the red eye in the state monitoring device. Then, it turned out that the following problems occur. The problem mainly occurs when a driver wearing glasses is photographed while illuminating a specified area with a light emitting means such as a flash in a state with little ambient light. In this case, in the captured face image, not only the driver's red eye but also the light emitting means reflected in the glasses is captured as a bright part. As described above, if a plurality of bright portions that are candidate eyes are photographed around the eyes of the driver, the specification of the position of the eyes in the face image may be incorrect. As a result, the accuracy of face recognition performed based on the position of the eyes may not be ensured.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a state monitoring device and a state monitoring program capable of ensuring the accuracy of face recognition performed based on the position of the eyes. That is.

  In order to achieve the above-mentioned object, the invention according to claim 1 is mounted on the moving object (1) and uses the face image (51, 52) obtained by photographing the face of the operator who controls the moving object. A state monitoring device for monitoring a person's state, and emitting means (15, S101) for emitting light in a band from red to near infrared toward a predetermined area (PA) defined in advance as an area where the face is located ), First image acquisition means (25, S102) for acquiring, as a face image, a first face image (51) captured by receiving incident light incident from a prescribed area, and a red band from the incident light Second image acquisition means (25, S102) for acquiring, as a face image different from the first face image, the second face image (52) captured by receiving the attenuated light obtained by attenuating the light of The bright part (55, 56) that is a candidate for the pilot's eyes An eye determination unit (27, S108) that determines that a bright part (55) that is not photographed in the second face image among a plurality of bright parts when the face image is photographed; And a face recognizing unit (24, S109) for recognizing the face based on the eye position determined by the determining unit.

  The invention according to claim 8 is a process for monitoring the state of the pilot using a face image (51, 52) mounted on the moving body (1) and photographing the face of the pilot operating the moving body. Is a state monitoring program that causes a computer to execute a light emission step (S101) for emitting light in a band from red to near-infrared toward a specified area (PA) defined in advance as an area where the face is located, A first image acquisition step (S102) for acquiring a first face image (51) photographed by receiving incident light incident from a specified area as a face image, and attenuating red band light from the incident light A second image acquisition step (S102) for acquiring the second face image (52) photographed by receiving the attenuated light as a face image different from the first face image; Naru bright part (55,5 ) In the first face image, the eye determination step (S108) that determines that the bright part (55) that is not captured in the second face image among the plurality of bright parts is the eye. A face recognition step (S109) for recognizing the face based on the eye position determined in the eye determination step is executed by the computer.

  According to these configurations and processing, in the second face image captured by receiving the attenuated light, the light of the red band is attenuated from the incident light, so that the driver's eye is a bright part. It is difficult to see. Therefore, when a plurality of bright parts that are candidates for the eyes of the pilot are photographed in the first face image, by determining those of these bright parts that are not photographed in the second face image as eyes, Even when the light emitting means is reflected in the driver's glasses, the position of the eye in the face image can be accurately specified. Therefore, the accuracy of face recognition performed based on the eye position can be ensured.

It is a figure for demonstrating arrangement | positioning in the vehicle of the state monitoring apparatus by 1st embodiment of this invention. It is a block diagram for demonstrating the electrical structure of the state monitoring apparatus by 1st embodiment. It is a perspective view for demonstrating the mechanical structure of a state monitoring apparatus. It is a figure which shows an example of a 1st face image typically. It is a schematic diagram for demonstrating the structure of an image pick-up element. It is a figure which shows an example of a 2nd face image typically. It is a flowchart which shows the process implemented by the image recognition part. It is a block diagram for demonstrating the electrical structure of the state monitoring apparatus by 2nd embodiment. It is a block diagram for demonstrating the electrical structure of the state monitoring apparatus by 3rd embodiment. It is a schematic diagram which shows the modification of FIG.

  Hereinafter, a plurality of embodiments of the present invention will be described with reference to the drawings. In addition, the overlapping description may be abbreviate | omitted by attaching | subjecting the same code | symbol to the corresponding component in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other part of the configuration. In addition, not only combinations of configurations explicitly described in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined even if they are not explicitly specified unless there is a problem with the combination. . Combinations that are not explicitly described between the configurations described in the plurality of embodiments are also disclosed by the following description.

  As shown in FIG. 1, the state monitoring device 100 according to the first embodiment of the present invention is mounted on a vehicle 1 as a “moving body” and is in a state of a driver (hereinafter referred to as “driver”) who controls the vehicle. It is a driver status monitor that monitors As shown in FIG. 2, the state monitoring apparatus 100 includes an imaging unit 10, a light emitting unit 15, a control circuit 20, and a housing 60 (see FIG. 3) that houses these configurations. The state monitoring device 100 is connected to an actuation unit 90 and a vehicle control device 96 mounted on the vehicle.

  The imaging unit 10 illustrated in FIGS. 1 and 2 is a device that generates a face image 51 (see also FIG. 4) obtained by photographing the driver's face in the state monitoring device 100 installed on the upper surface of the steering column 81. The state monitoring device 100 installed on the upper surface of the steering column 81 is a device that generates a face image 51 (see also FIG. 4) obtained by photographing the driver's face. The image capturing unit 10 captures a predetermined area PA defined in advance in the vehicle 1. This prescribed area PA includes an area where the face of the driver who is seated in the driver's seat is assumed to be located. Specifically, such a defined area PA is defined based on the eyelips assumed from the eye range of each eye of the driver, and is defined to include, for example, the 99th percentile of the eyelips.

  The imaging unit 10 is a so-called near infrared camera, and is configured by combining the imaging element 11 with an optical lens, an optical filter, and the like. The imaging element 11 generates an electrical signal corresponding to the intensity of received light by a plurality of pixels arranged along the imaging surface. The image sensor 11 is arranged in a posture in which the imaging surface is directed to the defined area PA. The image sensor 11 is in an exposure state based on a control signal from the control circuit 20 and receives incident light incident from the defined area PA. As a result, a monochrome face image 51 drawn with shades of white and black is generated. The face images 51 thus photographed are sequentially output from the imaging unit 10 to the control circuit 20.

  The light projecting unit 15 has a plurality of light emitting diodes 16. Each light emitting diode 16 is disposed so as to sandwich the imaging unit 10 (see FIG. 3), and emits illumination light in a band from red to near infrared toward the defined area PA. The on state and the off state of light emission of the light emitting diode 16 are controlled by a current supplied from the control circuit 20.

  The control circuit 20 is connected to the imaging unit 10, the light projecting unit 15, the actuation unit 90, and the like, and is a circuit that controls the operation of these components. The control circuit 20 is mainly configured by a microcomputer including a processor that performs various arithmetic processes, a RAM that functions as a work area for the arithmetic processes, and a flash memory that stores programs used for the arithmetic processes. . In addition, the control circuit 20 includes a power supply circuit that supplies power to the imaging unit 10, the light projecting unit 15, and the like.

  The control circuit 20 executes a plurality of functions such as a light emission control unit 21, an imaging control unit 23, an image recognition unit 24, a state determination unit 31, and a warning control unit 33 by executing a state monitoring program stored in advance by a processor. With blocks.

  The light emission control unit 21 is a functional block related to the light emission control of the light projecting unit 15. The light emission control unit 21 causes the light emitting diode 16 to emit light by applying a predetermined current to the light emitting diode 16. Based on the control value calculated by the image recognition unit 24, the light emission control unit 21 causes the light projecting unit 15 to emit illumination light in accordance with the timing at which the imaging unit 10 is in the exposure state.

  The imaging control unit 23 is a functional block related to imaging control of the imaging unit 10. The imaging control unit 23 controls the exposure start timing, gain, exposure time, and the like in the imaging unit 10 based on the control value calculated by the image recognition unit 24.

  The image recognition unit 24 is a functional block related to image processing of the face image 51 and the like. The image recognition unit 24 sets an imaging condition in the imaging unit 10 and a light emission condition in the light projecting unit 15 in order to acquire a face image 51 from which the driver's face can be extracted. The image acquisition unit 21 controls the imaging control unit 23 and the light emission control unit 21 to control the imaging unit 10 and the light projecting unit 15 in order to cause the imaging unit 10 and the light projecting unit 15 to perform operations in accordance with the set imaging conditions and light emission conditions. Calculate the value. The image recognition unit 24 acquires the face image 51 thus photographed from the imaging unit 10. The image recognizing unit 24 performs image processing on the acquired face image 51 to obtain values related to the driver's face orientation and the degree of eye opening (hereinafter referred to as “eye open degree”) and the degree of sleepiness of the driver. Calculate related values.

  The state determination unit 31 compares the value calculated by the image acquisition unit 21 with a preset threshold value. By this comparison processing, the state determination unit 31 estimates whether, for example, a sign of driving aside or a sign of dozing operation is detected. And the state determination part 31 which detected the above-mentioned sign determines with the state which should alert a driver | operator having arisen.

  The warning control unit 33 is connected to the actuation unit 90. The warning control unit 33 outputs a control signal to the actuation unit 90 when the state determination unit 31 determines that a situation that should warn the driver is occurring. Thus, the warning control unit 33 issues a warning to the driver by operating the actuation unit 90.

  As shown in FIG. 3, the housing 60 includes a main body member 63, a front cover member 66, a rear cover member (not shown), and the like.

  The main body member 63 holds the sub-board 62 on which the light projecting unit 15 and the imaging unit 10 are mounted. A main board 61 on which the control circuit 20 is formed is attached to the sub board 62 in a posture orthogonal to the sub board 62. The body member 63 is provided with an insertion hole 64 and a light distribution portion 65. The insertion hole 64 is provided in the central portion of the main body member 63 in the horizontal direction, and allows the imaging unit 10 mounted on the sub-board 62 to be inserted. The insertion hole 64 cooperates with a light blocking hole provided in the sub-substrate 66 to exhibit a light blocking function between the light projecting unit 15 and the imaging unit 10, thereby Light leakage to the imaging unit 10 is prevented. The light distribution unit 65 is disposed so as to sandwich the insertion hole 64 in the horizontal direction, and faces the light projecting unit 15 mounted on the sub-board 62. The light distribution unit 65 distributes light to the defined area PA (see FIG. 1) while transmitting the light emitted from the light projecting unit 15.

  A visible light filter 67 is provided on the front cover member 66. The visible light filter 67 mainly transmits light in the red to near-infrared band used for generating the face image 51 (see FIG. 4) and shields light in the visible light band that is unnecessary for generating the face image 51. To do. The visible light filter 67 covers an opening 68 formed at a position facing the light distribution portion 65 in the front cover member 66. The rear cover member is disposed on the opposite side of the front cover member 66 with the main body member 63 interposed therebetween. The rear cover member covers the substrates 61 and 62 to protect them from dust and dirt in the atmosphere.

  The actuation unit 90 shown in FIG. 2 includes, for example, a speaker 91, a seat vibration device 93, an air conditioner 95, and the like mounted on the vehicle 1 (see FIG. 1). The speaker 91 alerts the driver by reproducing audio data based on a control signal from the warning control unit 33. The seat vibration device 93 is installed inside the seat surface of the driver's seat or the like, and alerts the driver by vibrating the driver's seat based on a control signal from the warning control unit 33. The air conditioner 95 alerts the driver by an operation such as introducing outside air into the vehicle 1 based on a control signal from the warning control unit 33.

  Next, the face image 51 photographed by the state monitoring device 100 will be described in more detail.

  As shown in FIG. 4, when the driver's face is photographed while irradiating illumination light to the prescribed area PA in a state where the ambient light is low, both eyes of the driver appear as a bright portion 55. Such a so-called red-eye phenomenon occurs because light in the red band included in ambient light and illumination light is reflected by capillaries of the retina. This red eye acquires high contrast with respect to the surroundings of the red eye. Therefore, the image recognition unit 24 (see FIG. 2) can accurately specify the position of the eye in the face image 51 based on the position of the bright part 55 that appears in the face image 51.

  However, it is assumed that a driver wearing spectacles is photographed while irradiating the prescribed area PA with illumination light in a state with little ambient light. In this case, not only the driver's so-called red eye but also the light-emitting diode 16 (see FIG. 2) reflected in the glasses is photographed as the bright portion 56 in the photographed face image 51. As described above, if a plurality of bright portions 55 and 56 that are candidate eyes are photographed around both eyes of the driver, the specification of the eye position in the face image 51 may be erroneous. As a result, the accuracy of face recognition performed based on the position of the eyes may not be ensured.

  In order to solve the above problems and improve the accuracy of face recognition, the configuration of the state monitoring apparatus 100 in FIG. 2 and the processing performed by the control circuit 20 will be described in detail below.

  As shown in FIG. 5, the imaging element 11 is formed with a covered region 77 and an uncovered region 76. The covering region 77 is a region covered with the color filter 78 on the imaging surface. The color filter 78 allows light in the near infrared band to pass through while attenuating light in the red band. Therefore, the covering region 77 receives the attenuated light that is incident from the defined region PA (see FIG. 1) and passes through the color filter 78 and in which the red band light is attenuated. The uncovered area 76 is located outside the covered area 77 and is not covered with the color filter 78. Therefore, the uncovered area 76 receives incident light incident from the defined area PA.

  The image sensor 11 is a so-called VGA size element in which, for example, 640 × 480 pixels are arranged in each of the horizontal direction H and the vertical direction V. The image sensor 11 has a first pixel 70 and a second pixel 73 as a plurality of pixels. The first pixel 70 is a pixel that is not covered with the color filter 78. Therefore, the entire area of the first pixel 70 becomes the uncovered area 76. The second pixel 73 is a pixel covered with the color filter 78. Therefore, the entire area of the second pixel 73 becomes the covering region 77. The number of second pixels 73 provided on the imaging surface is smaller than the number of first pixels 70. Therefore, on the imaging surface of the imaging device 11, the area of the covered region 77 is narrower than the area of the non-covered region 76.

  With the configuration of the imaging device 11 described above, based on the output from the first pixel 70, the imaging unit 10 shown in FIG. 2 receives a face image (hereinafter referred to as “first” for convenience) taken by receiving incident light. 51 ”(described as“ face image ”). The imaging unit 10 also captures a face image captured by receiving attenuated light based on the output from the second pixel 73 (see FIG. 5) (hereinafter referred to as “second face image” for convenience). 52 is generated. According to the above, the number of pixels of the first face image 51 is larger than the number of pixels of the second face image 52. Furthermore, the imaging unit 10 can acquire the output from the second pixel 73 while acquiring the output from the first pixel 70 (see FIG. 5). Therefore, the first face image 51 and the second face image 52 are images taken substantially simultaneously.

  In order to realize highly accurate face recognition using the first face image 51 and the second face image 52, the image recognition unit 24 includes a first image acquisition block 25, a second image acquisition block 26, It has a dark place determination block 28 and an eye determination block 27.

  The first image acquisition block 25 acquires the first face image 51 from the imaging unit 10. On the other hand, the second image acquisition block 26 acquires the second face image 52 from the imaging unit 10 as a face image different from the first face image 51.

  The dark place determination block 28 determines whether or not the room of the vehicle 1 (see FIG. 1) is a dark place based on the information acquired from the vehicle control device 96. The vehicle control device 96 is a device that controls various devices mounted on the vehicle 1. The vehicle control device 96 can control the operation of the headlamp of the vehicle 1 as one of its functions. The dark place determination block 28 determines that the interior of the vehicle 1 is a dark place based on the state information of the headlamp of the vehicle 1 when the headlamp is in a lighting state. On the other hand, the dark place determination block 28 determines that the interior of the vehicle 1 is not a dark place when the headlamp is turned off. It should be noted that the switching between the lighting state and the unlighting state of the headlamp by the vehicle control device 96 may be performed based on the detection result of the external light sensor mounted on the vehicle 1 (see FIG. 1). It may be performed based on the operation state of the changeover switch provided in FIG.

  The eye determination block 27 includes a plurality of bright portions 55 and 56 that are candidates for the eyes when a plurality of bright portions 55 and 56 that are candidates for the eyes of the driver shown in FIG. From this, it is determined which bright part corresponds to the actual eye. Specifically, as shown in FIG. 6, in the second face image 52 photographed by receiving the attenuated light, the red band light is attenuated from the incident light. It is hard to become eyes. That is, in the second face image 52, the driver's eyes are not easily captured as the bright portion 55 as shown in FIG. Therefore, among the bright portions 55 and 56 that are candidates for the eyes photographed in the first face image 51, those not photographed in the second face image 52 in FIG. 5 are the bright portions 55 by the actual eyes. The probability is high.

  Based on the principle described above, the eye determination block 27 shown in FIG. 2 performs a process of comparing the first face image 51 and the second face image 52, thereby a plurality of bright portions 55 appearing in the first face image 51. , 56 (see FIG. 4), the bright portion 55 that has not been photographed in the second face image 52 is determined as an eye. The determination of specifying the eye position is performed on the condition that the dark place determination block 28 determines that the room is dark.

  Next, a process performed by the image recognition unit 24 in order to realize the determination for specifying the eye position described so far will be described in detail with reference to FIG. The process shown in FIG. 7 is started by the image recognition unit 24 when the ignition of the vehicle 1 (see FIG. 1) is turned on.

  In S101, the light emission control unit 21 outputs a control signal for instructing the light projecting unit 15 to emit light, and the imaging control unit 23 outputs a control signal for supporting shooting to the imaging unit 10, and the process proceeds to S102. Based on the control signal output in S101, the light projecting unit 15 emits illumination light toward the defined area PA. And the imaging part 10 image | photographs the 1st face image 51 and the 2nd face image 52 by receiving the light from regulation area | region PA containing illumination light.

  In S102, the first face image 51 and the second face image 52 based on the control signal output in S101 are acquired by the first image acquisition block 25 and the second image acquisition block 26, and the process proceeds to S103.

  In S103, it is determined based on the headlight status information acquired from the vehicle control device 96 whether the interior of the vehicle 1 is a dark place. If a positive determination is made in S103, the process proceeds to S105. On the other hand, if a negative determination is made in S103, the process proceeds to S104. In S104, face recognition processing without using red eyes is performed, and the process proceeds to S110. On the other hand, in S105, which is carried out on the condition that an affirmative determination is made in S103, the first face image 51 acquired in S102 is subjected to image processing, so that the driver's eye candidate is determined from the image 51. The bright parts 55 and 56 are extracted and the process proceeds to S106.

  In S106, as a result of the image processing performed in S105, it is determined whether or not a plurality of bright portions 55 and 56 that are candidates for the eye are photographed in a range preliminarily assumed as the vicinity region of the eye. If a negative determination is made in S106, the process proceeds to S107. In S107, face recognition is performed using the bright part 55, which is the red eye extracted in S105, and the process proceeds to S110. On the other hand, if a positive determination is made in S106, the process proceeds to S108.

  In S108, by comparing the first face image 51 and the second face image 52, the second face image 52 is not photographed among the plurality of bright portions 55 and 56 photographed in the first face image 51. The light part 55 is determined to be the eyes of the driver, and the process proceeds to S109. In S109, face recognition is performed using the bright portion 55 that is the red eye determined to be an eye in S108, and the process proceeds to S110.

  In S110, the result of face recognition by any of S104, S107, and S109 is output to the state determination unit 31, and the process proceeds to S111. Based on the face recognition result output in S110, the state determination unit 31 determines whether or not there is a state in which the driver should be warned, such as a sign of driving aside or a sign of dozing. .

  In S111, it is determined whether or not the ignition ON state of the vehicle 1 is continued. If a negative determination is made in S111 because the ignition is turned off, the process ends. On the other hand, if a positive determination is made in S111, the process returns to S101.

  According to the first embodiment described so far, in the second face image 52 photographed by receiving the attenuated light, the red band light is attenuated from the incident light. It is hard to be seen as the bright part 55. Therefore, among the bright portions 55 and 56 photographed in the first face image 51, the bright portion 55 that is not photographed in the second face image 52 is determined as the eye. Thereby, even when the light emitting diode 16 is reflected in the driver's glasses, the position of the eye in the first face image 51 can be accurately specified. Therefore, the accuracy of face recognition performed based on the eye position can be ensured.

  In addition, in the image sensor 11 used in the first embodiment, a second pixel 73 that is a part of the image sensor 11 is covered with a color filter 78 as a covering region 77. Therefore, the imaging unit 10 can obtain both an output for generating the second face image 52 and an output for generating the first face image 51 from one image sensor 11. Therefore, a configuration in which a part of the image sensor 11 is covered with the color filter 78 is particularly suitable for the state monitoring apparatus 100 that identifies the eye position by comparing the first face image 51 and the second face image 52.

  The second face image 52 in the first embodiment is a face image that is mainly necessary for specifying the position of the eye. Therefore, the second face image 52 may not be as clear as the first face image 51. Therefore, the number of second pixels 73 in the image sensor 11 is smaller than the number of first pixels 70. Thereby, since the area of the covering region 77 is smaller than the area of the non-covering region 76, the first face image 51 based on the output from the non-covering region 76 maintains a high resolution and effectively uses the illumination light. It can be a clear image. Therefore, the accuracy of face recognition by the image recognition unit 24 using the first face image 51 can be reliably ensured.

  Further, according to the first embodiment, the imaging unit 10 can generate the first face image 51 and the second face image 52 based on outputs acquired at substantially the same timing. Therefore, the difference in photographing timing between the first face image 51 and the second face image 52 can be substantially eliminated. As described above, it is possible to avoid a situation in which the position of the photographed driver is shifted between the first face image 51 and the second face image 52. Therefore, the accuracy of the eye position specified by the comparison between the first face image 51 and the second face image 52, and hence the accuracy of face recognition, can be further ensured.

  In addition, in the first embodiment, the eye position is identified by comparing the first face image 51 and the second face image 52 on the condition that the interior of the vehicle 1 is dark. . As described above, the bright portion 55 due to red eyes and the bright portion 56 due to the reflection of the light emitting diodes 16 on the glasses are both reflected in the first face image 51 mainly under a condition with little ambient light. Therefore, according to the condition that the position of the eye is specified on the condition that the room is in a dark place, the processing load in the state monitoring device 100 is maintained while maintaining high face recognition accuracy. Can be reduced.

  In the first embodiment, the vehicle 1 corresponds to the “moving body” recited in the claims, the imaging unit 10 corresponds to the “imaging means” recited in the claims, and the light projecting unit 15 includes The image recognition unit 24 corresponds to the “face recognition unit” described in the claims, and the first image acquisition block 25 corresponds to the “light emitting unit” described in the claims. The second image acquisition block 26 corresponds to “first image acquisition means”, the second image acquisition block 26 corresponds to “second image acquisition means”, and the eye determination block 27 corresponds to “eye”. The dark place determination block 28 corresponds to the “dark place determination means” described in the claims, the color filter 78 corresponds to the “attenuation filter” described in the claims, and the step. S101 is “light emission step” recited in the claims Step S102 corresponds to the “first image acquisition step” and “second image acquisition step” described in the claims, and step S108 corresponds to the “eye determination step” described in the claims. Step S109 corresponds to a “face recognition step” recited in the claims.

(Second embodiment)
The second embodiment of the present invention shown in FIG. 8 is a modification of the first embodiment. The state monitoring apparatus 200 according to the second embodiment includes a first imaging unit 110 and a second imaging unit 210 instead of the imaging unit 10 (see FIG. 2) of the first embodiment. Hereinafter, based on FIG. 8, the structure of the state monitoring apparatus 200 for acquiring the 1st face image 51 and the 2nd face image 52 is demonstrated in detail.

  The first imaging unit 110 and the second imaging unit 210 are both near-infrared cameras and have a configuration corresponding to the imaging unit 10 of the first embodiment. The first image pickup unit 110 includes a first image pickup element 111 corresponding to the image pickup element 11 (see FIG. 5) of the first embodiment, and the image pickup surface of the element 111 is set to a specified area PA (see FIG. 1). It is arranged in a facing posture. The first image sensor 111 receives incident light incident from the defined area PA. With the above configuration, the first imaging unit 110 generates the first face image 51 based on the output from the first imaging element 111 and sequentially outputs it to the image recognition unit 24.

  The second imaging unit 210 includes a second imaging element 211 corresponding to the imaging element 11 (see FIG. 5), and a color filter 278 that attenuates red band light. The second imaging unit 210 is arranged in a posture in which the imaging surface of the second imaging element 211 is directed to the defined area PA (see FIG. 1). The second imaging element 211 receives the attenuated light by being covered with the color filter 278. With the above configuration, the second imaging unit 210 generates the second face image 52 based on the output from the second imaging element 211 and sequentially outputs it to the image recognition unit 24.

  In the above configuration, the entire area of the first image sensor 111 becomes the uncovered area 76, and the entire area of the second image sensor 211 becomes the covered area 77. In the second embodiment, the pixel pitches and the number of pixels of the image sensors 111 and 211 are the same. Therefore, the area of the covering region 77 is substantially equal to the area of the non-covering region 76.

  The imaging control unit 23 outputs a control signal to each of the first imaging unit 110 and the second imaging unit 210. The imaging control unit 23 sets both the first imaging element 111 and the second imaging element 211 to the exposure state in accordance with the timing when the light emission control unit 21 sets the light emitting diode 16 of the light projecting unit 15 to the light emitting state. Thus, the imaging control unit 23 synchronizes the imaging timings of the first imaging element 111 and the second imaging element 211, so that the first face image 51 and the second face image 52 are captured at substantially the same timing. Is done.

  With the above configuration, the first image acquisition block 25 acquires the first face image 51 captured using the first imaging element 111 from the first imaging unit 110. On the other hand, the second image acquisition block 26 acquires the second face image 52 captured using the second image sensor 211 from the second imaging unit 210. Thus, the imaging part for imaging | photography the 1st face image 51 and the 2nd face image 52 may be provided separately.

  Also in the second embodiment described so far, the second image 52 that is captured using attenuated light is generated by the second imaging unit 210. Therefore, since the eye position can be specified by comparing the first face image 51 and the second face image 52, the accuracy of the face recognition performed based on the eye position is ensured.

  In addition, in the second embodiment, the second imaging unit 210 is provided as a configuration different from the first imaging unit 110. Therefore, the freedom degree of the image pick-up element employ | adopted as each image pick-up element 111,211 can be ensured. With the above configuration, it becomes easier to acquire the first face image 51 and the second face image 52 at a resolution suitable for specifying the eye position.

  In the second embodiment, the first imaging unit 110 corresponds to the “first imaging unit” described in the claims, and the second imaging unit 210 corresponds to the “second imaging unit” described in the claims. The color filter 278 corresponds to the “attenuation filter” recited in the claims.

(Third embodiment)
The third embodiment of the present invention shown in FIG. 9 is another modification of the first embodiment. The state monitoring apparatus 300 according to the third embodiment includes an imaging unit 310 instead of the imaging unit 10 (see FIG. 2) of the first embodiment. Hereinafter, based on FIG. 9, the structure of the state monitoring apparatus 300 for acquiring the 1st face image 51 and the 2nd face image 52 is demonstrated in detail.

  The imaging unit 310 includes a color filter 378 and a switching mechanism 313 in addition to the imaging element 11. The color filter 378 is configured to attenuate red band light and pass near infrared band light. The color filter 378 is formed in a size that can cover the image sensor 11. The switching mechanism 313 is a mechanism for moving the color filter 378.

  The imaging unit 310 described above is provided with an attenuation imaging mode and a non-attenuation imaging mode that are switchable as imaging modes. In the attenuation shooting mode, the imaging unit 310 moves the color filter 378 to a position covering the imaging surface of the imaging element 11 by the operation of the switching mechanism 313. Thus, by covering the image sensor 11 with the color filter 378, the image sensor 11 receives attenuated light. Therefore, the imaging unit 310 can generate the second face image 52 based on the output from the imaging device 11.

  On the other hand, in the non-attenuating shooting mode, the imaging unit 310 retracts the color filter 378 from the imaging surface of the imaging device 11 by the operation of the switching mechanism 313. Therefore, the image sensor 11 receives incident light. As described above, the imaging unit 310 can generate the first face image 51 based on the output from the imaging element 11.

  The imaging control unit 23 outputs a control signal to the imaging unit 310 so that the attenuation imaging mode and the non-attenuation imaging mode are alternately repeated. Thereby, in the imaging part 310, the 1st face image 51 and the 2nd face image 52 are image | photographed alternately by a slight time difference. Therefore, the image recognition unit 24 acquires the first face image 51 captured in the non-attenuation shooting mode by the first image acquisition block 25 and the second face image 52 captured in the attenuation shooting mode. Acquisition by the image acquisition block 26 is performed alternately. Note that the light emission control unit 21 emits illumination light from the light projecting unit 15 in accordance with the timing at which the image sensor 11 is exposed in each shooting mode.

  Also in the third embodiment described so far, the second face image 52 photographed using the attenuation light is generated in the attenuation photographing mode of the imaging unit 310. Accordingly, the eye position can be specified by comparing the first face image 51 and the second face image 52, and thus the accuracy of the face recognition performed based on the eye position is ensured.

  In the third embodiment, the imaging unit 310 corresponds to the “imaging unit” recited in the claims, and the color filter 378 corresponds to the “attenuation filter” recited in the claims.

(Other embodiments)
Although a plurality of embodiments according to the present invention have been described above, the present invention is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present invention. can do.

  In still another modification 1 of the first embodiment, an image sensor 411 shown in FIG. 10 is used instead of the image sensor 11 (see FIG. 5). A second pixel 473 included in the image sensor 411 is provided with a sub-pixel 474 covered with a color filter 78. The imaging unit 410 having the above configuration generates the first face image 51 (see FIG. 4) based on the output from the first pixel 70 and the output from the region excluding the sub pixel 474 in the second pixel 473. . In addition, the imaging unit 410 generates the second face image 52 (see FIG. 4) based on the output from the sub-pixel 474. As described above, in the embodiment in which the covering region 77 is provided as the sub-pixel 474, the second face image 52 is photographed together with the first face image 51 while maintaining the sensitivity of the image sensor 411 with respect to near infrared rays, as in the first embodiment. can do.

  In the modified example of the second embodiment, the first imaging element and the second imaging element are provided in one imaging unit. The imaging unit is further provided with a splitter that divides incident light from the defined area PA into each imaging element and a color filter positioned between the splitter and the second imaging element. As described above, the first image sensor that forms the uncovered area and the second image sensor that forms the covered area may be provided together in one image pickup unit.

  In the modification of the above embodiment, the dark place determination block is omitted. As a result, regardless of whether or not the interior of the vehicle is in a dark place, the eye determination block allows the bright portions that are candidates for a plurality of eyes to be captured in the first face image 51. From the inside, the determination which identifies the bright part corresponding to eyes is implemented.

  In the modification of the first embodiment, the number of first pixels and the number of second pixels are set to be approximately the same. As a result, the area of the covering region is approximately the same as the area of the non-covering region. In still another modification, the number of pixels of the second pixel is larger than the number of pixels of the first pixel. Thereby, the area of the covering region is narrower than the area of the non-covering region.

  In the modification of the third embodiment, the color filter may be installed and retracted electrically by the switching mechanism. For example, the color filter may be configured to block light in the red band by applying a voltage. In such a form, the configuration for applying a voltage to the color filter corresponds to the switching mechanism.

  In the modification of the above embodiment, the ratio and layout of the pixels provided with the color filter can be changed as appropriate. For example, the pixels having the color filter may be provided so as to be narrowed down to a range that captures a region near the eye in the image sensor.

  In the modified example of the first embodiment, the timing at which the output from the non-covering area is acquired and the timing at which the output from the covering area is acquired are different in the imaging unit. In the modification of the second embodiment, the timing at which the output from the first imaging element is acquired in the first imaging unit is different from the timing at which the output from the second imaging element is acquired in the second imaging unit. . As described above, a shift in shooting timing between the first face image and the second face image is acceptable.

  An image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) can be used as appropriate for each image sensor in the above embodiment. Further, the frequency band of light detected by the image sensor is not limited to the near infrared band, and may include the visible light band in addition to the near infrared band. Furthermore, it is desirable that the light emitting diodes be appropriately changed in frequency band, number, arrangement, and the like of emitted light so as to correspond to the specifications of the image sensor.

  In the above-described embodiment, the installation position of the imaging unit and the state monitoring device, which are the upper surface of the steering column 81, may be changed as appropriate as long as the defined area PA can be captured. The state monitoring device may be installed on the upper surface of the instrument panel, for example, or may be attached to a ceiling portion near the sun visor. Furthermore, the imaging unit may be provided separately from the main body of the state monitoring device and at a position suitable for photographing the defined area PA.

  The method for determining the prescribed area PA in the above embodiment may be changed as appropriate. For example, the defined area PA may be defined to include the 95th percentile of Ilips. Furthermore, the method for determining the prescribed area PA is not limited to the method for determining from the iris. For example, the prescribed area PA may be determined experimentally by actually sitting a plurality of drivers of different races, genders, ages, etc. on the driver's seat. Such a defined area PA is desirably defined in consideration of the movement of the face accompanying the driving operation.

  In the above-described embodiment, the plurality of functions provided by the control circuit 20 that has executed the state monitoring program may be provided by hardware and software different from the above-described control device, or a combination thereof. For example, functions corresponding to each functional block and sub-functional block may be provided by an analog circuit that performs a predetermined function without depending on a program.

  In the said embodiment, the example which applied this invention to the state monitoring apparatus mounted in a vehicle and monitoring the state of the driver | operator of a vehicle was shown. However, the present invention is applied not only to a so-called driver status monitor for a vehicle but also to a state monitoring device that monitors the state of the operator in various moving bodies (transportation equipment) such as ships and aircraft other than the vehicle. Is possible.

PA defined area, 1 vehicle (moving body), 10, 310, 410 imaging unit (imaging means), 110 first imaging unit, 210 second imaging unit, 11,411 imaging element, 111 first imaging element, 211 second Image sensor, 15 light projecting unit (light emitting unit), 24 image recognition unit (face recognition unit), 25 first image acquisition block (first image acquisition unit), 26 second image acquisition block (second image acquisition unit), 27 Eye determination block (eye determination means), 28 Dark place determination block (dark place determination means), 51 First face image (face image), 52 Second face image (face image), 55, 56 Bright part, 76 Non Covering area, 77 Covering area, 78, 278, 378 Color filter (attenuation filter), 100, 200, 300 Condition monitoring device

Claims (8)

A state monitoring device that is mounted on a moving body (1) and that monitors the state of the driver using face images (51, 52) obtained by photographing the face of the driver who controls the moving body,
A light emitting means (15, S101) for emitting light in a band from red to near-infrared toward a predetermined area (PA) defined as an area where the face is located;
First image acquisition means (25, S102) for acquiring, as the face image, a first face image (51) captured by receiving incident light incident from the prescribed region;
A second image obtained by receiving a second face image (52) taken by receiving attenuated light obtained by attenuating red band light from the incident light as the face image different from the first face image. Acquisition means (26, S102);
When a plurality of bright portions (55, 56) that are candidates for the pilot's eyes are photographed in the first face image, a bright portion that is not photographed in the second face image among the plurality of bright portions. (55) an eye determination means (27, S108) for determining that the eye is the eye;
A state monitoring apparatus comprising: a face recognition unit (24, S109) that recognizes the face based on the position of the eye determined by the eye determination unit. An attenuation filter (78) that attenuates light in the red band, a covering region (77) that receives the attenuated light by being covered by the attenuation filter, and a non-light that receives the incident light at a position outside the covering region. An image sensor (11) that forms a covered region (76), and generates the first face image based on an output from the non-covered region and the second face image based on an output from the covered region. An imaging means (10),
The first image acquisition unit acquires the first face image captured using the uncovered area from the imaging unit,
The state monitoring apparatus according to claim 1, wherein the second image acquisition unit acquires the second face image captured using the covering region from the imaging unit.   The state monitoring device according to claim 2, wherein an area of the covering region is smaller than an area of the non-covering region in the imaging element.   The state monitoring apparatus according to claim 2, wherein the imaging unit acquires an output from the covered area while acquiring an output from the non-covered area. A first imaging unit (110) having a first imaging element (111) for receiving the incident light, and generating the first face image based on an output from the first imaging element;
The second face image has a second image sensor (211) that receives the attenuated light by being covered by an attenuation filter (278) that attenuates light in the red band, and is based on an output from the second image sensor. Second imaging means (210) for generating
The first image acquisition unit acquires the first face image captured using the first imaging element from the first imaging unit,
The state monitoring apparatus according to claim 1, wherein the second image acquisition unit acquires the second face image captured using the second imaging element from the second imaging unit. An image sensor (311) that receives the incident light and an attenuation filter (378) that attenuates red band light. The image sensor that receives the attenuated light by covering the image sensor with the attenuation filter. Based on the output from the image sensor that receives the incident light by retracting the attenuation filter from the imaging surface of the image sensor and generating the second face image based on the output of Imaging means (310) that can be switched between a non-attenuating shooting mode for generating the first face image, and
The first image acquisition means acquires the first face image taken in the non-attenuating shooting mode from the imaging means,
The state monitoring apparatus according to claim 1, wherein the second image acquisition unit acquires the second face image captured in the attenuation shooting mode from the imaging unit. Dark place determination means (28, S103) for determining whether or not the room of the mobile body is a dark place,
The eye determining means is provided when a plurality of bright portions that are candidates for the eyes of the pilot are photographed in the first face image as a condition that the room is determined to be dark in the dark place determining means. The state monitoring according to any one of claims 1 to 6, wherein a bright part that is not photographed in the second face image among the plurality of bright parts is determined to be the eye. apparatus. A state monitoring program that is mounted on the mobile body (1) and causes the computer to execute processing for monitoring the state of the pilot using the face images (51, 52) obtained by photographing the face of the pilot operating the mobile body. There,
A light emitting step (S101) for emitting light in a band from red to near infrared toward a defined area (PA) defined in advance as an area where the face is located;
A first image acquisition step (S102) for acquiring, as the face image, a first face image (51) captured by receiving incident light incident from the prescribed region;
A second image obtained by receiving a second face image (52) taken by receiving attenuated light obtained by attenuating red band light from the incident light as the face image different from the first face image. An acquisition step (S102);
When a plurality of bright portions (55, 56) that are candidates for the pilot's eyes are photographed in the first face image, a bright portion that is not photographed in the second face image among the plurality of bright portions. (55) an eye determination step (S108) for determining that the eye is the eye;
A state monitoring program that causes a computer to execute a face recognition step (S109) for recognizing the face based on the eye position determined in the eye determination step.

Images