JP2015194884A - driver monitoring system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a driver monitoring system capable of improving the accuracy in face direction detection.SOLUTION: A driver monitoring system 100 includes: a camera 6 that generates plural image signals including optical information representing different optical characteristics; and a signal processing circuit 20 that recognizes feature points of a driver's face based on at least one image signal in plural image signals and determines whether or not the driver is inattentive driving or drowsy driving based on the feature points of the face.

Description

  The present application relates to a driver monitoring system for monitoring a driver.

  In recent years, driving a camera that uses a camera mounted in the vehicle to detect the driver's face by analyzing the captured image and monitoring whether the driver is looking aside or snoozing A person monitoring system has been proposed.

  In the driver monitoring system disclosed in Patent Document 1, the driver is photographed from the left and right using two cameras arranged at left and right positions on the dashboard in the vehicle. The position of the feature point of the face part is estimated from the captured images obtained from the two cameras, and the face orientation is detected based on the estimated position. The feature points of the facial parts are, for example, eyebrows, eyes, and mouth. According to this driver monitoring system, the face orientation can be detected with higher accuracy than a monocular camera.

  In the driver monitoring system disclosed in Patent Document 2, a driver is photographed using a compound eye camera in which the baseline directions of a plurality of lenses coincide with the vertical direction (vertical direction). A three-dimensional position of the feature point of the facial part is detected from the captured image based on the amount of parallax, and the driver's face orientation is estimated. According to this driver monitoring system, the face orientation can be detected with sufficient accuracy based on the three-dimensional information.

JP 2007-257333 A Japanese Patent No. 49987625

  In the conventional technology described above, further improvement in the accuracy of face orientation detection has been demanded. One non-limiting exemplary embodiment of the present application provides a driver monitoring system that can improve the accuracy of face orientation detection.

  In order to solve the above-described problem, an embodiment of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imagings into which light transmitted through the lens optical system is incident. An image sensor having a region, and the at least two images corresponding to the optical regions, which are disposed between the lens optical system and the image sensor and pass through the optical regions of the at least two optical regions. An imaging apparatus having an arrayed optical element that is incident on each imaging region, and has information on light that has passed through each optical region, and outputs a plurality of image signals acquired from each imaging region An imaging device, a face recognition unit that recognizes a feature point of a driver's face based on at least one image signal of the plurality of image signals, and the driver based on the feature point of the face And a signal processing circuit including a look or inattentive-dozing determination unit determines whether or not dozing, including driver monitoring system.

  The general and specific aspects described above may be implemented using methods and computer programs, or may be realized using a combination of methods and computer programs.

  According to the driver monitoring system concerning one mode of the present invention, a face direction can be detected more accurately using a single camera.

1 is a block configuration diagram of a driver monitoring system 100 according to Embodiment 1. FIG. It is a schematic diagram which shows a mode that the camera unit 10 is mounted in a vehicle. FIG. 3 is a front view of the camera unit 10 as viewed from the driver 2. 2 is a schematic diagram illustrating a configuration example of a camera 6. FIG. FIG. 3 is a front view of the optical element L2 according to Embodiment 1 as viewed from the subject side. (A) is a figure for demonstrating the optical characteristic of optical region D1, D2, D3, and D4 by Embodiment 1, (b) is optical region D1, D2, by the modification of Embodiment 1, and FIG. It is a figure for demonstrating the optical characteristic of D3 and D4. 1 is a perspective view of an arrayed optical element K according to Embodiment 1. FIG. (A) is an enlarged view showing an arrayed optical element K and an image sensor N according to Embodiment 1, and (b) is a position of the arrayed optical element K and a pixel on the image sensor N. It is a figure which shows a relationship. It is a schematic diagram of a human face. It is a block block diagram of driver | operator monitoring system 100A by the modification of Embodiment 1. FIG. It is a front view at the time of seeing the camera unit 10 by the modification of Embodiment 1 from a driver | operator. It is a block block diagram of the driver | operator monitoring system 200 by Embodiment 2. FIG. (A) is a figure for demonstrating the optical characteristic of optical region D1, D2, D3, and D4 by Embodiment 2, (b) is optical region D1, D2, by the modification of Embodiment 2, and FIG. It is a figure for demonstrating the optical characteristic of D3 and D4. FIG. 10 is a block configuration diagram of a driver monitoring system 300 according to a third embodiment. It is the front view which looked at the optical element L2 by Embodiment 3 from the to-be-photographed object side. (A) is a figure for demonstrating the optical characteristic of the optical area | regions D1 and D2 by Embodiment 3, (b) is the optical area | region D1, D2, D3, and D4 by the modification of Embodiment 3. FIG. It is a figure for demonstrating an optical characteristic. 10 is a perspective view of an arrayed optical element K according to Embodiment 3. FIG. (A) is a figure which expands and shows the array-like optical element K and the image pick-up element N by Embodiment 3, (b) is the positional relationship of the array-like optical element K and the pixel on the image pick-up element N FIG. FIG. 10 is a block configuration diagram of a driver monitoring system 300A according to a modification of the third embodiment. FIG. 10 is a block configuration diagram of a driver monitoring system 400 according to a fourth embodiment. It is a figure for demonstrating the optical characteristic of optical area | region D1, D2, D3, and D4 by Embodiment 4. FIG. FIG. 10 is a block configuration diagram of a driver monitoring system 400A according to a modification of the fourth embodiment. 3 is a flowchart showing an operation of the driver monitoring system 100 according to the first embodiment. 6 is a graph showing an example of a relationship between a distance to a subject and a sharpness (sharpness of an image) according to the second embodiment.

  The inventor of the present application has examined the configuration of the conventional driver monitoring system in detail and found the following problems.

  When shooting a driver using a camera mounted in the vehicle (hereinafter sometimes referred to as an “imaging device”), if strong sunlight from the side (horizontal day) is inserted into the vehicle, Due to the reflection of the horizontal sun, one half of the captured image may be saturated. In that case, even if the captured image is analyzed, the driver's face cannot be recognized correctly. This makes it difficult to accurately detect the driver's face orientation. Similar problems also occur in environments where the brightness changes rapidly, such as at the exit of a tunnel or under backlight. It is difficult to say that the conventional driver monitoring system has high robustness with respect to the shooting environment.

  In the driver monitoring systems of Patent Documents 1 and 2, since the distance between the driver and the camera is obtained using the principle of stereo vision, two cameras (including a compound eye camera) are required. Accordingly, the configuration of the driver monitoring system is not sufficient from the viewpoint of downsizing the camera unit on which the camera is mounted. Further, when the driver wears spectacles, the auxiliary light is reflected by the spectacles. Therefore, whiteout occurs in the image area where the eye, which is one of the facial parts, is located, and the movement of the eyes cannot be detected.

  A technique for detecting a heart rate by detecting a change in a driver's face color by analyzing a color image obtained from visible light is known. In addition to detecting the heart rate, for example, the driver's face orientation is detected by analyzing an image obtained from near-infrared light. In this case, a color image camera and a near-infrared light camera are required, and the scale of the driver monitoring system increases.

  In view of such problems of the prior art, the present inventors have come up with a driver monitoring system having a novel configuration. The outline of one embodiment of the present invention is as follows.

  A driver monitoring system according to one aspect of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imaging regions into which light transmitted through the lens optical system is incident. An image sensor having the optical system disposed between the lens optical system and the image sensor, and having passed through the optical regions of the at least two optical regions, the light of the at least two image regions corresponding to the optical regions. An imaging apparatus having an arrayed optical element incident on each imaging area, the imaging apparatus having information on light that has passed through each optical area and outputting a plurality of image signals acquired from each imaging area A face recognition unit for recognizing a feature point of the driver's face based on at least one image signal of the plurality of image signals; Or a signal processing circuit including a determining inattentive-dozing determination section whether or not the dozing.

  In one aspect, the apparatus further includes auxiliary illumination that irradiates near-infrared light, and the signal processing circuit synthesizes the plurality of image signals to generate a composite image signal; and the driver's face orientation. And a face orientation determination unit for determining, wherein the at least two optical regions have a first optical region having a first transmittance characteristic, and a second transmittance characteristic different from the first transmittance characteristic. The second optical region having, the third optical region having a third transmittance characteristic different from the first and second transmittance characteristics, and the first, second, and third transmittance characteristics A fourth optical region having a different fourth transmittance characteristic, wherein the at least two imaging regions are a first imaging region corresponding to the first optical region, and a second optical region corresponding to the second optical region. 2 imaging areas, corresponding to the third optical area A third imaging region, and a fourth imaging region corresponding to the fourth optical region, wherein the image composition unit includes a first image signal acquired from the first imaging region, the second imaging region, The second image signal acquired from the imaging region, the third image signal acquired from the third imaging region, and the fourth image signal acquired from the fourth imaging region are combined to generate the combination. An image signal is generated, the face recognition unit recognizes a feature point of the driver's face based on the composite image signal, and the face orientation determination unit determines the driver's face feature point based on the face feature point. The face orientation may be determined, and the armpit / slumber determination unit may determine whether the driver is looking aside according to the determination result of the driver's face orientation.

  In one aspect, the apparatus further includes auxiliary illumination that irradiates near-infrared light, and the signal processing circuit synthesizes the plurality of image signals to generate a composite image signal, and a distance to the driver. A distance calculating unit for calculating; and a face direction determining unit for determining the face direction of the driver, wherein the at least two optical regions include first and third optical regions having a first transmittance characteristic. , Second and fourth optical regions having a second transmittance characteristic different from the first transmittance characteristic, wherein the at least two imaging regions correspond to the first optical region. An imaging region, a second imaging region corresponding to the second optical region, a third imaging region corresponding to the third optical region, and a fourth imaging region corresponding to the fourth optical region. The image synthesizing unit includes the first imaging unit; The first image signal acquired from the area and the second image signal acquired from the second imaging area are combined to generate a first composite image signal, and acquired from the third imaging area The third image signal and the fourth image signal acquired from the fourth imaging region are combined to generate a second combined image signal, and the distance calculation unit is configured to output the first and second combined images. A distance to the driver is calculated based on the signal, and the face recognition unit recognizes a three-dimensional feature point of the driver's face based on the first composite image signal and the distance to the driver. The face orientation determination unit determines the driver's face orientation based on the three-dimensional feature points, and the driver's look-aside / slumber determination unit determines whether the driver is in accordance with the determination result of the driver's face orientation. You may determine whether you are looking aside.

  In one aspect, the apparatus further includes auxiliary illumination that irradiates near-infrared light, and the signal processing circuit synthesizes the plurality of image signals to generate a composite image signal, and a distance to the driver. It further includes a distance calculating unit for calculating, and a face direction determining unit for determining the driver's face direction, wherein the at least two optical regions are a first optical region having a first focusing characteristic, and the first A second optical region having a second focusing characteristic different from the focusing characteristic of the second optical region, a third optical region having a third focusing characteristic different from the first and second focusing characteristics, and the A fourth optical region having a fourth focusing characteristic different from the first, second and third focusing characteristics, wherein the at least two imaging regions correspond to the first optical region; Imaging region, a second imaging region corresponding to the second optical region , A third imaging region corresponding to the third optical region, and a fourth imaging region corresponding to the fourth optical region, wherein the image composition unit is acquired from the first imaging region A first image signal, a second image signal acquired from the second imaging region, a third image signal acquired from the third imaging region, and a first image signal acquired from the fourth imaging region. 4 to generate the composite image signal, and the distance calculation unit calculates a distance to the driver based on the first, second, third, and fourth image signals. The face recognizing unit recognizes a three-dimensional feature point of the driver's face based on the composite image signal and the distance to the driver, and the face orientation determining unit is based on the three-dimensional feature point. The driver's face orientation is determined, and the armpit / slumber determining unit The driver in accordance with a person of the face orientation determination result may determine whether the driver is looking aside.

  In one aspect, the apparatus further includes auxiliary illumination that irradiates near-infrared light, and the signal processing circuit synthesizes the plurality of image signals to generate a composite image signal, and a distance to the driver. A distance calculating unit for calculating and a face direction determining unit for determining the face direction of the driver, wherein the at least two optical regions have a first transmittance characteristic and a first focusing characteristic; An optical region, a second transmittance region different from the first transmittance property, and a second optical region having the first focusing property, the first transmittance property, and the first focusing property. A third optical region having a second focusing characteristic different from the first optical region, and a fourth optical region having the second transmittance characteristic and the second focusing characteristic, wherein the at least two imaging areas are , First corresponding to the first optical region An image area, a second imaging area corresponding to the second optical area, a third imaging area corresponding to the third optical area, and a fourth imaging area corresponding to the fourth optical area The image synthesizing unit synthesizes the first image signal acquired from the first imaging area and the second image signal acquired from the second imaging area to generate a first synthesized image signal. And combining the third image signal acquired from the third imaging region and the fourth image signal acquired from the fourth imaging region to generate a second combined image signal, and And a second composite image to generate a third composite image signal, and the distance calculation unit calculates a distance to the driver based on the first and second composite image signals, The face recognizing unit detects the third composite image signal and the driver. Recognizing a three-dimensional feature point of the driver's face based on the separation, the face direction determining unit determining the driver's face direction based on the three-dimensional feature point, and Depending on the determination result of the driver's face orientation, it may be determined whether or not the driver is looking aside.

  A driver monitoring system according to another aspect of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imagings into which light transmitted through the lens optical system is incident. An image sensor having a region, and disposed between the lens optical system and the image sensor, light that has passed through each optical region of the at least two optical regions is incident on each image region of the at least two image regions. First and second imaging devices each having an arrayed optical element to be output, the first imaging device having information on light that has passed through each optical region, and outputting a plurality of image signals obtained from the respective imaging regions A first imaging device that generates a first composite image signal by combining the second imaging device, auxiliary illumination that irradiates near infrared light, and the plurality of image signals output from the first imaging device. An image synthesizing unit; a second image synthesizing unit that generates a second synthesized image signal by synthesizing the plurality of image signals output from the second imaging device; and the first and second synthesized images. A distance calculation unit that calculates a distance to the driver using a parallax amount obtained from the signal, and a three-dimensional feature point of the driver's face based on the first composite image signal and the distance to the driver. A three-dimensional face recognition unit for recognizing, a face direction determining unit for determining the driver's face direction based on the three-dimensional feature points, and the driver looking aside according to the determination result of the driver's face direction And a signal processing circuit including an aside look determination unit for determining whether or not the first optical region has a first transmittance characteristic, and the first transmittance characteristic has at least two optical regions. A second optical region having a second transmittance characteristic different from A third optical region having a third transmittance characteristic different from the first and second transmittance characteristics, and a fourth transmittance characteristic different from the first, second and third transmittance characteristics And the at least two imaging areas include a first imaging area corresponding to the first optical area, a second imaging area corresponding to the second optical area, and the second imaging area. A third imaging region corresponding to the third optical region and a fourth imaging region corresponding to the fourth optical region, wherein the first image composition unit is acquired from the first imaging region. A first image signal, a second image signal acquired from the second imaging region, a third image signal acquired from the third imaging region, and a first image signal acquired from the fourth imaging region. 4 image signals are combined to generate the first composite image, and the second The image synthesis unit includes a first image signal acquired from the first imaging area, a second image signal acquired from the second imaging area, and a third image acquired from the third imaging area. The second synthesized image is generated by synthesizing the image signal and the fourth image signal acquired from the fourth imaging region.

  In one aspect, an attitude detection unit that detects the attitude of the driver based on a distance to the driver, and whether to deploy an airbag based on a detection result of the attitude detection unit, An airbag deployment control unit that controls the deployment and deployment speed of the bag may be further provided.

  In one aspect, the apparatus further includes auxiliary illumination that irradiates near-infrared light that vibrates in the direction of the first polarization axis, and the signal processing circuit includes an image selection unit that selects one of the plurality of image signals; And an eye opening detector that detects an opening degree of the driver's eyes, wherein the at least two optical regions transmit a light that vibrates in the direction of the first polarization axis. And a second optical region having a second polarization characteristic that transmits light oscillating in a direction of a second polarization axis perpendicular to the direction of the first polarization axis, The two imaging regions include a first imaging region corresponding to the first optical region and a second imaging region corresponding to the second optical region, and the image selection unit includes the first imaging region. The first image signal obtained from the first image signal By comparing the degree of saturation of the pixels in the second image with the degree of saturation of the pixels in the second image composed of the second image signal acquired from the second imaging region. One of the image signal and the second image signal is selected, and the face recognition unit recognizes a feature point of the driver's face based on the selected image signal, and the eye opening degree detection unit Detects the degree of opening of the driver's eyes based on the feature points of the face, and the armpit / dozing determination unit determines whether the driver is dozing according to the detection result of the degree of opening of the eyes It may be determined whether or not.

  In one aspect, the apparatus further includes auxiliary illumination that irradiates light that vibrates in the direction of the first polarization axis, and the signal processing circuit combines the plurality of image signals to generate a combined image signal; and An image selecting unit that selects one of the plurality of image signals; and an eye opening degree detecting unit that detects an opening degree of the driver's eyes, wherein the at least two optical regions include the first optical region. A first optical region having a first polarization characteristic and a first transmittance characteristic that transmits light oscillating in the direction of the polarization axis, and a first optical characteristic different from the first polarization characteristic and the first transmittance characteristic. A second optical region having a transmittance characteristic of 2, a second polarization characteristic that transmits light oscillating in a direction of a second polarization axis perpendicular to the direction of the first polarization axis, and the first transmittance A third optical region having characteristics, and the second polarization And a fourth optical region having the second transmittance characteristic, wherein the at least two imaging regions correspond to a first imaging region corresponding to the first optical region and a second optical region. A second imaging region, a third imaging region corresponding to the third optical region, and a fourth imaging region corresponding to the fourth optical region, wherein the image composition unit includes the first imaging region The first image signal acquired from the imaging region and the second image signal acquired from the second imaging region are combined to generate a first composite image signal, which is acquired from the third imaging region. The third image signal and the fourth image signal acquired from the fourth imaging region are combined to generate a second combined image signal, and the image selection unit is configured to generate the second combined image signal from the first combined image signal. The degree of saturation of the pixels in the constructed first image, and the second Selecting one of the first composite image signal and the second composite image signal by comparing the degree of saturation of the pixels in the second image composed of the composite image signal, and the face The recognition unit recognizes the feature point of the driver's face based on the selected composite image signal, and the eye opening degree detection unit calculates the opening degree of the driver's eye based on the feature point of the face. Then, the armpit / dozing determination unit may determine whether or not the driver is dozing according to the detection result of the opening degree of the eyes.

  A driver monitoring system according to another aspect of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imagings into which light transmitted through the lens optical system is incident. An image sensor having a region, and disposed between the lens optical system and the image sensor, light that has passed through each optical region of the at least two optical regions is incident on each image region of the at least two image regions. First and second imaging devices each having an arrayed optical element to be operated, the first and second imaging devices showing information of light that has passed through each of the optical regions, and outputting a plurality of image signals obtained from the imaging regions A first selection from the second imaging device, auxiliary illumination that irradiates near infrared light that vibrates in the direction of the first polarization axis, and the plurality of image signals output from the first imaging device A first image selection unit that selects an image signal; a second image selection unit that selects a second selection image signal from the plurality of image signals output from the second imaging device; A distance calculator that calculates the distance to the driver using the amount of parallax obtained from the second selected image signal, and recognizes a two-dimensional feature point of the driver's face based on the first selected image signal A face recognition unit; an eye opening detection unit that detects an opening degree of the driver's eyes based on the two-dimensional feature points; and the driver doeszes according to the detection result of the eye opening degree. A doze determination unit that determines whether or not the vehicle is a three-dimensional face recognition unit that recognizes a three-dimensional feature point of the driver's face based on the first selected image signal and the distance to the driver; Face orientation determination for determining the driver's face orientation based on a three-dimensional feature point of the face And a signal processing circuit including a side-by-side determination unit that determines whether or not the driver is looking aside according to a determination result of the driver's face orientation, and the at least two optical regions are A first optical region having a first polarization characteristic that transmits light oscillating in the direction of the first polarization axis and light oscillating in the direction of a second polarization axis perpendicular to the direction of the first polarization axis; A second optical region having a second polarization characteristic to be transmitted, wherein the at least two imaging regions are a first imaging region corresponding to the first optical region and a second optical region corresponding to the second optical region. The first image selection unit includes a saturation level of pixels in the first image configured from the first image signal acquired from the first imaging region, and the second image selection unit. A second image signal acquired from the imaging region Comparing one of the first image signal and the second image signal as the first selected image signal by comparing the degree of saturation of the pixels in the second image, The image selection unit includes a saturation degree of pixels in the first image configured from the first image signal acquired from the first imaging area, and a second image acquired from the second imaging area. Selecting one of the first image signal and the second image signal as the second selected image signal by comparing the degree of saturation of the pixels in the second image composed of the signals To do.

  In one aspect, an attitude detection unit that detects the attitude of the driver based on a distance to the driver, and whether to deploy an airbag based on a detection result of the attitude detection unit, An airbag deployment control unit that controls the deployment and deployment speed of the bag may be further provided.

  In a certain aspect, after turning on the auxiliary illumination, an auxiliary illumination control unit that turns off the auxiliary illumination according to the degree of saturation of the pixel may be further provided.

  A certain aspect WHEREIN: The auxiliary | assistant illumination which irradiates a near-infrared light is further provided, and the said signal processing circuit estimates the heart rate from the face direction determination part which determines the said driver's face direction, and the said driver | operator's face color change And a physical condition determining unit that determines the physical condition of the driver based on the estimated heart rate, wherein the at least two optical regions transmit the near-infrared light. A first optical region having a spectral characteristic; a second optical region having a second spectral characteristic that transmits red light in the visible light band; and a third spectral characteristic that transmits green light in the visible light band. A third optical region and a fourth optical region having a fourth spectral characteristic that transmits blue light in the visible light band, and the at least two imaging regions correspond to the first optical region. 1 imaging area A second imaging area corresponding to the second optical area, a third imaging area corresponding to the third optical area, and a fourth imaging area corresponding to the fourth optical area, and the face The recognition unit includes a first face recognition unit that recognizes feature points of the driver's face based on a first image signal acquired from the first imaging region, and the second, third, and fourth. A second face recognition unit that recognizes feature points of the driver's face based on at least one of the second, third, and fourth image signals respectively acquired from the imaging regions of The determination unit determines the driver's face orientation based on the feature points recognized by the first face recognition unit, and the aside look / slumber determination unit determines whether the driver's face orientation is determined. It is determined whether the driver is looking aside, and the heart rate estimation unit Face by focusing on feature points of the recognized face by the recognition unit detects the change in the complexion of the driver may estimate the heart rate.

  A driver monitoring system according to another aspect of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imagings into which light transmitted through the lens optical system is incident. An image sensor having a region, and disposed between the lens optical system and the image sensor, light that has passed through each optical region of the at least two optical regions is incident on each image region of the at least two image regions. First and second imaging devices each having an arrayed optical element to be operated, the first and second imaging devices showing information of light that has passed through each of the optical regions, and outputting a plurality of image signals obtained from the imaging regions Recognizing a two-dimensional feature point of the driver's face, a second imaging device, auxiliary illumination for irradiating near-infrared light, a distance calculator for calculating the distance to the driver using the amount of parallax A face recognition unit, a second face recognition unit that recognizes a three-dimensional feature point of the driver's face, and a face direction determination unit that determines the driver's face direction based on the face feature point; A side look determination unit that determines whether or not the driver is looking aside according to a determination result of the driver's face orientation; and a heart rate estimation unit that estimates a heart rate from a change in the driver's face color; A signal processing circuit including a physical condition determination unit that determines the physical condition of the driver based on the estimated heart rate, wherein the at least two optical regions transmit the near infrared light. A first optical region having a characteristic, a second optical region having a second spectral characteristic that transmits red light in the visible light band, and a third spectral characteristic that transmits green light in the visible light band. 3 optical region and blue light in visible light band A fourth optical region having a fourth spectral characteristic, and the at least two imaging regions are a first imaging region corresponding to the first optical region and a second corresponding to the second optical region. An imaging region, a third imaging region corresponding to the third optical region, and a fourth imaging region corresponding to the fourth optical region, and the distance calculation unit includes the first and second A distance to the driver is calculated using a parallax amount obtained from a first image signal respectively acquired from the first imaging region of the imaging device, and the first face recognition unit Recognizing a two-dimensional feature point of the driver's face based on at least one of the second, third, and fourth image signals acquired from the second, third, and fourth imaging regions of the imaging device, respectively. And the second face recognition unit includes the first imaging device. Recognizing a three-dimensional feature point of the driver's face based on the first image signal acquired from the first imaging region and the distance to the driver, the face orientation determination unit is configured to detect the three-dimensional feature point. The driver's face orientation is determined based on the driver's face orientation, and the driver determines whether the driver is looking aside according to the driver's face orientation determination result. The unit focuses on the two-dimensional feature points of the face, detects a change in the driver's face color, and estimates a heart rate.

  In one aspect, an attitude detection unit that detects the attitude of the driver based on a distance to the driver, and whether to deploy an airbag based on a detection result of the attitude detection unit, An airbag deployment control unit that controls the deployment and deployment speed of the bag may be further provided.

  A certain aspect WHEREIN: You may further provide the warning device which issues a warning to the said driver according to the determination result of the said looking-aside / slumber determination part.

  A method according to another aspect of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imaging regions on which light transmitted through the lens optical system is incident. Light that is disposed between the imaging element, the lens optical system, and the imaging element and that has passed through each optical area of the at least two optical areas is each of the at least two imaging areas corresponding to the optical areas. A plurality of image signals output from an imaging device having an arrayed optical element to be incident on the imaging region, the plurality of image signals having information of light that has passed through each of the optical regions, and acquired from each of the imaging regions A driver monitoring method for monitoring whether a driver is looking aside and / or dozing by analyzing at least one of image signals, the plurality of images Recognizing a feature point of the driver's face based on at least one image signal of the number, and determining whether the driver is looking aside or falling asleep based on the feature point of the face; Is included.

  A computer program according to another aspect of the present invention includes a lens, a lens optical system including at least two optical regions having different optical characteristics, and at least two imaging regions into which light transmitted through the lens optical system is incident. An image sensor having the optical system disposed between the lens optical system and the image sensor, and having passed through the optical regions of the at least two optical regions, the light of the at least two image regions corresponding to the optical regions. A plurality of image signals output from an imaging device having an arrayed optical element to be incident on each imaging region, the plurality of image signals having information on light that has passed through each optical region, and acquired from each imaging region A driver monitoring system that monitors whether the driver is looking aside and / or dozing by analyzing at least one of the image signals of A step of recognizing a feature point of the driver's face based on at least one image signal of the plurality of image signals to a computer of the driver monitoring system; and And determining whether the driver is looking aside or snoozing based on points.

  According to the driver monitoring system according to the above aspect, by setting various optical characteristics of the optical region, for example, shooting in a dynamic range (hereinafter referred to as “HDR”) using a single imaging device. In addition, it is possible to perform monocular distance measurement, shooting using light polarization characteristics, and shooting using light spectral characteristics. By analyzing the image information obtained by such photographing, it can be accurately determined whether the driver is looking aside or snoozing.

  Hereinafter, an embodiment of a driver monitoring system according to the present invention will be described with reference to the drawings.

(Embodiment 1)
First, with reference to FIGS. 1-3, the structure and function of the driver | operator monitoring system 100 by this Embodiment are demonstrated.

  FIG. 1 shows a block configuration diagram of a driver monitoring system 100 according to the present embodiment. The driver monitoring system 100 includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40. Note that the warning device 40 may not be a component of the driver monitoring system 100, and may be an external device connected to the driver monitoring system 100. In this specification, the warning device 40 will be described as being a component of the driver monitoring system 100.

  The camera unit 10 emits near-infrared light as auxiliary illumination light toward the driver. The camera 6 detects an image of the driver and generates a plurality of image signals having light information indicating different light transmittance characteristics.

  The signal processing circuit 20 combines a plurality of image signals to generate an HDR composite image signal, and determines whether the driver is looking aside based on the HDR composite image signal.

  When the signal processing circuit 20 determines that the driver is looking aside, the warning device 40 warns the driver that the driver is looking aside.

  The controller 30 controls the entire driver monitoring system 100. The controller 30 is configured by a semiconductor element or the like. The controller 30 is typically realized by a microcomputer. The controller 30 may be configured only by hardware, or may be realized by combining hardware and software.

  FIG. 2 is a schematic diagram illustrating how the camera unit 10 is mounted in the automobile 4. For example, the camera unit 10 is disposed on the steering column 5 in the vehicle. The camera unit 10 photographs the driver 2 so as to look up from the front through the steering wheel 3. Note that the camera unit 10 may not be disposed on the steering column 5 and may be disposed at any place where the face of the driver 2 can be photographed. For example, the camera unit 10 may be disposed on an upper part of a windshield or an upper part of a dashboard.

  The signal processing circuit 20 and the controller 30 may be provided in the camera unit 10. For example, the signal processing circuit 20 may be provided in an in-car navigation system (not shown). In this case, wired or wireless communication is performed bidirectionally between the camera unit 10 and the car navigation system. Alternatively, the function of the signal processing circuit 20 is implemented in a smartphone (not shown), and wired or wireless communication can be performed bidirectionally between the camera unit 10 and the smartphone. The smartphone may function as a warning device, and a warning may be displayed on the smartphone screen.

  FIG. 3 is a front view when the camera unit 10 is viewed from the driver 2. The camera unit 10 includes a camera 6 and auxiliary lighting 7. The auxiliary illumination 7 emits near-infrared light as auxiliary light toward the driver. The auxiliary illumination 7 is composed of, for example, an LED (Light Emitting Diode). FIG. 3 shows an example in which four LEDs are arranged two by two symmetrically with respect to the camera 6 from the viewpoint of uniformly irradiating the driver.

  An example of the configuration of the camera 6 will be described with reference to FIGS. The configuration of the camera 6 according to the embodiment of the present invention is not limited to that exemplified below.

  FIG. 4 is a schematic diagram illustrating a configuration example of the camera 6. The camera 6 includes a lens optical system L having V as an optical axis, an array-like optical element K disposed near the focal point of the lens optical system L, and an image sensor N.

  The image sensor N is, for example, a CMOS sensor or a CCD sensor.

  The lens optical system L includes a lens L1, a diaphragm S, and an optical element L2. The direction of the light beam B incident from a subject (not shown) is bent by the lens L1, and the diaphragm S eliminates unnecessary light beams. The lens L1 may be composed of a single lens or a plurality of lenses (lens group). The optical element L2 is disposed in the vicinity of the stop S. The diaphragm S may be disposed not on the subject side of the optical element L2 but on the image pickup element N side of the optical element L2. Further, when the lens optical system L is composed of a plurality of lenses, a diaphragm S may be disposed between the lenses.

  FIG. 5 is a front view of the optical element L2 as viewed from the subject side. The optical regions D1, D2, D3, and D4 are regions obtained by dividing a plane perpendicular to the optical axis V into four by straight lines l1 and l2 that intersect the optical axis V. The straight line l1 and the straight line l2 intersect each other perpendicularly on a plane perpendicular to the optical axis V. In principle, the optical areas D1, D2, D3 and D4 have equal areas. A broken line s indicates the position of the diaphragm S. However, the optical regions D1, D2, D3, and D4 may have different areas.

  6A shows the optical characteristics of the optical regions D1, D2, D3, and D4, and FIG. 6B shows the optical properties of the optical regions D1, D2, D3, and D4 according to the modification of the present embodiment. Show. The lens optical system L of the camera 6 of the present embodiment includes a plurality of optical regions having the light transmittance characteristics shown in FIG. As shown in FIG. 6A, in the optical regions D1, D2, D3, and D4, the light transmittances are 100%, 25%, 6.25%, and 1.5625%, respectively. That is, the transmittance ratio of the optical regions D1, D2, D3, and D4 is 64: 16: 4: 1, and the transmittances of the optical regions D1, D2, D3, and D4 are different from each other. The optical region can be formed by arranging ND filters (Neutral Density Filters) having different light transmittances.

  FIG. 7 is a perspective view of the arrayed optical element K. FIG. On the surface of the arrayed optical element K on the imaging element N side, optical elements M2 are arranged in a grid pattern. Each optical element M2 has a curved cross section (vertical and horizontal cross sections), and each optical element M2 protrudes toward the image sensor N side. Thus, the optical element M2 is a microlens, and the arrayed optical element K is a microlens array.

  FIG. 8A is an enlarged view showing the arrayed optical element K and the image sensor N, and FIG. 8B shows the positional relationship between the arrayed optical element K and the pixels on the image sensor N. FIG. The arrayed optical element K is disposed in the vicinity of the focal point of the lens optical system L and is disposed at a position away from the imaging surface Ni by a predetermined distance. The arrayed optical element K is arranged so that the surface on which the optical element M2 is formed faces the imaging surface Ni side.

  Pixels P are arranged in a matrix on the imaging surface Ni. Pixel P can be distinguished into pixels P1, P2, P3 and P4. Pixels P1, P2, P3, and P4 are pixels on which most of the light that has passed through the optical regions D1, D2, D3, and D4 is incident. In FIG. 8B, the pixels P1A, P1B,... Are classified as the pixel P1 and form the first imaging region C1. Pixels P2A, P2B,... Are classified as pixel P2 and form a second imaging region C2. Pixels P3A, P3B,... Are classified as pixels P3 and form a third imaging region C3. Pixels P4A, P4B,... Are classified as pixel P4, and form a fourth imaging region C4.

  The arrayed optical element K causes the light passing through the optical region D1 to enter the pixels in the first imaging region C1 corresponding to the optical region D1, and the light passing through the optical region D2 corresponds to the optical region D2. The light that has entered the pixels in the second imaging region C2 and passed through the optical region D3 is incident on the pixels in the third imaging region C3 corresponding to the optical region D3, and the light that has passed through the optical region D4 is The light is incident on the pixels in the fourth imaging region C4 corresponding to the optical region D4.

  On the imaging surface Ni, a plurality of unit regions M2I are arranged in the vertical direction (column direction) and the horizontal direction. Each unit region M2I is provided with four pixels composed of pixels P1 to P4 in two rows and two columns. One unit region M2I on the imaging surface Ni corresponds to one optical element M2 in the arrayed optical element K. On the imaging surface Ni, a microlens Ms is provided so as to cover the surfaces of the pixels P1, P2, P3, and P4.

  Light from the same part of the subject is incident on the pixels (for example, pixels P1A, P2A, P3A, and P4A) arranged in the unit region M2I.

  The imaging element N generates a first image signal I1 according to the amount of light incident on the imaging region C1, and generates a second image signal I2 according to the amount of light incident on the second imaging region C2. Then, the third image signal I3 is generated according to the amount of light incident on the imaging region C3, and the fourth image signal I4 is generated according to the amount of light incident on the imaging region C4.

  The camera unit 10 outputs a first image signal I1, a second image signal I2, a third image signal I3, and a fourth image signal I4 corresponding to the imaging regions C1, C2, C3, and C4, respectively.

  In addition, although the example which divides | segments an optical region into 4 was shown, the optical region may be divided into 2 divisions, 3 divisions, or 5 divisions or more. The structure of the optical region can be appropriately determined in consideration of design specifications and the like.

  Again, the details of the signal processing circuit 20 will be described with reference to FIG.

  The signal processing circuit 20 includes an image composition unit 21, a face recognition unit 22, a face orientation determination unit 23, and an aside look determination unit 24.

  The signal processing circuit 20 is configured by a semiconductor element or the like. The signal processing circuit 20 can typically be realized by an image signal processor (ISP). The signal processing circuit 20 may be configured only by hardware, or may be realized by combining hardware and software.

  In the present embodiment, an example is shown in which the signal processing circuit 20 and the controller 30 are implemented in the driver monitoring system 100 as independent circuits, but the present invention is not limited to this. For example, the functions of the signal processing circuit 20 and the controller 30 may be mounted on a one-chip integrated circuit. For example, the integrated circuit can be realized by a general-purpose processor. A computer program that exhibits the function of each component may be mounted in the memory inside the processor, and the function of each component may be realized by the processor executing the computer program sequentially.

  The signal processing circuit 20 is based on the composite image signal I5 of the first image signal I1, the second image signal I2, the third image signal I3, and the fourth image signal I4 output from the image sensor N. The direction of the driver's face is detected, and it is determined whether or not the driver is looking aside. The signal processing circuit 20 outputs a determination result as to whether or not the driver is looking aside to the warning device 40 at the subsequent stage.

  The image composition unit 21 generates first image information I1a from the first image signal I1, generates second image information I2a from the second image signal I2, and generates third image information I3 from the third image signal I3. Image information I3a is generated, and fourth image information I4a is generated from the fourth image signal I4.

  For example, the first, second, third and fourth image information I1a, I2a, I3a and I4a are image information indicating luminance values. When the image signal output from the camera unit 10 is RAW data, the image composition unit 21 converts the RAW data (image signal) into a luminance signal (image information). The camera unit 10 may output a luminance signal or a color difference signal, not limited to RAW data. In this case, the signal processing circuit 20 may process the received luminance signal as it is. The image signal output from the camera unit 10 may include a luminance signal. The signal processing circuit 20 may perform image analysis based on the image signal output from the camera unit 10, or may perform image analysis based on the converted image information. In the present specification, the signal processing circuit 20 according to each embodiment will be described on the assumption that the signal processing circuit 20 performs image analysis based on the converted image information.

  The image composition unit 21 synthesizes the first, second, third, and fourth image information I1a, I2a, I3a, and I4a to generate HDR composite image information I5a. The composition of image information is realized by widely using known techniques relating to HDR. Each of the first, second, third, and fourth image information I1a, I2a, I3a, and I4a has information on light that has passed through optical regions having different light transmittances. The image composition unit 21 synthesizes these pieces of image information to generate HDR composite image information. Thereby, the composite image information I5a in which whiteout and blackout are suppressed is obtained.

  The face recognition unit 22 recognizes feature points of the driver's face based on the composite image information I5a. FIG. 9 is a schematic diagram of a human face. As shown in FIG. 9, examples of the feature points of the driver's face include eyebrows 65 and 66, corners 61 and 62, and mouths 63 and 64. The face recognition unit 22 analyzes the composite image information I5a and extracts (recognizes) facial feature points from the image indicated by the composite image information I5a. For example, since a known pattern recognition method can be used for extracting facial feature points, details of the method for extracting facial feature points are omitted in this specification.

  The face orientation determination unit 23 determines the driver's face orientation using the facial feature amount extracted by the face recognition unit 22. Specifically, the face orientation determination unit 23 creates a pattern matching template based on the facial feature amount. The face orientation determination unit 23 performs face tracking by estimating the driver's face orientation on images sequentially output from the camera 6 and executing template matching using a template. Note that details of estimation of the driver's face orientation and template matching are disclosed in Patent Document 2, for example.

  The face orientation determination unit 23 determines whether or not the driver is looking aside according to the determination result of the driver's face orientation. Specifically, the face orientation determination unit 23 determines the driver's face orientation based on the estimation result of the face orientation and the correlation value obtained by template matching. The face orientation determination unit 23 outputs the determination result of the face orientation to the aside look determination unit 24 in the subsequent stage. Note that the above description is an example, and a known method can be widely used for detecting the face orientation.

  The look-ahead determination unit 24 receives the detection result of the face direction, and determines that the driver is looking aside when the face direction is within a predetermined range continuously for a predetermined time. The “face orientation” mainly means the direction of the driver's eyes. For example, the predetermined range is a range in which the elevation angle and depression angle of the line of sight are each 10 degrees or more.

  In the specification of the present application, the aside look determination unit 24 and the dozing determination unit 28 described later are collectively referred to as “aside look / slumber determination unit”.

  The warning device 40 may be constituted by, for example, a monitor that displays a warning message and / or a speaker that emits a warning sound. The warning device 40 issues a warning to the driver according to the determination result of the look-ahead determination unit 24 to urge attention.

  Next, the operation of the driver monitoring system 100 will be described with reference to FIG.

  FIG. 23 shows a flowchart of the operation of the driver monitoring system 100.

  The camera unit 10 outputs the first image signal I1, the second image signal I2, the third image signal I3, and the fourth image signal I4 corresponding to the imaging regions C1, C2, C3, and C4, respectively (step). S100).

  Next, the image composition unit 21 generates first image information I1a from the first image signal I1, generates second image information I2a from the second image signal I2, and generates third image information I3 from the third image signal I3. The third image information I3a is generated, and the fourth image information I4a is generated from the fourth image signal I4 (step S101).

  Next, the image composition unit 21 synthesizes the first, second, third, and fourth image information I1a, I2a, I3a, and I4a to generate HDR composite image information I5a (step S102).

  Next, the face recognition unit 22 recognizes feature points of the driver's face based on the composite image information I5a (step S103).

  Next, the face orientation determination unit 23 determines the driver's face orientation based on the facial feature points recognized by the face recognition unit 22 (step S104).

  Next, the look-ahead determination unit 24 determines whether or not the driver is looking aside according to the determination result of the driver's face orientation. The look-ahead determination unit 24 receives the detection result of the face orientation, and determines that the driver is looking aside when the face orientation is within a predetermined range continuously for a predetermined time (step S105).

  Next, when the look-ahead determination unit 24 determines that the driver is driving aside, the warning device 40 issues a warning to the driver (step S106). On the other hand, when the aside look determination unit 24 determines that the driver is not looking aside, the warning device 40 does not issue a warning to the driver (step S107).

  According to the present embodiment, HDR imaging can be performed using a single imaging device. Since image analysis using HDR images is performed, high robustness is ensured even in an environment where the brightness changes, and it can be accurately determined whether the driver is looking aside.

  Next, a modification of the present embodiment will be described with reference to FIG.

  FIG. 6B shows light transmittances in the optical regions D1, D2, D3, and D4, respectively. In the modification of the present embodiment, the light transmittance characteristics (filter characteristics) of the optical regions D1 to D4 are different from the filter characteristics shown in FIG. 6A of the above-described embodiment. Further, the signal processing circuit 20 according to the modification of the present embodiment further includes a distance calculation unit (not shown) that calculates the distance to the driver.

  As shown in FIG. 6B, in the optical regions D1, D2, D3, and D4, the light transmittances are 100%, 6.25%, 100%, and 6.25%, respectively. That is, the transmittance ratio of the optical regions D1, D2, D3, and D4 is 16: 1: 16: 1.

  The image composition unit 21 synthesizes the first image information I1a and the second image information I2a to generate the HDR first composite image information I5a1, and the third image information I3a and the fourth image information I4a. Are combined to generate HDR second composite image information I5a2.

  The distance calculation unit calculates the distance to the driver based on the phase difference (parallax) obtained from the first composite image information I5a1 and the second composite image information I5a2. This corresponds to the principle of the so-called AF (Auto Focus) phase system used in digital cameras.

  The distance calculation unit calculates the distance to the driver based on the first composite image information I5a1 and the second composite image information I5a2.

  The face recognition unit 22 recognizes the three-dimensional feature point of the driver's face based on the first composite image information I5a1 and the distance information to the driver.

  The face orientation determination unit 23 determines the driver's face orientation using the three-dimensional feature points of the face.

  The aside look determination unit 24 determines whether or not the driver is looking aside according to the determination result of the driver's face orientation.

  According to the modification of the present embodiment, in addition to the above-described HDR effect, a stereo imaging effect can be obtained using a single imaging device. As described above, since the image analysis based on the three-dimensional feature point of the face including not only the feature point of the face but also the distance information to the driver is performed, the face direction can be detected with higher accuracy. As a result, it can be determined more accurately whether the driver is looking aside.

  Next, another modification of the present embodiment will be described with reference to FIGS.

  FIG. 10 shows a block configuration of a driver monitoring system 100A equipped with two cameras 6A and 6B. FIG. 11 is a front view of the camera unit 10 in the driver monitoring system 100A as viewed from the driver.

  As shown in FIGS. 10 and 11, two cameras 6A and 6B are arranged in the camera unit 10, and the camera unit 10 functions as a compound eye camera. Note that the two cameras do not have to be arranged in a direction (horizontal direction) perpendicular to the vertical direction as shown in the drawing, and may be arranged in the vertical direction.

  The driver monitoring system 100A includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40. Further, the driver monitoring system 100A may optionally include an attitude detection unit 51 and an airbag deployment control unit 52.

  The camera unit 10 includes two cameras 6 </ b> A and 6 </ b> B and an auxiliary illumination 7. The cameras 6A and 6B include optical regions D1, D2, D3, and D4 having the optical characteristics shown in FIG. 6 (a) or (b). Since the structure of the cameras 6A and 6B is the same as the structure of the camera 6 described above, HDR shooting can be performed by the camera unit 10.

  The camera unit 10 includes a first image signal I1A, a second image signal I2A, a third image signal I3A, a fourth image signal I4A generated by the camera 6A, and a first image generated by the camera 6B. A signal I1B, a second image signal I2B, a third image signal I3B, and a fourth image signal I4B are output.

  The signal processing circuit 20 includes a first image synthesis unit 21A, a second image synthesis unit 21B, a face model generation unit 22A, a face orientation determination unit 23, an armpit determination unit 24, and a distance calculation unit 25. Have.

  The first image composition unit 21A generates first image information I1a from the first image signal I1A, generates second image information I2a from the second image signal I2A, and generates third image information I3A from the third image signal I3A. Third image information I3a is generated, and fourth image information I4a is generated from the fourth image signal I4A.

  The first image combining unit 21A combines the first image information I1a, the second image information I2a, the third image information I3a, and the fourth image information I4a to generate the first combined image information I5a. .

  The second image composition unit 21B generates the first image information I1b from the first image signal I1B, generates the second image information I2b from the second image signal I2B, and generates the second image information I3B from the third image signal I3B. Third image information I3b is generated, and fourth image information I4b is generated from the fourth image signal I4B.

  The second image composition unit 21B synthesizes the first image information I1b, the second image information I2b, the third image information I3b, and the fourth image information I4b to generate second composite image information I5b. .

The distance calculation unit 25 calculates the amount of parallax based on the first composite image information I5a and the second composite image information I5b, and calculates the distance D5 to the driver according to the following equation 1.
Distance: D5 = L × f / (z × p) (Formula 1)
Here, L (mm) is the length of the base line, f (mm) is the focal length of the lens optical system L, p (mm / pixel) is the pixel pitch of the image sensor N, and z (Mm) is the amount of parallax.

  The distance calculation unit 25 can obtain the amount of parallax by block matching using, for example, the technique disclosed in Patent Document 2. The calculation of the amount of parallax is not limited to the method disclosed in Patent Document 2, and a wide variety of known techniques can be used.

  The face model generation unit (three-dimensional face recognition unit) 22A recognizes the three-dimensional feature point of the driver's face based on the first composite image information I5a and the distance information D5 to the driver, and generates a three-dimensional face model. Generate. For example, a three-dimensional face model can be created using the technique disclosed in Patent Document 2.

  The face orientation determination unit 23 determines the driver's face orientation based on the three-dimensional feature points. Specifically, the face orientation determination unit 23 creates a pattern matching template based on a three-dimensional face model. The face orientation determination unit 23 performs face tracking by estimating the driver's face orientation on images sequentially output from the camera 6A and executing template matching using a template. The face orientation determination unit 23 determines the driver's face orientation based on the estimation result of the face orientation and the correlation value obtained by template matching. The face orientation determination unit 23 outputs the determination result of the face orientation to the aside look determination unit in the subsequent stage.

  The look-ahead determination unit 24 receives the determination result of the face orientation, and determines that the driver is looking aside when the face orientation is within a predetermined range continuously for a predetermined time. For example, the predetermined range is a range in which the elevation angle and depression angle of the line of sight are each 10 degrees or more.

  The warning device 40 issues a warning to the driver according to the determination result of the aside look determination unit 24.

  According to another modification of the present embodiment, the effect of stereo shooting can be obtained in addition to the above-described HDR effect. In this way, image analysis based on not only facial feature points but also three-dimensional facial features including distance information to the driver can be performed, so that the face orientation can be detected more accurately and the driver can look aside. Can be determined with higher accuracy.

  The driver monitoring system 100A may further include an attitude detection unit 51 and an airbag deployment control unit 52 as an option.

  The posture detection unit 51 detects the posture of the driver based on the first composite image information I5a and the distance information D5 to the driver.

  The airbag deployment control unit 52 determines whether to deploy the airbag based on the detection result of the posture detection unit 51. For example, the airbag deployment control unit 52 determines whether to deploy the airbag according to the position of the driver's head relative to the headrest of the seat. Alternatively, the airbag deployment control unit 52 may determine that the airbag is not deployed when the driver's face is located in the vicinity of the airbag placement position. The airbag deployment control unit 52 controls the deployment and deployment speed of the airbag 41 according to the determination result of whether or not to deploy the airbag 41.

  Conventionally, a weight sensor is mounted on a car seat, and the posture of the driver is detected according to a change in weight detected by the weight sensor.

  According to another modification of the present embodiment, since the posture can be detected by image processing, a physical sensor such as a weight sensor need not be provided in order to detect the posture. In this way, it is possible to reduce the scale and cost of the operation monitoring system.

(Embodiment 2)
First, the configuration and functions of the driver monitoring system 200 according to the present embodiment will be described with reference to FIG.

  FIG. 12 shows a block configuration diagram of the driver monitoring system 200 according to the present embodiment. Note that the same reference numerals are assigned to the same components as those of the driver monitoring system 100 according to the first embodiment.

  The driver monitoring system 200 includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40. The driver monitoring system 200 may include an attitude detection unit 51 and an airbag deployment control unit 52 as options.

  The camera unit 10 irradiates near infrared light as auxiliary illumination light toward the driver. The camera 6C detects the image of the driver and generates a plurality of image signals having light information indicating different focusing characteristics.

  The signal processing circuit 20 combines a plurality of image signals to generate a combined image signal. Further, the signal processing circuit 20 obtains the distance to the driver from the plurality of image signals, and determines whether or not the driver is looking aside based on the composite image signal and the distance to the driver.

  When the signal processing circuit 20 determines that the driver is looking aside, the warning device 40 warns the driver that the driver is looking aside.

  Further, as described in the first embodiment, the airbag 41 in the vehicle is deployed according to the processing results of the attitude detection unit 51 and the airbag deployment control unit 52.

  An example of the configuration of the camera 6C will be described with reference to FIGS. 12 and 13 (a).

  FIG. 13A shows the optical characteristics of the optical regions D1, D2, D3, and D4, and FIG. 13B shows the optical properties of the optical regions D1, D2, D3, and D4 according to the modification of the present embodiment. Show. Note that the configuration of the camera 6C in the present embodiment is the same as the configuration of the camera 6 in the first embodiment except that the optical regions have different focusing characteristics. I won't go into details about that.

  The optical regions D1, D2, D3, and D4 have a first focusing characteristic, a second focusing characteristic, a third focusing characteristic, and a fourth focusing characteristic, respectively. The focusing characteristics of the optical regions D1, D2, D3, and D4 are different from each other. Specifically, as shown in FIG. 13A, the optical region D1 is formed from an optical lens having a focal length f1, and the optical region D2 is formed from an optical lens having a focal length f2. The region D3 is formed from an optical lens having a focal length f3, and the optical region D4 is formed from an optical lens having a focal length f4. The focal lengths f1, f2, f3 and f4 are different from each other. The focusing characteristics of the lens optical system L can be made different by adjusting the radius of curvature, aspheric coefficient, and refractive index of the optical regions D1, D2, D3, and D4.

  In the present embodiment, “the focusing characteristics are different” means that at least one of the characteristics contributing to light collection in the optical system is different when compared with light of a predetermined wavelength. Specifically, when compared with light of a predetermined wavelength, the focal length of the lens optical system L by the light passing through the optical regions D1, D2, D3, and D4, the distance to the in-focus subject, and the sharpness are constant. This means that the distance range that is greater than or equal to the value is different.

  The imaging element N outputs a first image signal I1, a second image signal I2, a third image signal I3, and a fourth image signal I4 corresponding to the imaging regions C1, C2, C3, and C4, respectively. The first image signal I1, the second image signal I2, the third image signal I3, and the fourth image signal I4 have a first focusing characteristic, a second focusing characteristic, a third focusing characteristic, and Each has light information obtained from the fourth focusing characteristic.

  Again, the details of the signal processing circuit 20 will be described with reference to FIG.

  The signal processing circuit 20 includes an image synthesis unit 21, a face model generation unit 22 </ b> A, a face orientation determination unit 23, an aside look determination unit 24, and a distance calculation unit 25.

  The image composition unit 21 generates first image information I1a from the first image signal I1, generates second image information I2a from the second image signal I2, and generates third image information I3 from the third image signal I3. Image information I3a is generated, and fourth image information I4a is generated from the fourth image signal I4.

  The image composition unit 21 synthesizes the first image information I1a, the second image information I2a, the third image information I3a, and the fourth image information I4a to generate composite image information I5a.

  The distance calculation unit 25 uses the first image information I1a, the second image information I2a, the third image information I3a, and the fourth image information I4a to each pixel in the image formed from the composite image signal I5. The distance D5 to the driver corresponding to is calculated. Since the focusing characteristics of the four optical regions D1, D2, D3, and D4 are different from each other, the images indicated by the first image information I1a, the second image information I2a, the third image information I3a, and the fourth image information I4a The sharpness (value calculated using luminance) differs depending on the distance to the subject. A memory (not shown) stores a correlation between the sharpness of the image of light that has passed through each of the optical regions D1, D2, D3, and D4 and the distance to the subject. The distance calculation unit 25 is based on the sharpness of each image formed from the first image information I1a, the second image information I2a, the third image information I3a, and the fourth image information I4a and the correlation. The distance to the subject can be obtained.

  An example of the relationship between the distance to the subject and the sharpness will be described with reference to FIG.

  FIG. 24 is a graph showing an example of the relationship between the distance to the subject and the sharpness. As the point image intensity distribution changes, the sharpness also changes. Since the sharpness of the image increases as the size of the point image decreases, the relationship between the distance to the subject and the sharpness is as shown in FIG. In the graph of FIG. 24, G1 indicates the sharpness of a predetermined region of an image generated by the pixel P1 (an image generated by light that has passed through the optical region D1), and G2, G3, and G4 are the pixel P2, The sharpness of a predetermined area of the images generated at P3 and P4 is shown. The range of Z in FIG. 24 shows a region where G2 changes and G1 hardly changes. In the range of Z, the distance to the subject can be obtained using such a relationship. For example, in the Z range, since there is a correlation between the distance to the subject and the ratio between the sharpness G1 and G2, the correlation between the distance to the subject and the ratio between the sharpness G1 and G2 is stored in the memory in advance. Just keep it.

  In addition to the sharpness, the distance to the subject may be obtained based on the contrast ratio of the luminance values, or the distance to the subject may be obtained using the point image intensity distribution. For example, International Publication No. 2013/080552 discloses details of a method for obtaining a distance to a subject.

  The face model generation unit 22A recognizes a three-dimensional feature point of the driver's face based on the composite image information I5a and distance information to the driver, and creates a three-dimensional face model.

  The face orientation determination unit 23 determines the driver's face orientation based on the three-dimensional feature points. Specifically, the face orientation determination unit 23 creates a pattern matching template based on a three-dimensional face model. The face orientation determination unit 23 performs face tracking by estimating the driver's face orientation on images sequentially output from the camera 6C and executing template matching using a template. The face orientation determination unit 23 determines the driver's face orientation based on the estimation result of the face orientation and the correlation value obtained by template matching.

  The look-ahead determination unit 24 receives the detection result of the face direction, and determines that the driver is looking aside when the face direction is within a predetermined range continuously for a predetermined time.

  The warning device 40 issues a warning to the driver according to the determination result of the aside look determination unit 24.

  According to the present embodiment, the image analysis based on the three-dimensional feature point of the face including the distance information to the driver as well as the feature point of the face is performed by monocular distance measurement. It can detect and it can determine more accurately whether the driver is looking aside.

  The driver monitoring system 200 may further include an attitude detection unit 51 and an airbag deployment control unit 52 as an option.

  The posture detection unit 51 detects the driver's posture based on the composite image information I5a and the distance to the driver.

  The airbag deployment control unit 52 determines whether to deploy the airbag based on the detection result of the posture detection unit 51. The airbag deployment control unit 52 controls the deployment and deployment speed of the airbag 41 according to the determination result of whether or not to deploy the airbag 41.

  Next, a modification of the present embodiment will be described with reference to FIG.

  FIG. 13B shows optical characteristics in the optical regions D1, D2, D3, and D4, respectively. In the modification of the present embodiment, the optical characteristics of the optical regions D1 to D4 are different from the optical characteristics shown in FIG. 13A of the above-described embodiment.

  As shown in FIG. 13B, the optical areas D1 and D2 have a first focusing characteristic, and the optical areas D3 and D4 have a second focusing characteristic. Specifically, the optical regions D1 and D2 are formed from an optical lens having a focal length f1, and the optical regions D3 and D4 are formed from an optical lens having a focal length f2. The focal lengths f1 and f2 are different values. Furthermore, in the optical regions D1 and D3, the light transmittance is 100%, and in the optical regions D2 and D4, the light transmittance is 6.25%. That is, the ratio of the light transmittance of the optical regions D1, D2, D3, and D4 is 16: 1: 16: 1.

  The image composition unit 21 synthesizes the first image information I1a and the second image information I2a to generate HDR first composite image information I5a, and the third image signal I3a and the fourth image signal I4a. Are combined to generate HDR second composite image information I5b. The image composition unit 21 further synthesizes the first composite image information I5a and the second composite image information I5b to generate third composite image information I5c.

  The distance calculation unit 25 uses the first composite image information I5a and the second composite image information I5b to calculate the distance to the driver corresponding to each pixel in the image formed from the third composite image information I5c. Calculate.

  According to the modification of the present embodiment, the effects of monocular ranging and HDR can be obtained. Since the image analysis based on the three-dimensional feature point of the face including the distance information to the driver is performed, the face orientation can be detected with higher accuracy. As a result, it can be determined more accurately whether the driver is looking aside, and high robustness is ensured even in an environment where the brightness changes.

(Embodiment 3)
First, the configuration and functions of the driver monitoring system 300 according to the present embodiment will be described with reference to FIG.

  FIG. 14 shows a block configuration diagram of a driver monitoring system 300 according to the present embodiment. The same components as those of driver monitoring systems 100 and 200 according to Embodiments 1 and 2 are denoted by the same reference numerals.

  The driver monitoring system 300 includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40. In addition, the driver monitoring system 300 may include an auxiliary lighting control unit 53 as an option.

  The camera unit 10 irradiates the driver with near infrared light that vibrates in the direction of the first polarization axis as auxiliary illumination light. The camera 6D detects an image of the driver and generates two image signals having light information indicating different polarization characteristics.

  The signal processing circuit 20 selects one of the two image signals according to a predetermined condition. The signal processing circuit 20 recognizes the feature point of the driver's face based on the selected image signal, detects the opening degree of the driver's eyes, and determines whether or not the driver is asleep.

  When the signal processing circuit 20 determines that the driver is asleep, the warning device 40 warns the driver that the driver is asleep.

  An example of the configuration of the camera 6D will be described with reference to FIGS. The configuration of the camera 6D according to the embodiment of the present invention is not limited to that exemplified below.

  FIG. 15 is a front view of the optical element L2 as viewed from the subject side. The optical regions D1 and D2 are regions obtained by dividing a plane perpendicular to the optical axis V into two. In principle, the optical regions D1 and D2 have an equal area. A broken line s indicates the position of the diaphragm S. However, the optical regions D1 and D2 may have different areas.

  FIG. 16A shows the optical characteristics of the optical areas D1 and D2, and FIG. 16B shows the optical characteristics of the optical areas D1, D2, D3, and D4 according to the modification of the present embodiment. As shown in FIG. 16A, the optical region is divided into two. The optical region D1 has a first polarization characteristic that transmits only light that vibrates in the direction of the first polarization axis (for example, the vertical direction), and the optical region D2 is orthogonal to the direction of the first polarization axis. It has the 2nd polarization characteristic which permeate | transmits only the light which vibrates in the direction (for example, horizontal direction) of a 2nd polarization axis. The optical regions D1 and D2 can be formed by polarization filters whose light polarization directions are orthogonal to each other.

  FIG. 17 is a perspective view of the arrayed optical element K. FIG. On the surface on the imaging element N side of the arrayed optical element K, a plurality of optical elements M1 elongated in the horizontal direction within a plane perpendicular to the optical axis V are arranged in the vertical direction. Each optical element M1 has a cross section (a cross section in the vertical direction and a horizontal direction) having an arc shape protruding toward the image sensor N. Thus, the arrayed optical element K has a lenticular lens configuration.

  FIG. 18A is an enlarged view showing the arrayed optical element K and the image sensor N, and FIG. 18B shows the positional relationship between the arrayed optical element K and the pixels on the image sensor N. FIG. The arrayed optical element K is arranged so that the surface on which the optical element M1 is formed faces the imaging surface Ni side. Pixels P are arranged in a matrix on the imaging surface Ni. Pixel P can be distinguished into pixels P1 and P2.

  The pixels P1 are arranged in a row in the horizontal direction (row direction). Every other pixel P1 is arranged in the vertical direction (column direction). The pixels P2 are arranged in a row in the horizontal direction (row direction). Every other pixel P2 is arranged in the vertical direction (column direction). Further, the rows of the pixels P1 and the rows of the pixels P2 are alternately arranged in the vertical direction (column direction).

  The arrayed optical element K is arranged so that one of the optical elements M1 corresponds to two rows of pixels including one row of pixels P1 and one row of pixels P2 on the imaging surface Ni. On the imaging surface Ni, a microlens Ms is provided so as to cover the surfaces of the pixels P1 and P2.

  In the arrayed optical element K, most of the light beam B1 (light beam B1 indicated by a solid line in FIG. 18) that has passed through the optical region D1 (FIG. 15) on the optical element L2 reaches the pixel P1 on the imaging surface Ni. The most part of the light beam that has passed through the optical region D2 (light beam B2 indicated by a broken line in FIG. 18) is designed to reach the pixel P2 on the imaging surface Ni. Specifically, the above configuration is realized by appropriately setting parameters such as the refractive index of the arrayed optical element K, the distance from the imaging surface Ni, and the radius of curvature of the surface of the optical element M1.

  In FIG. 18B, the pixel group P1 forms a first imaging region C1, and the pixel group P2 forms a second imaging region C2.

  The camera unit 10 outputs a first image signal I1 having light information indicating the first polarization characteristic and a second image signal I2 having light information indicating the second polarization characteristic.

  Again, the details of the signal processing circuit 20 will be described with reference to FIG.

  The signal processing circuit 20 includes an image generation unit 29, an image selection unit 26, a face recognition unit 22, an eye opening detection unit 27, and a dozing determination unit 28.

  The image generation unit 29 generates first image information I1a from the first image signal I1, and generates second image information I2a from the second image signal I2.

  The image selection unit 26 determines the degree of saturation of pixels corresponding to the area near the eyes in the first image generated from the first image information I1a and the second image information I2a. One of the first image information I1a and the second image information I2a is selected by comparing the degree of saturation of the pixel corresponding to the region near the eye. The image selection unit 26 selects image information indicating that the image corresponding to the region near the eyes is an image with a small degree of saturation.

  The face recognition unit 22 recognizes feature points of the driver's face based on the selected image information. Specifically, the face recognizing unit 22 analyzes the selected image information and recognizes the feature point of the eye (瞼).

  The eye opening detection unit 27 detects the opening degree of the driver's eyes based on the recognized feature points.

  The dozing determination unit 28 determines whether or not the driver is dozing according to whether or not the opening degree of the eyes is not more than a predetermined amount continuously for a predetermined time. For example, the dozing determination unit 28 determines that the driver is dozing when the eye opening degree is 50% or less continuously for 2 seconds.

  The warning device 40 issues a warning to the driver according to the determination result of the aside look determination unit 24.

  Conventionally, when the driver wears spectacles, auxiliary illumination light is reflected by the spectacles, and pixel saturation occurs in the area near the eyes in the image. Was difficult.

  Since the auxiliary illumination 7 is near-infrared light that vibrates in the direction (vertical direction) of the first polarization axis, reflected light from a mirror surface such as glasses also vibrates in the direction of the first polarization axis. In the present specification, the polarization of light that vibrates in the vertical direction is referred to as vertical polarization. On the other hand, since reflection on the skin or the like becomes diffuse reflection, the reflected light on the skin becomes light (non-polarized light) that vibrates in all directions. For this reason, in the first image, image saturation may occur in the region near the eyes due to the reflected light from the glasses by the auxiliary illumination 7. Since the auxiliary illumination light is vertically polarized light and does not pass through the optical region D2, in the second image, image saturation does not occur in the region near the eyes. Therefore, the image selection unit 26 selects the second image information I2a in which no image saturation occurs in the region near the eyes.

  According to the present embodiment, even when the driver wears spectacles, the opening degree of the eyes can be accurately detected without being affected by the reflected light from the spectacles, and whether or not the driver is dozing is accurately determined. Can be determined.

  In order to improve robustness against external light, the driver monitoring system 300 may further include an auxiliary lighting control unit 53.

  The auxiliary illumination control unit 53 turns on the auxiliary illumination 7 and then turns off the auxiliary illumination according to the degree of saturation of the pixels. Specifically, when the auxiliary illumination 7 is turned on, the auxiliary illumination control unit 53 performs pixel detection in both the area near the eyes in the first image and the area near the eyes in the second image. When the saturation is confirmed, the auxiliary illumination 7 is turned off. In addition, when the auxiliary illumination 7 is turned off, the auxiliary illumination control unit 53 turns on the auxiliary illumination 7 when pixel saturation is not confirmed in the region near the eyes in the second image.

  In addition to the auxiliary illumination light, pixel saturation may occur in a region near the eyes due to reflected light from spectacles due to external light. In such a case, the pixel saturation occurs in the region near the eye in the first image due to the reflection of the auxiliary illumination light that is vertically polarized, and the vicinity of the eye in the second image due to the horizontally polarized external light. This is a state in which pixel saturation occurs in the region. That is, pixel saturation is confirmed in both the region near the eyes in the first image and the region near the eyes in the second image.

  Therefore, by turning off the auxiliary illumination 7, it is possible to reduce pixel saturation in a region near the eyes in the first image. The image selection unit 26 may select the first image information I1a in which image saturation does not occur in the region near the eyes.

  As a result, even if the driver is wearing spectacles and the outside light is strong, it is possible to accurately detect the opening degree of the eyes without being affected by the reflected light from the spectacles. Whether or not can be determined accurately.

  In addition, since auxiliary lighting is necessary at night, it is preferable to turn on the auxiliary lighting 7 except when pixel saturation occurs. If the influence of the reflected light from the spectacles due to external light is reduced and pixel saturation in the region near the eyes in the second image is eliminated, the auxiliary illumination 7 may be turned on.

  Next, a modification of the present embodiment will be described with reference to FIG.

  The optical region shown in FIG. 16B is divided into four. The optical regions D1 and D2 have a first polarization characteristic that transmits only light that vibrates in the direction of the first polarization axis (for example, the vertical direction). The optical regions D3 and D4 have a second polarization characteristic that transmits only light that vibrates in the direction of the second polarization axis (eg, the horizontal direction) orthogonal to the direction of the first polarization axis. The light transmittance in the optical regions D1 and D3 is 100%, and the light transmittance in the optical regions D2 and D4 is 6.25%. That is, the transmittance ratio of the optical regions D1, D2, D3, and D4 is 16: 1: 16: 1. For example, the optical region can be formed by stacking an ND filter and a polarizing filter. In the modification of the present embodiment, since the optical region is divided into four, a microlens array as shown in FIG.

  The signal processing circuit 20 according to this modification includes an image composition unit 21 instead of the image generation unit 29. Other components are common to the components of the signal processing circuit 20 of the present embodiment.

  The image composition unit 21 generates first image information I1a from the first image signal I1, generates second image information I2a from the second image signal I2, and generates third image information I3 from the third image signal I3. Image information I3a is generated, and first image information I4a is generated from the fourth image signal I4.

  The image composition unit 21 synthesizes the first image information I1a and the second image information I2a to generate the first image composition information I5a, and synthesizes the third image information I3a and the fourth image information I4a. The second image composition information I5b is generated.

  The image selection unit 26 includes a saturation degree of pixels corresponding to a region near the eye in the first image generated from the first image composition information I5a, and a second image formed from the second image composition information I5b. One of the first image composition information I5a and the second image composition information I5b is selected by comparing the degree of saturation of the pixel corresponding to the region near the eye in the image.

  According to the modification of the present embodiment, as in the first embodiment, image analysis using an HDR image can be performed, and thus robustness against external light can be further improved.

  Next, another modification of the present embodiment will be described with reference to FIG.

  FIG. 19 shows a block configuration of a driver monitoring system 300A equipped with two cameras 6E and 6F. Two cameras 6E and 6F are arranged in the camera unit 10, and the camera unit 10 functions as a compound eye camera.

  The driver monitoring system 300 </ b> A includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40. Further, the driver monitoring system 300A may optionally include an attitude detection unit 51 and an airbag deployment control unit 52.

  The camera unit 10 includes two cameras 6E and 6F and an auxiliary illumination 7. The cameras 6E and 6F include an optical region having the optical characteristics shown in FIG. 16 (a) or (b). In the following description, it is assumed that the cameras 6E and 6F include optical regions D1 and D2 having the optical characteristics shown in FIG. The structure of the cameras 6E and 6F is the same as that of the camera 6D of the present embodiment.

  The camera unit 10 outputs the first image signal I1A and the second image signal I2A generated by the camera 6E, and the first image signal I1B and the second image signal I2B generated by the camera 6F.

  The first image generation unit 29A generates first image information I1a from the first image signal I1A, and generates second image information I2a from the second image signal I2A. The second image generation unit 29B generates first image information I1b from the first image signal I1B, and generates second image information I2b from the second image signal I2B.

  The first image selection unit 26A has a pixel saturation level corresponding to a region near the eye in the first image generated from the first image information I1a and a second generated from the second image information I2a. One of the first image information I1a and the second image information I2a is selected by comparing the degree of saturation of the pixel corresponding to the region near the eyes in the image. The first image selection unit 26A selects image information indicating that the image corresponding to the region near the eye has a small saturation degree.

  The second image selection unit 26B has a pixel saturation level corresponding to a region near the eye in the first image generated from the first image information I1b, and a second level generated from the second image information I2b. One of the first image information I1b and the second image information I2b is selected by comparing the degree of saturation of the pixel corresponding to the region in the vicinity of the eyes in the image. The first image selection unit 26A selects image information indicating that the image corresponding to the region near the eye has a small saturation degree.

  The distance calculation unit 25 obtains the amount of parallax based on the image information selected by the first image selection unit 26A and the image information selected by the second image selection unit 26B, and the distance to the driver according to the above-described equation 1. D5 is calculated.

  The face recognition unit 22 recognizes feature points of the driver's face based on the image information selected by the first image selection unit 26A.

  The eye opening detection unit 27 detects the opening degree of the driver's eyes based on the recognized feature points.

  The dozing determination unit 28 determines whether or not the driver is dozing according to whether or not the opening degree of the eyes is not more than a predetermined amount continuously for a predetermined time. For example, the dozing determination unit 28 determines that the driver is dozing when the opening degree of the eyes is 50% or less continuously for a predetermined time.

  The face model generation unit (3D face recognition unit) 22A recognizes the 3D feature points of the driver's face based on the image information selected by the first image selection unit 26A and the distance information D5 to the driver, Create a 3D face model.

  The face orientation determination unit 23 creates a pattern matching template based on the three-dimensional face model. The face orientation determination unit 23 performs face tracking by estimating the driver's face orientation on images sequentially output from the camera 6E and executing template matching using a template. The face orientation determination unit 23 determines the driver's face orientation based on the estimation result of the face orientation and the correlation value obtained by template matching.

  The look-ahead determination unit 24 receives the determination result of the face orientation and determines that the driver is looking aside when the face orientation is within a predetermined range continuously for a predetermined time. For example, the predetermined range is a range in which the elevation angle and depression angle of the line of sight are each 10 degrees or more.

  The warning device 40 issues a warning to the driver according to the determination result of the aside look determination unit 24.

  According to another modification of the present embodiment, robustness against external light can be improved using polarization characteristics. Further, the determination of the aside driving and the determination of the dozing operation can be performed at the same time. Furthermore, because of the effects of stereo shooting, image analysis based on 3D feature points of the face including not only the feature points of the face but also the distance information to the driver is performed, so that the face orientation can be detected more accurately and driving It is possible to more accurately determine whether the person is looking aside or snoozing.

  The driver monitoring system 300A may further include an auxiliary illumination control unit 53, an attitude detection unit 51, and an airbag determination deployment unit 52 as options. In this case, by using two cameras, the determination of the aside driving, the determination of the dozing operation, and the determination of the airbag deployment can be simultaneously performed by image processing, and the cost reduction of the driving monitoring system can be realized.

(Embodiment 4)
First, the configuration and function of the driver monitoring system 400 according to the present embodiment will be described with reference to FIG.

  FIG. 20 is a block diagram of a driver monitoring system 400 according to this embodiment. The same components as those of driver monitoring systems 100 and 200 according to Embodiments 1 and 2 are denoted by the same reference numerals.

  The driver monitoring system 400 includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40.

  The camera unit 10 emits near infrared light as auxiliary illumination light toward the driver. The camera 6G detects an image of the driver, and generates an image signal (hereinafter referred to as “IR image signal”) indicating color information and a color image signal.

  The signal processing circuit 20 detects the driver's face direction based on the IR image signal, and determines whether or not the driver is looking aside. Further, the signal processing circuit 20 estimates the heart rate from the change in the driver's face color based on the color image information, and determines the physical condition of the driver.

  The warning device 40 warns the driver when the signal processing circuit 20 determines that the driver is asleep or determines that the driver is in poor physical condition.

  FIG. 21 shows the optical characteristics of the optical regions D1, D2, D3 and D4. The optical region is divided into four, the optical region D1 has a first spectral characteristic that transmits only near-infrared light, and the optical region D2 transmits only red light in the visible light band. The optical region D3 has a third spectral characteristic that transmits only green light in the visible light band, and the optical region D4 has a third spectral characteristic that transmits only blue light in the visible light band. 4 spectral characteristics.

  The camera unit 10 includes a first image signal I1 having near-infrared light information, a second image signal I2 having red (R) light information, and a green (G) light information. 3 image signal I3 and a fourth image signal I4 having blue (B) light information.

  Again, the details of the signal processing circuit 20 will be described with reference to FIG.

  The signal processing circuit 20 includes an image generation unit 29, first and second face recognition units 22B and 22C, a face orientation determination unit 23, a heart rate estimation unit 31, an armpit determination unit 24, and a physical condition determination unit 32. And have.

  The image generation unit 29 generates IR image information indicating near-infrared light information from the first image signal I1 (IR image signal), and from the second, third, and fourth image signals I2, I3, and I4. Generate color image information. The color image information is not limited to luminance and color difference information, and may include luminance values (pixel values) of RGB components.

  The face recognition unit 22 includes a first face recognition unit 22B and a second face recognition unit 22C.

  The first face recognition unit 22B recognizes feature points of the driver's face based on the IR image information.

  The second face recognition unit 22C recognizes feature points of the driver's face based on the color image information. For example, when the color image information indicates the luminance value of each component of RGB, the second face recognition unit 22C can recognize the feature point of the driver's face based on the luminance value of each component. . Alternatively, the second face recognition unit 22C may recognize a feature point of the driver's face based on the luminance value of the G component.

  The face orientation determination unit 23 determines the driver's face orientation using the facial feature points recognized by the first face recognition unit 22B.

  The look-ahead determination unit 24 receives the determination result of the face orientation and determines that the driver is looking aside when the face orientation is within a predetermined range continuously for a predetermined time. For example, the predetermined range is a range in which the elevation angle and depression angle of the line of sight are each 10 degrees or more.

  The heart rate estimation unit 31 focuses on the facial feature points recognized by the second face recognition unit 22C, detects changes in the driver's face color, and estimates the heart rate. As a method for estimating a heart rate from a change in face color, for example, the following method is known.

  When a minute change in blood pressure due to a heartbeat occurs, the amount of reflected light on the face changes. A slight increase in blood pressure slightly reduces the amount of reflected light on the face, and a slight decrease in blood pressure slightly increases the amount of reflected light on the face. If a lot of blood flows through the face, the skin of the face absorbs light more, so the reflectance of light changes according to the heartbeat. By independently analyzing the RGB components, it is possible to separate only the color fluctuation due to the heartbeat and regard the fluctuation amount as the heartbeat change. For example, it is possible to estimate the pulse rate from the peak value of the G component waveform after generating a luminance waveform for each RGB component of each frame of the captured moving image and removing noise common to the three waveforms.

  The physical condition determination unit 32 determines that the driver is in poor physical condition when the estimated heart rate continuously exceeds a predetermined value or less than a predetermined value for a predetermined time. The predetermined time and the predetermined value can be determined from a medical viewpoint.

  The warning device 40 issues a warning to the driver according to the determination results of the looking-aside determination unit 24 and the physical condition determination unit 32.

  Conventionally, two cameras, an infrared camera for face orientation determination and a color camera for estimating the heart rate, are required, and it is difficult to reduce the size of the camera unit 10.

  According to the present embodiment, since the IR image and the color image can be simultaneously acquired by one camera, the determination of the face direction and the determination of the physical condition can be performed simultaneously. In addition, the size of the camera unit 10 can be reduced.

  Next, a modification of the present embodiment will be described with reference to FIG.

  FIG. 22 shows a block configuration of a driver monitoring system 400A equipped with two cameras 6H and 6I. Two cameras 6H and 6I are arranged in the camera unit 10, and the camera unit 10 functions as a compound eye camera.

  The driver monitoring system 400A includes a camera unit 10, a signal processing circuit 20, a controller 30, and a warning device 40. Further, the driver monitoring system 400A may include an attitude detection unit 51 and an airbag deployment control unit 52 as options.

  The camera unit 10 includes two cameras 6 </ b> H and 6 </ b> I and auxiliary lighting 7. Similarly to the camera 6G, the cameras 6H and 6I include an optical region having the optical characteristics shown in FIG. The structure of the cameras 6H and 6I is the same as that of the camera 6G of the present embodiment.

  The camera unit 10 includes first image signals I1A and I1B having near-infrared light information and second image signals having red (R) light information as image signals acquired from the cameras 6H and 6I. I2A, I2B, third image signals I3A, I3B having green (G) light information, and fourth image signals I4A, I4B having blue (B) light information are output.

  The signal processing circuit 20 includes first and second image generation units 29A and 29B, a distance calculation unit 25, a face recognition unit 22, a face model generation unit 22A, a heart rate estimation unit 31, and a physical condition determination unit 32. And a face orientation determination unit 23 and an aside look determination unit 24.

  The first image generation unit 29A generates IR image information A indicating near-infrared light information from the first image signal I1A acquired from the camera 6H, and generates second, third, and fourth image signals. Color image information A is generated from I2A, I3A, and I4A.

  The second image generation unit 29B generates IR image information B indicating near-infrared light information from the first image signal I1B acquired from the camera 6I, and the second, third, and fourth image signals. Color image information B is generated from I2B, I3, and I4B. Note that the color image information A and B may include luminance values of RGB components.

  The distance calculation unit 25 calculates the amount of parallax based on the IR image information A and the IR image information B, and calculates the distance D5 to the driver according to the above equation 1.

  The face recognition unit 22 recognizes a two-dimensional feature point of the driver's face based on the color image information A. For example, when the color image information indicates the luminance value of each component of RGB, the face recognition unit 22 may recognize the feature point of the driver's face based on the luminance value of each component.

  The face model generation unit 22A recognizes the three-dimensional feature point of the driver's face based on the IR image information A and the distance information D5 to the driver, and creates a three-dimensional face model.

  The face orientation determination unit 23 determines the driver's face orientation using a three-dimensional face model.

  The look-ahead determination unit 24 receives the detection result of the face direction, and determines that the driver is looking aside when the face direction is within a predetermined range continuously for a predetermined time. For example, the predetermined range is a range in which the elevation angle and depression angle of the line of sight are each 10 degrees or more.

  The heart rate estimation unit 31 focuses on the two-dimensional feature points of the face recognized by the face recognition unit 22, detects a change in the driver's face color, and estimates the heart rate.

  The physical condition determination unit 32 determines that the driver is in poor physical condition when the estimated heart rate continuously exceeds a predetermined value or less than a predetermined value for a predetermined time.

  The warning device 40 issues a warning to the driver according to the determination results of the looking-aside determination unit 24 and the physical condition determination unit 32.

  According to the present embodiment, the effect of stereo shooting can be obtained, and the determination of the face orientation and the determination of the physical condition can be simultaneously performed by image processing. In addition, the size of the camera unit 10 can be reduced.

  The driver monitoring system 400A may optionally include an attitude detection unit 51 and an airbag determination deployment unit 52. In this case, by using two cameras, the determination of the aside driving, the determination of the physical condition, and the determination of the airbag deployment can be simultaneously performed by image processing, and the driving monitoring system can be reduced in cost.

  As described in the first to fourth embodiments, each image of light that has passed through at least two optical regions having different optical characteristics is simultaneously acquired by one imaging apparatus. The signal processing circuit 20 is based on the synthesized or selected image signal by synthesizing a plurality of image signals of the simultaneously acquired light images or by selecting any one of the plurality of image signals. Recognize feature points of driver's face.

  Similarly to the driver monitoring system according to the first embodiment, the operation flow of the driver monitoring system according to the second to fourth embodiments is defined by the processing unit divided for each function of each processing unit in the signal processing circuit 20. Can do.

  The technique of the present invention can also be applied to software (program) that defines the above-described process for determining whether or not the driver is looking aside. The operation defined in such a program is, for example, as shown in FIG. Such a program can be provided by being recorded on a portable recording medium or can be provided through a telecommunication line. Various operations described in the above embodiment can be realized by a processor built in the computer executing such a program.

  The driver monitoring system disclosed in the present application can be used for a system mounted on a vehicle such as an automobile, a train, and an airplane.

2: Driver 3: Steering wheel 4: Automobile 5: Steering column 6, 6A, 6B, 6C, 6D, 6E, 6F, 6G, 6H, 6I: Camera 7: Auxiliary lighting 10: Camera unit 20: Signal processing circuit 21 : Image composition unit 21A: First image composition unit 21B: Second image composition unit 22: Face recognition unit 22A: Face model generation unit 22B: First face recognition unit 22C: Second face recognition unit 23: Face Direction determination unit 24: Aside look determination unit 25: Distance calculation unit 26: Image selection unit 26A: First image selection unit 26B: Second image selection unit 27: Eye opening detection unit 28: Determination unit 29: Image generation unit 29A: 1st image generation part 29B: 2nd image generation part 30: Controller 31: Heart rate estimation part 32: Physical condition determination part 40: Warning device 41: Air bag 51: Posture Knowledge unit 52: Airbag determination deployment unit 52: Airbag deployment control unit 53: Auxiliary illumination control unit 61, 62: Eye corner 63, 64: Mouth 65, 66: Eyebrow 100, 100A, 200, 300, 300A, 400, 400A : Driver monitoring system K: Array optical element L: Lens optical system L1: Lens L2: Optical element M1 and M2: Optical element M2I: Unit area Ms: Micro lens N: Imaging element Ni: Imaging surface P: Pixel S: Aperture V: Optical axis

Claims (16)

A lens, a lens optical system including at least two optical regions having different optical characteristics, an image pickup device having at least two image pickup regions on which light transmitted through the lens optical system is incident, the lens optical system, and the image pickup device And an array-like optical element that causes light that has passed through each optical region of the at least two optical regions to enter each imaging region of the at least two imaging regions corresponding to each optical region. An imaging device that has information on light that has passed through each of the optical regions, and that outputs a plurality of image signals acquired from the imaging regions;
A face recognition unit for recognizing a feature point of a driver's face based on at least one image signal of the plurality of image signals;
A driver monitoring system comprising: a signal processing circuit including: an armpit / dozing determination unit that determines whether the driver is looking aside or falling asleep based on the feature points of the face. Auxiliary illumination that emits near-infrared light is further provided,
The signal processing circuit includes:
An image synthesis unit that synthesizes the plurality of image signals to generate a synthesized image signal;
A face orientation determining unit that determines the driver's face orientation;
The at least two optical regions include a first optical region having a first transmittance characteristic, a second optical region having a second transmittance characteristic different from the first transmittance characteristic, the first and A third optical region having a third transmittance characteristic different from the second transmittance characteristic and a fourth optical region having a fourth transmittance characteristic different from the first, second and third transmittance characteristics. Including the optical region of
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The image synthesis unit includes a first image signal acquired from the first imaging area, a second image signal acquired from the second imaging area, and a third image acquired from the third imaging area. And the fourth image signal acquired from the fourth imaging region to generate the composite image signal,
The face recognition unit recognizes feature points of the driver's face based on the composite image signal,
The face orientation determination unit determines the driver's face orientation based on the facial feature points,
2. The driver monitoring system according to claim 1, wherein the side-by-side / slumber determining unit determines whether or not the driver is looking aside according to a determination result of the driver's face orientation. Auxiliary illumination that emits near-infrared light is further provided,
The signal processing circuit includes:
An image synthesis unit that synthesizes the plurality of image signals to generate a synthesized image signal;
A distance calculator for calculating the distance to the driver;
A face orientation determining unit that determines the driver's face orientation;
The at least two optical regions include first and third optical regions having a first transmittance characteristic, and second and fourth optical regions having a second transmittance characteristic different from the first transmittance characteristic. An optical region,
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The image synthesizing unit synthesizes the first image signal acquired from the first imaging area and the second image signal acquired from the second imaging area to generate a first synthesized image signal. A second image signal is generated by synthesizing the third image signal acquired from the third imaging region and the fourth image signal acquired from the fourth imaging region;
The distance calculation unit calculates a distance to the driver based on the first and second composite image signals,
The face recognition unit recognizes a three-dimensional feature point of the driver's face based on the first composite image signal and the distance to the driver;
The face orientation determination unit determines the driver's face orientation based on the three-dimensional feature points,
2. The driver monitoring system according to claim 1, wherein the side-by-side / slumber determining unit determines whether or not the driver is looking aside according to a determination result of the driver's face orientation. Auxiliary illumination that emits near-infrared light is further provided,
The signal processing circuit includes:
An image synthesis unit that synthesizes the plurality of image signals to generate a synthesized image signal;
A distance calculator for calculating the distance to the driver;
A face orientation determining unit that determines the driver's face orientation;
The at least two optical regions include a first optical region having a first focusing property, a second optical region having a second focusing property different from the first focusing property, the first and A third optical region having a third focusing characteristic different from the second focusing characteristic, and a fourth optical characteristic having a fourth focusing characteristic different from the first, second and third focusing characteristics. Including the optical region of
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The image synthesis unit includes a first image signal acquired from the first imaging area, a second image signal acquired from the second imaging area, and a third image acquired from the third imaging area. And the fourth image signal acquired from the fourth imaging region to generate the composite image signal,
The distance calculation unit calculates a distance to the driver based on the first, second, third and fourth image signals,
The face recognition unit recognizes a three-dimensional feature point of the driver's face based on the composite image signal and the distance to the driver;
The face orientation determination unit determines the driver's face orientation based on the three-dimensional feature points,
2. The driver monitoring system according to claim 1, wherein the side-by-side / slumber determining unit determines whether or not the driver is looking aside according to a determination result of the driver's face orientation. Auxiliary illumination that emits near-infrared light is further provided,
The signal processing circuit includes:
An image synthesis unit that synthesizes the plurality of image signals to generate a synthesized image signal;
A distance calculation unit that calculates a distance to the driver; and a face direction determination unit that determines the driver's face direction;
The at least two optical regions are a first optical region having a first transmittance property and a first focusing property, a second transmittance property different from the first transmittance property, and the first A second optical region having a focusing characteristic, a third optical region having a second focusing characteristic different from the first transmittance characteristic and the first focusing characteristic, and the second transmittance. A fourth optical region having a characteristic and the second focusing characteristic;
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The image synthesizing unit synthesizes the first image signal acquired from the first imaging area and the second image signal acquired from the second imaging area to generate a first synthesized image signal. , Combining the third image signal acquired from the third imaging region and the fourth image signal acquired from the fourth imaging region to generate a second combined image signal, and Generating a third composite image signal by combining the second composite image;
The distance calculation unit calculates a distance to the driver based on the first and second composite image signals,
The face recognition unit recognizes a three-dimensional feature point of the driver's face based on the third composite image signal and the distance to the driver;
The face orientation determination unit determines the driver's face orientation based on the three-dimensional feature points,
2. The driver monitoring system according to claim 1, wherein the side-by-side / slumber determining unit determines whether or not the driver is looking aside according to a determination result of the driver's face orientation. A lens, a lens optical system including at least two optical regions having different optical characteristics, an image pickup device having at least two image pickup regions on which light transmitted through the lens optical system is incident, the lens optical system, and the image pickup device And an array of optical elements that allow light that has passed through the optical regions of the at least two optical regions to enter the imaging regions of the at least two imaging regions. A first and second imaging device having information on light that has passed through each optical region and outputting a plurality of image signals obtained from each imaging region;
Auxiliary illumination that emits near-infrared light,
A first image synthesis unit that synthesizes the plurality of image signals output from the first imaging device to generate a first synthesized image signal;
A second image synthesizing unit that synthesizes the plurality of image signals output from the second imaging device to generate a second synthesized image signal;
A distance calculation unit that calculates the distance to the driver using the amount of parallax obtained from the first and second composite image signals;
A three-dimensional face recognition unit that recognizes a three-dimensional feature point of the driver's face based on the first composite image signal and the distance to the driver;
A face orientation determination unit that determines the driver's face orientation based on the three-dimensional feature points;
A signal processing circuit including a side-by-side determination unit that determines whether or not the driver is looking aside according to a determination result of the driver's face orientation,
The at least two optical regions include a first optical region having a first transmittance characteristic, a second optical region having a second transmittance characteristic different from the first transmittance characteristic, the first and A third optical region having a third transmittance characteristic different from the second transmittance characteristic and a fourth optical region having a fourth transmittance characteristic different from the first, second and third transmittance characteristics. Including the optical region of
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The first image synthesizing unit is acquired from the first image signal acquired from the first imaging region, the second image signal acquired from the second imaging region, and the third imaging region. Generating the first combined image by combining the third image signal and the fourth image signal acquired from the fourth imaging region,
The second image synthesis unit is acquired from the first image signal acquired from the first imaging region, the second image signal acquired from the second imaging region, and the third imaging region. A driver monitoring system that generates the second composite image by combining the third image signal and the fourth image signal acquired from the fourth imaging region. An auxiliary illumination that irradiates near-infrared light that vibrates in the direction of the first polarization axis;
The signal processing circuit includes:
An image selection unit that selects one of the plurality of image signals;
An eye opening degree detecting unit for detecting an opening degree of the driver's eyes, and
The at least two optical regions include a first optical region having a first polarization characteristic that transmits light oscillating in the direction of the first polarization axis, and a second perpendicular to the direction of the first polarization axis. A second optical region having a second polarization characteristic that transmits light oscillating in the direction of the polarization axis;
The at least two imaging areas include a first imaging area corresponding to the first optical area and a second imaging area corresponding to the second optical area,
The image selection unit includes a saturation degree of a pixel in the first image configured from the first image signal acquired from the first imaging area, and a second level acquired from the second imaging area. Selecting either one of the first image signal and the second image signal by comparing the degree of saturation of a pixel in the second image composed of the image signal;
The face recognition unit recognizes feature points of the driver's face based on the selected image signal,
The eye opening detector detects an opening degree of the driver's eyes based on the facial feature points,
2. The driver monitoring system according to claim 1, wherein the aside and doze determination unit determines whether or not the driver is dozing according to the detection result of the opening degree of the eyes. Further comprising auxiliary illumination for irradiating light oscillating in the direction of the first polarization axis;
The signal processing circuit includes:
An image synthesis unit that synthesizes the plurality of image signals to generate a synthesized image signal;
An image selection unit that selects one of the plurality of image signals;
An eye opening degree detecting unit for detecting an opening degree of the driver's eyes, and
The at least two optical regions are a first optical region having a first polarization characteristic and a first transmittance characteristic that transmit light oscillating in the direction of the first polarization axis, the first polarization characteristic, and A second optical region having a second transmittance characteristic different from the first transmittance characteristic; a second optical region that transmits light that vibrates in a direction of a second polarization axis perpendicular to the direction of the first polarization axis; A third optical region having two polarization properties and the first transmittance property, and a fourth optical region having the second polarization property and the second transmittance property,
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The image synthesizing unit synthesizes the first image signal acquired from the first imaging area and the second image signal acquired from the second imaging area to generate a first synthesized image signal. A second image signal is generated by synthesizing the third image signal acquired from the third imaging region and the fourth image signal acquired from the fourth imaging region;
The image selection unit includes a saturation degree of pixels in the first image composed of the first composite image signal and a saturation degree of pixels in the second image composed of the second composite image signal. And selecting one of the first composite image signal and the second composite image signal,
The face recognition unit recognizes feature points of the driver's face based on the selected composite image signal,
The eye opening detector detects an opening degree of the driver's eyes based on the facial feature points,
2. The driver monitoring system according to claim 1, wherein the aside and doze determination unit determines whether or not the driver is dozing according to the detection result of the opening degree of the eyes. A lens, a lens optical system including at least two optical regions having different optical characteristics, an image pickup device having at least two image pickup regions on which light transmitted through the lens optical system is incident, the lens optical system, and the image pickup device And an array of optical elements that allow light that has passed through the optical regions of the at least two optical regions to enter the imaging regions of the at least two imaging regions. A first imaging device and a second imaging device for indicating information of light that has passed through each optical region and outputting a plurality of image signals obtained from the respective imaging regions;
Auxiliary illumination for irradiating near infrared light oscillating in the direction of the first polarization axis;
A first image selection unit that selects a first selection image signal from the plurality of image signals output from the first imaging device;
A second image selection unit that selects a second selection image signal from the plurality of image signals output from the second imaging device;
A distance calculation unit that calculates a distance to the driver using the amount of parallax obtained from the first and second selection image signals;
A face recognition unit for recognizing a two-dimensional feature point of the driver's face based on the first selected image signal;
An eye opening detection unit that detects an opening degree of the driver's eyes based on the two-dimensional feature points;
A dozing determination unit that determines whether the driver is dozing according to the detection result of the opening degree of the eyes,
A three-dimensional face recognition unit for recognizing a three-dimensional feature point of the driver's face based on the first selected image signal and the distance to the driver;
A face orientation determination unit that determines the face orientation of the driver based on the three-dimensional feature points of the face;
A signal processing circuit including a side-by-side determination unit that determines whether or not the driver is looking aside according to a determination result of the driver's face orientation,
The at least two optical regions include a first optical region having a first polarization characteristic that transmits light oscillating in the direction of the first polarization axis, and a second perpendicular to the direction of the first polarization axis. A second optical region having a second polarization characteristic that transmits light oscillating in the direction of the polarization axis;
The at least two imaging areas include a first imaging area corresponding to the first optical area and a second imaging area corresponding to the second optical area,
The first image selection unit is acquired from the saturation degree of the pixel in the first image composed of the first image signal acquired from the first imaging region and the second imaging region. One of the first image signal and the second image signal is selected as the first selection by comparing a saturation degree of a pixel in the second image composed of the second image signal. Select as image signal,
The second image selection unit is acquired from the saturation level of the pixels in the first image composed of the first image signal acquired from the first imaging region and the second imaging region. Comparing one of the first image signal and the second image signal to the second selection by comparing a degree of saturation of a pixel in the second image composed of the second image signal; Driver monitoring system to select as an image signal.   The driver monitoring system according to any one of claims 7 to 9, further comprising an auxiliary lighting control unit that turns off the auxiliary lighting according to a degree of saturation of the pixel after the auxiliary lighting is turned on. Auxiliary illumination that emits near-infrared light is further provided,
The signal processing circuit includes:
A face orientation determination unit for determining the driver's face orientation;
A heart rate estimator for estimating a heart rate from a change in the face color of the driver;
A physical condition determining unit that determines the physical condition of the driver based on the estimated heart rate,
The at least two optical regions include a first optical region having a first spectral characteristic that transmits the near infrared light, and a second optical having a second spectral property that transmits red light in a visible light band. A third optical region having a third spectral characteristic that transmits green light in the visible light band, and a fourth optical region having a fourth spectral characteristic that transmits blue light in the visible light band ,
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The face recognition unit includes a first face recognition unit that recognizes feature points of the driver's face based on a first image signal acquired from the first imaging region, the second, third, and A second face recognition unit that recognizes feature points of the driver's face based on at least one of the second, third, and fourth image signals respectively acquired from the fourth imaging region,
The face orientation determination unit determines the driver's face orientation based on the feature points recognized by the first face recognition unit,
The aside and dozing determination unit determines whether or not the driver is looking aside according to the determination result of the driver's face orientation,
The driving according to claim 1, wherein the heart rate estimation unit estimates a heart rate by detecting a change in the driver's face color by focusing on a facial feature point recognized by the second face recognition unit. Monitoring system. A lens, a lens optical system including at least two optical regions having different optical characteristics, an image pickup device having at least two image pickup regions on which light transmitted through the lens optical system is incident, the lens optical system, and the image pickup device And an array of optical elements that allow light that has passed through the optical regions of the at least two optical regions to enter the imaging regions of the at least two imaging regions. A first imaging device and a second imaging device for indicating information of light that has passed through each optical region and outputting a plurality of image signals obtained from the respective imaging regions;
Auxiliary illumination that emits near-infrared light,
A distance calculator that calculates the distance to the driver using the amount of parallax;
A first face recognition unit for recognizing a two-dimensional feature point of the driver's face;
A second face recognition unit for recognizing a three-dimensional feature point of the driver's face;
A face orientation determination unit that determines the driver's face orientation based on the facial feature points;
An aside determination unit that determines whether the driver is looking aside according to the determination result of the driver's face orientation;
A heart rate estimator for estimating a heart rate from a change in the face color of the driver;
A signal processing circuit including a physical condition determination unit that determines the physical condition of the driver based on the estimated heart rate,
The at least two optical regions include a first optical region having a first spectral characteristic that transmits the near infrared light, and a second optical having a second spectral property that transmits red light in a visible light band. A third optical region having a third spectral characteristic that transmits green light in the visible light band, and a fourth optical region having a fourth spectral characteristic that transmits blue light in the visible light band ,
The at least two imaging regions include a first imaging region corresponding to the first optical region, a second imaging region corresponding to the second optical region, and a third corresponding to the third optical region. An imaging region, and a fourth imaging region corresponding to the fourth optical region,
The distance calculation unit calculates a distance to the driver using a parallax amount obtained from a first image signal respectively acquired from the first imaging region of the first and second imaging devices,
The first face recognition unit is based on at least one of second, third, and fourth image signals respectively acquired from the second, third, and fourth imaging regions of the first imaging device. Recognizing the two-dimensional feature points of the driver's face,
The second face recognition unit is configured to provide a three-dimensional feature of the driver's face based on a first image signal acquired from the first imaging region of the first imaging device and a distance to the driver. Recognize the point,
The face orientation determination unit determines the driver's face orientation based on the three-dimensional feature points,
The aside look determination unit determines whether the driver is looking aside according to a determination result of the driver's face orientation,
The driver monitoring system, wherein the heart rate estimator estimates a heart rate by detecting a change in the driver's face color by paying attention to a two-dimensional feature point of the face. An attitude detection unit that detects the attitude of the driver based on the distance to the driver;
7. An air bag deployment control unit that determines whether or not to deploy an airbag based on a detection result of the posture detection unit and controls deployment and a deployment speed of the airbag, further comprising: The driver monitoring system according to any one of 9 and 12.   2. The driver monitoring system according to claim 1, further comprising a warning device that issues a warning to the driver according to a determination result of the armpit / slumber determination unit. A lens, a lens optical system including at least two optical regions having different optical characteristics, an image pickup device having at least two image pickup regions on which light transmitted through the lens optical system is incident, the lens optical system, and the image pickup device And an array-like optical element that causes light that has passed through each optical region of the at least two optical regions to enter each imaging region of the at least two imaging regions corresponding to each optical region. By analyzing at least one of the plurality of image signals output from the imaging device, having information on the light that has passed through each of the optical regions, and acquired from each of the imaging regions, A driver monitoring method for monitoring whether a driver is looking aside and / or asleep,
Recognizing feature points of the driver's face based on at least one image signal of the plurality of image signals;
And a step of determining whether or not the driver is looking aside or falling asleep based on the feature points of the face. A lens, a lens optical system including at least two optical regions having different optical characteristics, an image pickup device having at least two image pickup regions on which light transmitted through the lens optical system is incident, the lens optical system, and the image pickup device And an array-like optical element that causes light that has passed through each optical region of the at least two optical regions to enter each imaging region of the at least two imaging regions corresponding to each optical region. By analyzing at least one of the plurality of image signals output from the imaging device, having information on the light that has passed through each of the optical regions, and acquired from each of the imaging regions, A computer program used in a driver monitoring system for monitoring whether a driver is looking aside and / or is asleep. The driver monitoring system of the computer,
Recognizing feature points of the driver's face based on at least one image signal of the plurality of image signals;
And a step of determining whether or not the driver is looking aside or falling asleep based on the facial feature points.

Images