JP2017011634A - Imaging device, control method for the same and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable matching of the luminance level by performing AE processing on infrared light as in the case of visible light even when reflection light of infrared light varies according to distance or paint.SOLUTION: An imaging device includes a visible light imaging sensor for separating light passing through an imaging lens into visible light and infrared light and forming an image obtained from the visible light, an infrared light imaging sensor for forming an image obtained from the infrared light, image processing means for converting signals obtained from the visible light imaging sensor and the infrared light imaging sensor into digital image signals, object detection means for detecting an object from an acquired image, an AE processing unit configured to perform an AE process after detection of the object, and image combining means for combining the visible light image and the infrared light image after the AE processing. The visible light image is subjected to the AE processing over the entire image, the infrared light image is subjected to the AE processing based on surrounding information of the object detected by the object detecting means, and the visible light image and the infrared light image are matched with each other in luminance level and then subjected to image composition.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a program, and more particularly, to an imaging apparatus capable of capturing and synthesizing a visible light captured image and an infrared light captured image, a control method thereof, and a program.

  In recent years, in-vehicle cameras and surveillance cameras that use a visible light camera and an infrared light camera are widely used.

  In Patent Document 1, an obstacle such as a person or a car is picked up by an infrared light camera, a road surface is picked up by a visible light camera, and both images are combined so that an obstacle is generated even in an environment where fog is generated. And a method of obtaining an image in which information such as a white line on the road appears.

  In Patent Document 2, when a visible light image and an infrared light image are combined, a combination parameter is changed according to the surrounding environment, and an object detection image suitable for the object detection process is generated. A method has been proposed.

JP 09-37147 A JP 2013-247492 A

  However, in both Patent Documents 1 and 2, when an image captured with visible light and an image captured with infrared light are combined, a difference in luminance level becomes large, and an image with low utility value as a combined image I understood that. This problem is caused by the fact that the reflected light changes depending on the distance, paint, etc. in the infrared light, so that the luminance level is not uniform even if the AE treatment is performed as in the case of the visible light.

  Moreover, it has been found that this problem appears remarkably when photographing using infrared illumination. This is because there is a large difference in brightness between the subject that the infrared illumination reaches and the subject that the infrared illumination does not reach in a situation where the subject that the infrared illumination reaches and the subject that does not reach it are mixed. This is because the difference between the AE process and the AE process at the time of visible light photographing becomes large.

  The present invention provides an imaging device capable of obtaining an image having high utility value by aligning the brightness levels of images when combining an image captured with visible light and an image captured with infrared light. The purpose is to do.

  In order to achieve the above object, an imaging apparatus according to the present invention includes a visible light imaging unit that performs imaging using visible light, an infrared light imaging unit that performs imaging using infrared light, and the visible light imaging unit. Object detection means for detecting an object from a visible light captured image, infrared light exposure detection means for detecting the exposure amount of the infrared light imaging means, and infrared light exposure control means for controlling the exposure amount of infrared light An imaging device having a visible light captured image by the visible light imaging means and an image combining means for combining an infrared light captured image by the infrared light imaging means, wherein the infrared light exposure detection means is the target The infrared light exposure detection region in the infrared light captured image and the target brightness of the infrared light exposure control means are set according to the detection result by the object detection means.

  According to the imaging apparatus according to the present invention, the luminance levels of the visible light image and the infrared light image can be made uniform in the synthesis of the visible light image and the infrared light image. By aligning the brightness level of each image, an image with high utility value can be obtained.

It is a figure which shows the structure of the imaging device which concerns on embodiment of this invention. It is a figure which shows the structure of the imaging part (imaging means) concerning embodiment of this invention. It is a figure which shows the picked-up image in embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a diagram showing a configuration of a camera according to the first embodiment of the present invention. FIG. 2 is a diagram showing the detailed configuration of the visible light imaging unit (imaging means) 101 and the infrared light imaging unit 106. In the figure, some components in the imaging apparatus are omitted.

  In FIG. 1, a visible light imaging unit 101 and an infrared light imaging unit 106 include a lens 201 composed of several lens groups, a dichroic mirror 202, an infrared light imaging sensor 203, and a visible light imaging sensor 204 as shown in FIG. , An imaging light source aperture 205, an infrared light imaging aperture 206, a CDS circuit (not shown), an AGC amplifier, an A / D converter, and an LED light source that emits near-infrared light. In addition, an operation unit (not shown) is provided, and an AE processing area in visible light photographing can be set in the operation unit. The setting of the AE processing area refers to the entire image, center-weighted metering, spot metering, and the like.

  The light that has passed through the lens group is wavelength-separated by a dichroic mirror. Visible light passes through the dichroic mirror and forms an image on the visible imaging sensor 204. The visible imaging sensor includes a CCD sensor, a CMOS sensor, and the like. The infrared light is reflected by the dichroic mirror and imaged on the infrared light image sensor 203. The infrared light imaging sensor includes a CCD sensor, a CMOS sensor, an InGaAs sensor, and the like. Each image pickup device photoelectrically converts the formed optical image and outputs it as an analog image signal.

  Further, both the visible light imaging unit 101 and the infrared light imaging unit 106 are subjected to correlated double sampling processing on an electrical signal input from the imaging device by a CDS (Correlated Double Sampling) circuit (not shown). . Further, an AGC (Automatic Gain Control) amplifier (not shown) performs an amplification process on the electric signal input from the CDS circuit. Further, an analog signal amplified by the AGC amplifier is converted into a digital signal by an A / D converter (not shown).

  The digital signal converted by the visible light imaging unit 101 calculates an evaluation value for automatic exposure control (hereinafter referred to as AE) of the visible light image in the detection unit 102. Based on the luminance signal of the entire captured image, the AE evaluation value controls the gain at the shutter speed, the imaging aperture 205, and the AGC so as to achieve an appropriate exposure. Thereafter, the image signal processing unit 103 performs image processing such as color conversion, WB processing, and gamma correction.

  Then, the object detection unit 104 performs object detection processing from the image output from the image signal processing unit 103. In this embodiment, a face image is used as an object, and face detection processing is performed. As the face detection process, there is a method using an eigenface by principal component analysis. This method is described in “M.A. Turk and A.P. Pentland,“ Face recognition using eigenfaces ”, Proc. Of IEEE Conf. On Computer Vision and Pattern Recognition, pp. 586-591, 1991.” Further, as disclosed in Japanese Patent Laid-Open No. 9-251534, face detection processing may be performed by a method using feature points such as eyes, nose, and mouth.

  In these methods, it is determined whether or not the input image is a human face by a pattern matching method between the input image and a plurality of standard patterns. As a result of the face detection in the face detection process, information on the feature points of the face image (nose / mouth / face contour shape, sunglasses frame shape / color, hat shape / color, and their respective sizes and feature points For example). Note that the face detection process itself can be realized by a known technique as described above, and thus detailed description thereof is omitted here. Further, in this embodiment, the presence or absence of sunglasses is detected for the detected face image.

  The detection of the sunglasses is performed by comparing the average luminance of the detected peripheral area of the eye with a predetermined threshold value. If it is darker than the predetermined threshold, it is determined that the user is wearing sunglasses. In the face detection described above, the determination may be made based on the degree of eye feature point extraction, and the degree of edge in the area corresponding to the eye may be used as a determination criterion. As the face detection result, the position and size of the detected face and the average luminance of the face area are output to the AE processing unit 108.

  Since the imaging apparatus in the present embodiment shares an optical system from the objective lens to the dichroic mirror, the position (view angle) of the captured image is the same in visible light imaging and infrared light imaging.

  The visible light image data is temporarily stored in the image data storage unit 105.

  On the other hand, the digital signal converted by the infrared light imaging unit 106 calculates the evaluation value for AE of the infrared light image in the detection unit 107. The AE evaluation value is calculated over the entire image.

  Thereafter, the AE processing unit performs infrared light AE processing from the detection result of the object detection unit and the result of the detection unit 107. The infrared light AE process is performed on the face area of the object detection result, and the average luminance of the visible light image is set as the target luminance, and exposure control is performed so that the average luminance of the infrared light image approaches the target luminance. I do. The content of the exposure control is the control of the infrared illumination light source in addition to the shutter speed, the infrared imaging aperture 206, and the gain control in the AGC, as in the above-described visible light AE process. Further, the difference between the target luminance and the infrared image luminance is output to the image signal processing unit.

  The image signal processing unit 109 performs image processing such as gamma correction and outputs the processed image to the image composition unit 110. At this time, the γ value of γ conversion is corrected in order to fill the difference between the target luminance from the AE processing unit and the infrared luminance.

  In the image composition unit 110, the infrared light captured image after image processing and the visible light captured image temporarily stored in the image data storage unit are combined. In the combining process, a combined area is obtained from the above-described detection result of sunglasses, and the sunglasses area is combined with an infrared light image and the other areas are combined with a visible light image. The image data that has undergone the image composition processing is displayed by the image display unit 107.

  FIG. 4 shows an example of a result obtained by photographing a person wearing sunglasses using infrared illumination. 4A shows a visible light image of a person wearing sunglasses, FIG. 4B shows an infrared light image in which exposure control is performed using an infrared image alone, and exposure control is performed using only a visible light image and an infrared image. 4C is a composite image with the infrared light image, FIG. 4D is an infrared light image subjected to exposure control according to the present embodiment, and red is subjected to exposure control according to the visible light image and the present embodiment. A composite image with the external light image is shown in FIG.

  In FIG. 4A, a face image with sunglasses and a background are photographed. On the other hand, in the infrared light image in which exposure control is performed with the infrared image alone shown in FIG. 4B, exposure is performed on the person in the illumination range by infrared illumination and the entire background outside the illumination range of infrared illumination. Since the control is performed, the half of the background that is outside the illumination range of the infrared illumination is shown, and the face image that is the illumination range of the infrared illumination is whiteout. Since the face image is whiteout, the eyes behind the sunglasses cannot be confirmed, and the eyes behind the sunglasses cannot be confirmed even in the composite image FIG.

  On the other hand, in the infrared light image subjected to exposure control according to the present embodiment, the background of the visible light image shown in FIG. 4A is extracted and exposure control is performed at the face position. However, the whiteout of the face image is suppressed. For this reason, in FIG. 4E, the synthesized image is synthesized in such a way that the eyes behind the sunglasses can be visually recognized, and the luminance is also uniform, so that a synthesized image with no sense of incongruity is obtained.

  According to the above-described embodiment, it is possible to align the luminance levels of the visible light image and the infrared light image in the synthesis process of the visible light image and the infrared light image. Since the synthesized part looks natural, the quality of the synthesized image can be improved and the recognition accuracy can be improved.

[Other Embodiments]
In the first embodiment, the visible light photographed image is subjected to AE processing as usual, and the infrared light photographed image is detected by the object detection means, the AE processing is performed from the surrounding information, and the visible light photographed image is obtained. And the synthesizing means performed after matching the luminance levels of the infrared light image. However, it is not always necessary to detect the object, and the AE process may be performed on the fixed area in the image.

<Visible area>
In the first embodiment, the AE process for the entire image is performed. However, center-weighted metering, spot metering, or the like may be used.

101 imaging unit, 102 object detection unit, 103 AE processing unit,
104 image signal processing unit, 105 image data storage unit, 106 image composition unit,
107 Image display

Claims (5)

  A visible light imaging means for photographing with visible light; an infrared light imaging means for photographing with infrared light; an object detection means for detecting an object from a visible light image captured by the visible light imaging means; Infrared light exposure detecting means for detecting the exposure amount of the external light imaging means, an infrared light exposure control means for controlling the exposure amount of the infrared light, a visible light image captured by the visible light imaging means, and the infrared light imaging An image synthesizing unit that synthesizes an infrared light image captured by the unit, wherein the infrared light exposure detection unit is configured to detect the infrared light image in the infrared light image according to the detection result of the object detection unit. An imaging apparatus, wherein an infrared light exposure detection area and a target luminance of the infrared light exposure control means are set.   Visible light imaging means for photographing with visible light, visible light exposure detecting means for detecting the exposure amount of the visible light imaging means, visible light exposure control means for controlling the exposure amount of visible light, and photographing with infrared light Infrared light imaging means for performing infrared light exposure detecting means for detecting the exposure amount of the infrared light imaging means, infrared light exposure control means for controlling the exposure amount of infrared light, and visible light by the visible light imaging means An imaging apparatus having an optical image and an image synthesis unit that synthesizes an infrared light image captured by the infrared light imaging unit, the visible light exposure detection region of the visible light exposure detection unit, and the infrared light exposure The infrared light exposure detection area of the detection means can be set independently, and the target luminance of the infrared light exposure control means is the luminance of the visible light photographed image.   The imaging apparatus according to claim 2, wherein the visible light AE processing unit controls a shutter speed, an imaging aperture, and gain control in the AGC of the imaging unit.   The infrared light AE processing means further has an infrared illumination, and controls the shutter speed of the imaging means, the photographing aperture, the gain control in the AGC, and the light amount of the infrared light illumination. The imaging device according to claim 2.   The visible light imaging means and the infrared light imaging means share a part of the optical system, and the wavelength separation means wavelength-separates visible light and infrared light to obtain a visible light image and an infrared light image. The imaging apparatus according to any one of claims 1 to 4, wherein the imaging apparatus is characterized in that