JP2020091453A - Control device, image capturing apparatus and control method thereof - Google Patents

Abstract

To capture an image with high quality by accurately using an optical filter.SOLUTION: A control device controls an image capturing apparatus having an image sensor for outputting an image signal by photoelectrically converting light incident through an imaging optical system including an optical filter and driving means for changing a position of the optical filter. The control device includes control means which controls the driving means so as to gradually insert the optical filter into an opening of the imaging optical system, determines the position of the optical filter on the basis of luminance of the image obtained by the image sensor while gradually inserting the optical filter, and controls the driving means so that the position of the optical filter is the determined position.SELECTED DRAWING: Figure 3

Description

The present invention relates to a control device, an imaging device and a control method thereof.

2. Description of the Related Art Conventionally, in an image pickup device such as a network camera, there is an image pickup device capable of taking an image with visible light and infrared light by using an image pickup element having sensitivity to not only visible light but also infrared light. In such an imaging device, a method is used in which an infrared cut filter that cuts infrared light and transmits only visible light is inserted into and removed from the optical path of the imaging optical system according to the illuminance of the subject, and the shooting mode is switched. Has been. Hereinafter, this method is referred to as "auto day night". In such an image pickup apparatus, a mode in which an infrared cut filter is attached to acquire a color image in YUV format or RGB format (day mode), and a mode in which the infrared cut filter is removed to acquire a monochrome image (night mode) are used. Are switching. In the night mode, while losing the color information, it becomes possible to recognize the subject even in a low illuminance environment by incorporating infrared light.

Patent Document 1 presents a method of detecting the illuminance (that is, the brightness) of a subject based on the pixel signal level in order to switch the shooting mode. Specifically, the pixel signal level at the time of shifting from the day mode to the night mode (hereinafter referred to as “night shift threshold”) is stored, and a signal level obtained by adding a certain amount to this level is changed from the night mode to the day mode. The threshold for switching to (hereinafter, referred to as “day shift threshold”).

In recent years, there has been an increasing demand for expanding the dynamic range of network cameras for acquiring images for recognition in the field of surveillance and in-vehicle use. For example, when photographing the license plate of a car with the headlights turned on at night, if the exposure of the camera is adjusted so that the exposure of the license plate is appropriate, the headlights in the vicinity of the license plate will be overexposed. It causes halation. As a result, the halation spreads to the license plate and the contrast decreases, making recognition difficult. On the other hand, when the exposure of the camera is adjusted so that the headlight does not cause halation, the license plate, which has a lower brightness than the headlight, is crushed in black, which makes recognition difficult.

In Patent Document 2, as a technique for photographing a license plate without causing halation in such photographing, a filter that blocks visible light and transmits infrared light (hereinafter, referred to as "visible cut filter"). The method of using is presented. For example, assume that a car is photographed in night mode at night and the photographing range is illuminated by using infrared light illumination, or the ambient light includes infrared light. In such a case, when the halation of the headlight is detected while photographing the license plate of the car, the visible cut filter is attached to the lens of the camera. In recent years, the light source of the headlight has changed from the conventional halogen light to the LED, but since many LED type headlights have a spectrum only in the visible light, the light from the headlight can be drastically changed by passing the visible cut filter. The amount of light can be reduced. Therefore, by mounting the visible cut filter, exposure adjustment for photographing the license plate becomes easy. The shooting mode in which the halation is suppressed by using the visible cut filter in this way is hereinafter referred to as a “halation suppression mode”.

Further, in recent years, there has been a demand for network cameras for road monitoring and the like to photograph the face of a driver of a car even if the windshield of the car is reflected. In response to such a demand, there is a photographing method in which a polarizing filter is attached to an imaging optical system to suppress the reflection of the windshield.

Focusing on the reflected light that is reflected by the windshield, the proportion of P-polarized light in the reflected light that is incident on the windshield at an angle near the Brewster's angle (about 56 degrees for windshield and air) is low, and S-polarized light is Many are included. Therefore, if the polarizing filter is installed so that the polarization axis is parallel to the S-polarized light, the S-polarized light component is blocked and the reflected light on the windshield surface is removed. It is possible to take a picture of the face. The light reflected by the driver's face can be regarded as non-polarized light although it has low brightness, and therefore passes through a certain amount of the polarizing filter. However, the polarization filter has a demerit that the light received by the camera is reduced because the light in the polarization axis direction is blocked even with respect to unpolarized natural light, and the entire captured image becomes dark.

On the other hand, in Japanese Patent Application Laid-Open No. 2004-242, a technique in which a polarization filter is attached to a camera to take an image when the brightness of the image is brighter than a predetermined value, and the polarization filter is removed from the camera when the brightness of the image is less than a predetermined value Is proposed. For the sake of convenience, hereinafter, a mode in which a polarizing filter is attached and an image is taken will be referred to as a “reflection suppressing mode”.

As described above, in the network camera, the photographing mode is switched by using the optical filters such as the infrared cut filter, the visible cut filter, and the polarization filter, which have different transmittances for each wavelength or each polarization direction, and are highly recognizable. You are getting an image.

JP, 2004-120202, A JP, 2016-220002, A JP, 2010-109562, A

However, there are the following problems in switching the shooting modes associated with the insertion and removal of these optical filters. One is the issue regarding Auto Day Night.

In general, color information is important in a network camera for monitoring purposes, so it is desirable to perform color photography (day mode) even under lower illuminance. However, when the method described in Patent Document 1 is used, the following trade-off occurs.

(1) If the difference between the day transition threshold and the night transition threshold is reduced in order to switch from a lower illuminance to the day mode, the illuminance is insufficient for the day mode, and the day mode and the night mode are switched. There is a possibility of causing hunting by repeating the transition of.

(2) If the difference between the day shift threshold and the night shift threshold is set large in order to reduce the possibility of hunting, the switching timing from the night mode to the day mode will be delayed and the illuminance will be sufficient for color photography. Nevertheless, it does not switch to day mode.

(1) occurs because it is not possible to distinguish visible light from infrared light from a photographed image in night mode. The more infrared light there is, the more the day mode shifts to day mode even if the illuminance is insufficient. More likely.

Another problem is a problem when shifting to the halation suppression mode. Patent Document 2 discloses a method of determining the timing of inserting a visible cut filter, but only discloses that the night mode is shifted to after a certain period of time, and the visible cut filter should be removed. No method is disclosed for determining the proper timing. This is because the presence or absence of halation is determined based on the brightness value of the subject. That is, since visible light is cut off once in the halation suppression mode, the image does not change even if the vehicle turns off the headlights, so it cannot be determined that the mode may be returned to the night mode. For monitoring purposes, since it is desired to acquire an image with a high SN ratio and good visibility, it is desirable to immediately return to night mode and capture visible light if there is no halation. Further, if day mode photography is possible, it is desirable to quickly return to day mode and perform color photography to acquire color information. However, after switching to the halation suppression mode, it is not possible to detect the turning off of the headlights, so there is a problem that shooting is continued in the halation suppression mode even in a state in which the mode can be switched to the day mode or the night mode.

Furthermore, another problem is a problem when shifting to the reflection suppression mode. The reflection of the windshield of the car cannot be suppressed by the polarization filter depending on the position of the camera that captures the image, the position of the car that is the subject, the angle of the windshield, and the positional relationship of the reflected object (cloud, building, sun, etc.). There are cases. As the incident angle of the light emitted from the reflected object seen from the windshield is farther from the Brewster's angle, the P-polarized component is mixed with the reflected light, so that even if the S-polarized component is blocked by the polarizing filter. , P-polarized component remains, and as a result, the reflection remains.

As described above, there are two types of glare, one that can be suppressed by setting the polarization filter at an appropriate angle and the other that cannot be suppressed. When a picture is taken with a polarization filter inserted despite the reflection being uncontrollable, the polarization filter only impairs the brightness of the entire photographed image and deteriorates the image quality. Patent Document 3 discloses a method of inserting a polarizing filter and photographing when the brightness of an image is a predetermined value or more. However, according to this method, even when "unrestricted reflection" is generated on the windshield of the car, if the brightness value of the image is equal to or more than the threshold value, the polarizing filter is inserted. As a result, the glare is not removed, and the image quality is deteriorated because the image is taken only by unnecessarily reducing the light amount.

Furthermore, another problem is a problem in transition from the reflection suppression mode to another mode. According to the method disclosed in Patent Document 3, even if the ambient light (the position of the sun) changes and the reflection disappears from the situation in which the image is captured with a polarization filter attached, the polarization Shooting continues with the filter attached. This is because it is not possible to determine from the image taken with the polarizing filter attached to the camera whether or not the reflection is present in the first place. For this reason, it is not possible to determine when to remove the polarization filter, and even though there is no reflection, the image quality is deteriorated because shooting is performed with the light amount unnecessarily reduced. ..

The present invention has been made in view of the above problems, and an object of the present invention is to accurately use an optical filter to capture a high-quality image.

In order to achieve the above-mentioned object, the control device of the present invention changes the positions of an image pickup device that photoelectrically converts light incident through an imaging optical system including an optical filter and outputs an image signal, and the position of the optical filter. A drive unit for controlling the image pickup apparatus, the control unit controlling the drive unit so as to insert the optical filter stepwise into the opening of the imaging optical system, Control for deciding the position of the optical filter based on the brightness of the image obtained by the image pickup device while inserting the filter step by step, and controlling the driving means so that the position of the optical filter becomes the decided position. Have means.

According to the present invention, it is possible to capture a high-quality image by accurately using the optical filter.

The figure which shows schematic structure of the imaging device in the 1st Embodiment of this invention. The figure explaining control of the infrared cut filter in 1st Embodiment. 6 is a flowchart showing processing when shifting from night mode to day mode in the first embodiment. FIG. 6 is a diagram illustrating an infrared cut filter and exposure control according to the second embodiment. The figure explaining the relation between the insertion state of an infrared cut filter and a diaphragm in a 2nd embodiment. 9A and 9B are views for explaining a problem when an insertion state of an infrared cut filter and a diaphragm are changed in the third embodiment. The figure explaining the infrared cut filter and control of exposure in a 3rd embodiment. The figure explaining control of an infrared cut filter and exposure time in a 4th embodiment. The figure explaining the relationship between the position of a visible cut filter and halation in a 5th embodiment. The flowchart which shows the process at the time of switching from night mode to halation suppression mode in 5th Embodiment. The figure which shows the structure of the optical filter in 6th Embodiment. The figure explaining the relationship between the position of a visible cut filter, and halation in a 6th embodiment. The figure which shows schematic structure of the imaging device in 7th Embodiment. The figure which shows the relationship between the insertion state of a polarization filter and a reflection in 7th Embodiment. The figure which shows the relationship between the insertion state of a polarization filter and a reflection in 7th Embodiment. The flowchart which shows the switching control between day mode and glare suppression mode in 7th Embodiment. The figure which shows the structure of the optical filter in 8th Embodiment. The figure which shows the structure of the optical filter in 9th Embodiment. The figure which shows the schematic structure of the imaging device in 10th Embodiment. The figure explaining control of the optical filter in a 10th embodiment.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative positions, and the like of the component parts exemplified in the present embodiment should be appropriately changed depending on the configuration and various conditions of the device to which the present invention is applied, and the present invention However, the present invention is not limited to this.

<First Embodiment>
FIG. 1 shows a schematic configuration of the image pickup apparatus according to the first embodiment of the present invention. In FIG. 1, an image pickup apparatus 100 includes an image forming optical system 101, an image pickup element 102 having sensitivity to visible light and infrared light, a control unit 103, and a filter driving unit 104. The image forming optical system 101 includes a lens 105. And 108, a diaphragm 106, and an optical filter 107. The image sensor 102 photoelectrically converts light incident through the imaging optical system and outputs an image signal.

Although the imaging optical system 101 has two lenses here for simplification of the drawing, it may have three or more lenses. In addition, in the first embodiment, the optical filter 107 is an infrared cut filter, and is also referred to as an infrared cut filter 107 hereinafter. Although the infrared cut filter 107 is between the diaphragm 106 and the lens 108 in FIG. 1, it may be between the lens 105 and the diaphragm 106, or the diaphragm 106 may be omitted. The infrared cut filter 107 is preferably arranged near the position where the diameter of the light beam propagating between the lens 105 and the lens 108 is the smallest.

If the infrared cut filter 107 is provided in the vicinity of the image sensor 102, if the infrared cut filter 107 is inserted stepwise as described later, the captured image is displayed along the edge of the infrared cut filter 107. In addition, an unnatural luminance step is generated, which causes deterioration of image quality. On the other hand, as described above, by inserting the infrared cut filter stepwise at the position where the diameter of the light beam in the imaging optical system 101 is small, the image is almost equal to the state where only the infrared light is narrowed, and the image There is an effect that only the overall brightness is lowered and an unnatural image is not produced. Further, in order to suppress a focus shift (optical path length difference) caused by the insertion/removal of the infrared cut filter 107, a transparent glass that is inserted/removed complementarily to the infrared cut filter in the imaging optical system 101 is used to adjust the focus shift. May be performed.

The opening and closing of the diaphragm 106 is controlled by the control unit 103. The infrared cut filter 107 is inserted/removed in the X direction by the filter driving unit 104. The imaging optical system 101 forms a subject image on the image sensor 102. The control unit 103 drives the image sensor 102, controls the exposure time and gain, and reads signals from the image sensor 102. Further, the control unit 103 controls the operation of the filter driving unit 104 based on the pixel signal from the image sensor 102.

The image pickup apparatus 100 has a day mode (first photographing mode) in which the infrared cut filter 107 is inserted in the optical path to acquire a color image in YUV format or RGB format. It also has a night mode (second shooting mode) in which the infrared cut filter 107 is retracted from the optical path to obtain a monochrome image, and the day mode and the night mode are switched to drive.

Next, the brightness of the subject (that is, the brightness due to the visible light component) when the infrared cut filter 107 is completely inserted in the optical path is predicted according to the plurality of images in which the insertion amount of the infrared cut filter 107 is changed. The method will be described. Note that how to use the prediction method described here for determination when switching the day/night mode will be described later.

FIG. 2 shows a view of the imaging optical system 101 from the incident light side (−Z side), a photographed image at that time, and the relationship between the insertion amount of the infrared cut filter 107 and the luminance of the photographed image. In order to make the description easy to understand, only the diaphragm 106 and the infrared cut filter 107 are shown and the others are omitted. In addition, the ambient light in the shooting range is light having both visible light and infrared light components, such as sunlight and incandescent lamps.

In FIG. 2, states (a) to (e) show how the infrared cut filter 107 is gradually inserted into the optical path by the control unit 103. From the state (a) to the state (e), the range in which the opening of the diaphragm 106 and the infrared cut filter 107 overlap with each other expands. The range where the aperture of the diaphragm 106 and the infrared cut filter 107 overlap is an area where infrared light is blocked and only visible light is transmitted. The aperture of the diaphragm 106 that does not overlap the infrared cut filter 107 is an area through which visible light and infrared light pass. From state (a) to state (e), since the range through which only visible light is transmitted gradually increases, the illuminance ratio of the subject due to the infrared light component gradually decreases, resulting in shooting. The entire image gradually darkens.

As can be seen from FIG. 2, if the area (area A) of the aperture of the diaphragm 106 in each state and the area (area B) of the infrared cut filter 107 overlapping the aperture are known, the state is shown in (e). It is possible to predict the brightness when the infrared cut filter 107 is completely inserted. Equation (1) shown below is an equation used to predict the luminance (luminance e) in the state (e) in which the infrared cut filter 107 is completely inserted.
Luminance e=luminance a−(luminance a−luminance b)×{area A/(area A−area B)} (1)

The brightness a and the brightness b indicate the brightness of the subject in the states (a) and (b), respectively. It should be noted that the brightness in the state (c) and the brightness in the state (d) may be used as the brightness a and the brightness b, and the brightness in a state in which the infrared cut filter 107 is inserted by an arbitrary amount other than the state shown in FIG. May be used. In addition, the brightness of three or more points may be used for the brightness prediction. When using the brightness of three or more points, the brightness value of the subject may be predicted by the least square method or the like.

Next, in the first embodiment, how to use the method of predicting the brightness of the subject when the infrared cut filter 107 is completely inserted in the optical path to determine whether to switch from night mode to day mode Will be described.

FIG. 3 is a flowchart showing the flow when switching from night mode to day mode. First, in S1, shooting in the night mode is started, and in S2, it is determined whether or not the brightness of the subject is equal to or higher than a predetermined first threshold value. The first threshold value here may be an arbitrary value, but a lower value allows color imaging from lower illuminance.

Next, in S2, when it is determined that the brightness of the subject is equal to or higher than the first threshold value, the process proceeds to S3, and the stepwise insertion of the infrared cut filter 107 is started. Here, the brightness of the subject is acquired while gradually inserting the infrared cut filter 107 using a plurality of frames, and the brightness when the infrared cut filter 107 is completely inserted is predicted by using Expression (1). To do. It should be noted that the insertion amount of the infrared cut filter 107 may be set to an arbitrary amount for each frame. However, the smaller the insertion amount, the smaller the decrease in the brightness of the captured image when the infrared cut filter 107 is inserted in S3. Become.

In S4, the brightness predicted in S3 is compared with the second threshold value. If the brightness is equal to or higher than the second threshold value, the shift to the day mode is confirmed, and the process proceeds to S6. It should be noted that the second threshold is a low illuminance photographing performance of the image pickup apparatus 100 (a value determined by the sensitivity and exposure time of the image pickup element 102, the F value of the imaging optical system, etc.) when performing color photographing in the day mode. It is better to set the illuminance required by. With such a value, color photography can be started from a lower illuminance.

In S6, the infrared cut filter 107 is completely inserted in the optical path, and the day mode is switched to perform color photography. Further, after the shift to the day mode is determined by the determination in S4, the insertion speed of the infrared cut filter 107 may be changed to drive all at once and the time required for the shift may be shortened. Further, it is possible to determine the exposure condition for obtaining the proper exposure of the subject in the day mode from the predicted brightness value. For this reason, the control unit 103 may change the exposure condition so as to suppress a large change in luminance in accordance with the timing of inserting the infrared cut filter 107 at once. By controlling in this way, it is possible to quickly shoot with proper exposure after shifting to day mode.

Further, in S3, the change of the color difference signal is acquired while inserting the infrared cut filter stepwise by using a plurality of frames to predict the adjustment value necessary for proper white balance when switching to the day mode. It is also possible to use. In this case, if the day mode shift is confirmed in the determination of S4, the white balance adjustment value is changed at the same time as the shift to the day mode, so that it is possible to quickly obtain an appropriate captured image after the day mode is switched.

On the other hand, in S4, when the brightness value predicted in S3 is less than the second threshold value, it is determined that the illuminance is insufficient for the day mode, the process proceeds to S5, and the infrared cut filter 107 is set outside the optical path. And then return to shooting in night mode. At this time, the first threshold value may be updated to a value higher than the previous value, or a negative offset may be set for the brightness value of the subject measured in S2. In either case, if the difference between the brightness value predicted in the determination in S4 and the second threshold value is added to the first threshold value or subtracted from the subject brightness, the determination accuracy in S2 is improved. be able to.

Note that, with the above-described method, the luminance can be predicted from at least two frames, but the luminance prediction may be continued over more frames to improve the accuracy of the predicted luminance. .. In addition, when the luminance is continuously predicted over a plurality of frames, if the ratio of visible light to infrared light significantly changes to a third threshold value or more in only some frames, the external lighting is turned on. The frame may be excluded from the determination by predicting that it is a disturbance component due to. Here, the ratio of visible light to infrared light is the amount of change in luminance when the infrared cut filter 107 is inserted in constant areas. The smaller the third threshold value is, the stronger the disturbance becomes, but the larger the third threshold value is, the smaller the number of frames required for the determination of the day mode switching is preferable.

As described above, according to the first embodiment, even if the first threshold value is set low, the infrared cut filter is inserted stepwise to predict the brightness in the day mode, and the predicted brightness is set. It becomes possible to perform switching control based on. By such control, it is possible to switch to the day mode without causing hunting.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. Note that the configuration of the image capturing apparatus according to the second embodiment is the same as that of the image capturing apparatus 100 according to the first embodiment described with reference to FIG. However, since the control by the control unit 103 of the second embodiment is different from that of the first embodiment, this difference will be described below.

In addition, also in the second embodiment, the determination at the time of mode switching is performed according to the flowchart of FIG. The difference from the first embodiment is that in S3, the infrared cut filter is inserted stepwise while performing automatic exposure adjustment (AE) by changing the exposure time of the image sensor 102, changing the gain, and the like. In the second embodiment, the diaphragm 106 is not changed. Hereinafter, a specific operation will be described with reference to FIG.

FIG. 4 shows captured images from the nth frame to the (n+6)th frame. First, from the n-th frame to the (n+1)-th frame, the infrared cut filter 107 is inserted into the aperture of the diaphragm 106 so as to increase by a certain amount of area. As a result, the illuminance of the subject due to the infrared light component is reduced, and the image in the (n+1)th frame is darkened by a certain amount. The control unit 103 calculates the exposure conditions necessary for proper exposure from the image of the (n+1)th frame, and changes the exposure conditions such as the exposure time and the gain of the image sensor 102 over the (n+2)th frame (AE) to obtain the proper exposure. To do. Further, the infrared cut filter 107 is inserted into the aperture of the diaphragm 106 by a certain amount of area between the (n+2)th frame and the (n+3)th frame, but the (n+3)th frame has the same brightness as the (n+1)th frame. Thereafter, recovery to proper exposure by AE and stepwise insertion of the infrared cut filter 107 are repeated.

In general, in exposure control by AE, the width of exposure change per frame is often limited to improve resistance to external disturbance (noise such as momentarily receiving light of unintended illumination). For this reason, if a large change in the brightness of the subject occurs, it will take time until the proper exposure is achieved. If blackout or whiteout occurs during this period, the quality of the image deteriorates.

On the other hand, in the second embodiment, by suppressing the insertion amount of the infrared cut filter 107 within the light amount that can be changed by the AE during one frame, it is possible to reduce the change in the brightness of the captured image, and to reduce Increase the number of frames of an image in an exposure state close to exposure. That is, in the present embodiment, the horizontal movement amount of the infrared cut filter 107 is large at the end of the aperture of the diaphragm 106 and the center of the aperture of the diaphragm 106 so that the insertion area is constant for each frame. Make it smaller in the vicinity. Although the amount of change may be arbitrary for each frame, when the luminance change amount is acquired over a plurality of frames to increase the accuracy of the day mode transition determination, the infrared cut filter insertion amount is set to be a constant area. It is preferable to do so because the change in brightness becomes linear and the brightness prediction becomes easier.

Further, in predicting the brightness in the day mode, the brightness of the frames before and after the infrared cut filter 107 is inserted (for example, the (n+1)th and (n+3)th frames, the (n+3)th frame and the (n+5th)th frame are used, and It may be calculated from the equation (1). Alternatively, by looking at the change in brightness across a plurality of frames, the total amount of light changed by the AE control is divided from the change amount, and from that value and the total area value of the inserted infrared cut filter, Equation (1) is obtained. Similar calculations may be performed. Although the infrared cut filter is inserted and the exposure is controlled between each frame in FIG. 4, a plurality of frames may be used for the insertion of the infrared cut filter and the exposure adjustment by the AE.

As described above, according to the second embodiment, it is possible to implement the stepwise insertion of the infrared cut filter and the mode switching determination while suppressing the brightness variation of the subject, and it is possible to switch to the day mode while preventing hunting. become.

In the second embodiment, switching from the night mode to the day mode is described, but the AE control can also be applied when switching from the day mode to the night mode, and the brightness variation of the subject can be changed. The amount can be suppressed.

<Third Embodiment>
Next, a third embodiment of the present invention will be described. The configuration of the image capturing apparatus according to the third embodiment is similar to that of the image capturing apparatus 100 according to the first embodiment described with reference to FIG. However, since the control by the control unit 103 of the third embodiment is different from that of the first embodiment, this difference will be described below.

In the third embodiment as well, the determination at the time of mode switching is performed according to the flowchart in FIG. 3, but here it is assumed that the aperture 106 is photographed in a small aperture state in the night mode. In the present embodiment, similarly to the second embodiment, the exposure adjustment by AE is also performed during the period in which the infrared cut filter 107 is gradually inserted in S3. FIG. 5 shows the area ratio of the aperture of the diaphragm 106 and the infrared cut filter 107 before and after the diaphragm is opened.

In FIG. 5A, the diaphragm 106 is in a small diaphragm state, and the infrared cut filter 107 is inserted up to a position of 20% in area ratio with respect to the opening of the diaphragm 106. FIG. 5B shows a state in which the exposure adjustment by AE is performed from the state of FIG. 5A, and as a result, the diaphragm 106 is opened. In FIG. 5B, although the position of the infrared cut filter 107 has not changed from that of FIG. 5A, the area occupied by the infrared cut filter 107 relatively increases due to the opening of the diaphragm 106. It occupies 40% of the aperture of the diaphragm 106. In this way, when the aperture 106 changes due to AE, the luminance change due to the insertion of the infrared cut filter 107 and the luminance change due to the aperture change are mixed, and the luminance prediction becomes complicated.

FIG. 6 shows a change in the ratio of infrared light to visible light when the diaphragm 106 and the infrared cut filter 107 both move. For the sake of explanation, the area where the infrared cut filter 107 does not overlap in the aperture opening is referred to as area A, and the area where the infrared cut filter 107 overlaps in the aperture opening is referred to as area B.

When the infrared cut filter 107 is inserted into the aperture of the diaphragm 106 by 20% from the nth frame to the (n+1)th frame, the brightness of the subject in the (n+1)th frame is reduced by the amount of the reduced infrared light component. .. Next, when the diaphragm 106 is opened by the AE from the (n+1)th frame to the (n+2)th frame, the ratio of the area B to the area A is substantially increased. Therefore, the ratio of infrared light is further reduced from N+1 frames. Next, the infrared cut filter 107 is inserted from the (n+2)th frame to the (n+3)th frame. Here, it is assumed that 60% is inserted into the aperture of the diaphragm 106. When the diaphragm 106 is further opened by the AE from the (n+3)th frame to the (n+4)th frame, the ratio of the area B to the area A is substantially decreased by opening the diaphragm 106 this time. Therefore, the ratio of infrared light in the (n+4)th frame increases. Finally, since the infrared cut filter 107 is completely inserted in the (n+5)th frame, the infrared light ratio becomes 0%.

A linear interpolation between the ratio of the infrared light in the nth frame and the ratio of the infrared light in the (n+5)th frame (0%) (dotted line C in FIG. 6) indicates that the diaphragm 106 is moved by the AE. The change in the ratio of infrared light when there is no light is shown, which corresponds to the second embodiment. When the ratio of the infrared light changes linearly in this way, the brightness of the subject can be easily predicted by using the above-mentioned formula (1). On the other hand, when the diaphragm 106 is moved by the AE, the change in the infrared light ratio changes as shown by the solid line D in FIG. As shown in FIG. 6, in the (n+2)th frame and the (n+4)th frame, the change in the infrared light ratio due to the movement of the diaphragm 106 is mixed with the change in the infrared light ratio due to the stepwise insertion of the infrared cut filter 107. It becomes difficult to predict the brightness of the subject.

Therefore, in the third embodiment, when the diaphragm 106 is operated by the AE, the infrared cut filter 107 is moved in such a direction as to cancel the ratio of infrared light that changes when the diaphragm 106 is opened. This makes it possible to measure the change in the infrared light ratio only by inserting the infrared cut filter 107, and to easily predict the brightness of the subject.

FIG. 7 shows how the ratio of infrared light changes with respect to the operations of the diaphragm 106 and the infrared cut filter 107 in this embodiment. The nth frame, the (n+1)th frame, and the (n+3)th frame are the same as those in FIG. 6, but when the diaphragm 106 is opened by the AE in the (n+2)th frame, the infrared cut filter 107 is pulled out from the diaphragm opening portion by a certain amount, Control is performed so that the ratio of the area A and the area B does not change. Further, in the (n+4)th frame, when the diaphragm 106 is opened, the infrared cut filter 107 is further inserted by a fixed amount so that the ratio of the area A and the area B does not change. In this control, the relationship between the position of the infrared cut filter 107 and the F value of the diaphragm 106 may be held in the control unit 103 in advance as a table, or may be calculated by a function.

By controlling in this way, the insertion amount of the infrared cut filter 107 for each frame can always be made constant in the area ratio with respect to the open portion of the diaphragm 106. Therefore, it is possible to easily predict the brightness of the subject when the infrared cut filter 107 is completely inserted by using the formula (1).

As described above, according to the third embodiment, it is possible to implement the stepwise insertion of the infrared cut filter and the mode switching determination while suppressing the brightness variation of the subject, and it is possible to switch to the day mode while preventing hunting. become.

<Fourth Embodiment>
Next, a fourth embodiment of the present invention will be described. Note that the configuration of the imaging device according to the fourth embodiment is similar to that of the imaging device 100 according to the first embodiment described with reference to FIG. However, since the control by the control unit 103 of the fourth embodiment is different from that of the first embodiment, this difference will be described below.

Also in the fourth embodiment, the determination at the time of mode switching is performed according to the flowchart of FIG. The difference from the first embodiment is that in S3, shooting with a short exposure time for shooting a bright part of a subject and shooting with a long exposure time for shooting a dark part of a subject are performed. ..

FIG. 8 is a diagram showing an example of an insertion state of the infrared cut filter into the opening of the diaphragm 106 and images obtained in each state in the fourth embodiment. FIG. 8 shows a case where short-time exposure and long-time exposure are alternately taken from the left. However, there is no limitation on the exposure time and the order of shooting, and the order does not have to be the same. In addition, FIG. 8 illustrates a case where the upper half of the subject is a bright part where the illuminance (or brightness) is high and the lower half is a dark part where the illuminance (or illuminance) is low. Therefore, in the short-second frame, the upper half of the captured image has a proper exposure, and the lower half has a black shadow. On the other hand, in the case of a long-time frame, the lower half of the captured image has the proper exposure, and the upper half has overexposure.

In the present embodiment, in S3, as shown in FIG. 8, the infrared cut filter 107 is inserted stepwise only in the short second frame. By this control, the infrared light is cut in the short-second frame, so that the brightness is gradually reduced. Using the image obtained in this short-time frame, the brightness of the subject when the infrared cut filter 107 is completely inserted can be easily predicted by the above-described formula (1).

On the other hand, as shown in FIG. 8, an image taken while controlling the infrared cut filter 107 and the exposure time is stored in a memory (not shown). Then, the control unit 103 extracts the range within the proper exposure for each frame from the two images of the short-time frame and the long-time frame stored in the memory, and combines them to form an image with a wide dynamic range. To generate. The dark portion in the short-second frame is the range in which the proper exposure is made in the long-second frame, and therefore has no effect on the composite image. The decrease in signal intensity in the bright part of the short-second frame appears as a change in brightness in the upper half of the composite image. As described above, in the first embodiment, the signal intensity of the entire captured image changes as shown in FIG. 2, whereas in the fourth embodiment, the range in which the signal intensity changes occurs in a part of the captured image. It becomes possible to suppress.

As described above, according to the fourth embodiment, it is possible to switch to the day mode while suppressing changes in the signal intensity of the captured image in a part of the captured image and preventing hunting.

In addition, in the fourth embodiment, the case where an image is captured with two types of exposure times has been described. However, even if images are captured with three types of different exposure times and combined using more frames. good. In that case, a larger range can be accommodated in the proper exposure.

Furthermore, by simultaneously controlling the exposure time of the short-second frame by combining the method of the second embodiment, it is possible to suppress the signal strength reduction in the bright part of the short-second frame. Further, in the fourth embodiment, the case where the infrared cut filter is inserted stepwise in the short second frame has been described, but the same control may be performed in the long second frame. When the infrared cut filter is gradually inserted in the long-second frame, the signal intensity changes only in the proper exposure range of the long-second frame, that is, in the lower half of the composite image.

<Fifth Embodiment>
Next, a fifth embodiment of the present invention will be described. Note that the configuration of the imaging device according to the fifth embodiment is similar to that of the imaging device 100 according to the first embodiment described with reference to FIG. However, the optical filter 107 in the fifth embodiment is a visible cut filter, and in the following description, the optical filter 107 is referred to as a visible cut filter 507 in order to distinguish it from the first to fourth embodiments.

The image pickup apparatus 100 has a halation suppression mode (third photographing mode) in which the visible cut filter 507 is inserted in at least a part of the optical path to obtain a monochrome image. It also has a night mode (second shooting mode) in which the visible cut filter 507 is completely removed from the optical path to obtain a monochrome image, and shooting is performed by switching between the night mode and the halation suppression mode.

FIG. 9 shows a diagram of the imaging optical system 101 viewed from the incident light side (−Z side), a captured image at that time, and the relationship between the insertion amount of the visible cut filter 507 and the brightness of the subject. It should be noted that only the aperture 106 and the visible cut filter 507 are shown and the others are omitted for the sake of clarity. In addition, the ambient light in the shooting range is light having both visible light and infrared light components, such as sunlight and incandescent lamps.

In FIG. 9, 510 is a vehicle, 511 is a license plate, 512 is an LED type headlight whose main component is visible light, and headlight 512 is on. Reference numeral 513 represents halation caused by a lit headlight. In FIG. 9, states (a) to (e) show how the visible cut filter 507 is gradually inserted into the optical path by the control unit 103. From the state (a) to the state (e), the range in which the opening of the diaphragm 106 and the visible cut filter 507 overlap with each other expands. The range where the aperture of the diaphragm 106 and the visible cut filter 507 overlap is an area where visible light is blocked and only infrared light is transmitted. The aperture of the diaphragm 106 that does not overlap the visible cut filter 507 is an area through which visible light and infrared light pass. From the state (a) to the state (e), the range through which only infrared light passes gradually increases, so the illuminance ratio of the subject due to the visible light component gradually decreases, and halation is suppressed. However, as a result, the entire captured image gradually darkens.

As can be seen from FIG. 9, if the area (area A) of the aperture of the diaphragm 106 and the area (area B) of the visible cut filter 507 that overlaps the aperture in each state are known, then using equation (1), The luminance for each insertion amount of the visible cut filter 507 can be predicted. In Expression (1), in the fifth embodiment, the brightness a and the brightness b indicate the brightness of the subject in the state (a) and the state (b) of FIG. 9, respectively. Further, as the brightness a and the brightness b, the brightness in the state (c) and the state (d) of FIG. 9 may be used, and the visible cut filter 507 is inserted by an arbitrary amount other than the state shown in FIG. State brightness may be used. In addition, the brightness of three or more points may be used for the brightness prediction. When using the brightness of three or more points, the brightness value of the subject may be predicted by the least square method or the like.

By the method described above, halation is suppressed, and it is possible to predict the insertion amount of the visible cut filter that is necessary for allowing the target imaging target to be imaged with high visibility. Here, for example, in the state (c) of FIG. 9, since halation is suppressed to a degree sufficient for recognizing the license plate 511, in this state, the visible cut filter may be stopped and imaging may be performed. In this manner, by performing the image capturing in the state (c), it becomes possible to recognize the license plate 511 even when the vehicle turns off the headlight.

In the state (c) of FIG. 9, the aperture of the diaphragm 106 still has a region for transmitting both visible light and infrared light, so that the brightness of the image decreases when the headlight is turned off. By detecting this decrease in brightness, it can be recognized that the headlight has been turned off, and it can be determined that the visible cut filter 507 is no longer needed. Therefore, it becomes possible to quickly remove the visible cut filter 507 and perform shooting with a high SNR in the night mode.

It should be noted that although the description has been given here using the state (c) of FIG. 9, it is clear that the same detection can be performed in any state other than the state (e). Further, in the above description, the case where the brightness of the subject is used has been described, but the magnitude of halation may be used instead of the brightness. In order to recognize the magnitude of halation, the control unit 103 may recognize a circular shape having a brightness equal to or higher than a certain threshold by means such as image recognition, and use information such as the diameter or radius of the circular shape.

FIG. 10 is a flowchart showing the flow when switching from the night mode to the halation suppression mode in the fifth embodiment. First, in S11, shooting in the night mode is started, and in S12, it is determined whether or not the brightness of the subject is equal to or higher than a predetermined fourth threshold value, that is, the presence or absence of halation. When it is determined that the brightness of the subject is equal to or higher than the fourth threshold value, the process proceeds to S13, and stepwise insertion of the visible cut filter 507 is started (halation suppression mode). Here, the brightness of the subject is acquired while gradually inserting the visible cut filter 507 using a plurality of frames.

Next, in S14, when the brightness of the subject becomes less than the predetermined fifth threshold value, the process proceeds to S15, the insertion of the visible cut filter 507 is stopped, and the imaging in the halation suppression mode is continued. If the fifth threshold value is smaller than the fourth threshold value, halation is sufficiently suppressed, and the luminance is set so that the object to be photographed (here, the license plate 511) can be photographed in a highly recognizable state. Good. Here, even if the insertion amount of the visible cut filter necessary for the brightness of the subject to be less than the fifth threshold value is predicted by using Expression (1) and the insertion speed of the visible cut filter 507 is increased to the predicted position. Good. The higher the insertion speed of the visible cut filter 507, the more the frames with low recognizability due to halation can be reduced. Also, in order to facilitate the prediction of the luminance, the insertion speed is set so that the insertion amount of the visible cut filter 507 per frame (that is, the area change amount of the visible cut filter 507 occupying the opening of the diaphragm 106) becomes constant. May be adjusted for each frame. Further, when the aperture 106 changes due to AE, it is possible to prevent the luminance change due to the aperture change for each frame and the luminance change due to the insertion of the visible cut filter 507 from being mixed. Specifically, similarly to the third embodiment, the insertion amount of the visible cut filter 507 for each frame is adjusted in the direction in which the change of the visible light component due to the change of the aperture 106 for each frame is canceled. As a result, the luminance change for each frame becomes linear, and thus the prediction using the equation (1) can be easily performed.

Next, in S16, it is detected whether or not there is a change in the brightness of the subject in a state where the visible cut filter 507 is fixed and the image is taken, and the position of the visible cut filter 507 remains fixed while there is no change. To continue shooting. When the brightness changes and becomes high, the process returns to S12, and when the brightness of the subject exceeds the fourth threshold, the process moves to S13 and the visible cut filter 507 is further inserted. On the contrary, if the luminance changes to low in S16, the process proceeds to S17 and is compared with a predetermined sixth threshold value. Here, the sixth threshold value may be set to a value of luminance when halation does not occur when the visible cut filter is completely removed. When it is less than the sixth threshold value in S17, it is determined that the headlight 512 is turned off, the process proceeds to S18, and the visible cut filter 507 is completely removed.

After that, the process returns to S11 to return to shooting in the night mode, and shooting with high SNR is possible. On the other hand, in S17, when it is the sixth threshold value or more, the process proceeds to S19, and the visible cut filter 507 is removed stepwise. Then, in S20, the seventh threshold value is compared with the current brightness. Here, a value equal to or larger than the sixth threshold and smaller than the fifth threshold is set as the seventh threshold. The stepwise removal of the visible cut filter 507 is continued until it becomes equal to or higher than the seventh threshold value in S20. When it becomes equal to or more than the seventh threshold value, the process proceeds to S15, and the visible cut filter 507 is fixed at that position.

As described above, according to the fifth embodiment, while the halation of the headlight is suppressed and the object to be photographed is photographed, even when the headlight is turned off, the turning-off is detected and the photographing in the high SNR night mode is performed. It becomes possible to switch to.

<Sixth Embodiment>
Next, a sixth embodiment of the present invention will be described. The configuration of the image capturing apparatus according to the sixth embodiment is the same as that of the image capturing apparatus 100 according to the first embodiment described with reference to FIG. However, the optical filter 107 in the sixth embodiment is a filter in which a visible cut filter and an infrared cut filter are combined, and in the following description, the optical filter 107 will be referred to in order to distinguish it from the first to fifth embodiments. It is referred to as an optical filter 607.

The configuration of the optical filter 607 in the sixth embodiment will be described with reference to FIG. The optical filter 607 has a configuration in which two different filters, a visible cut filter portion 610 and an infrared cut filter portion 611, are connected together. Then, the filter driving unit 104 drives the optical filter 607 with respect to the opening of the diaphragm 106 in the arrow X direction in the drawing. The image pickup apparatus 100 according to the sixth embodiment has a halation suppression mode (third photographing mode) in which the visible cut filter portion 610 is inserted into at least a part of the optical path to obtain a monochrome image. It also has a day mode (first shooting mode) in which the visible cut filter portion 610 is completely removed from the optical path and the infrared cut filter portion 611 is completely inserted in the optical path to obtain a color image. Switching between suppression mode and shooting.

FIG. 12 shows a view of the imaging optical system 101 from the incident light side (−Z side), a captured image at that time, and the relationship between the position of the optical filter 607 and the brightness of the subject. It should be noted that only the diaphragm 106 and the optical filter 607 are shown and the others are omitted for the sake of clarity. In addition, the ambient light in the shooting range is light having both visible light and infrared light components, such as sunlight and incandescent lamps.

In FIG. 12, 612 represents a vehicle as an object, 613 represents a license plate, 614 represents an LED type headlight, and the headlight 614 is lit. 615 represents the halation caused by the lit headlight 614.

In FIG. 12, states (a) to (e) show how the visible cut filter portion 610 of the optical filter 607 is gradually inserted into the optical path by the control unit 103. From the state (a) to the state (e), the range in which the opening of the diaphragm 106 and the visible cut filter portion 610 overlap with each other expands. The range where the aperture of the diaphragm 106 and the visible cut filter portion 610 overlap is an area where visible light is blocked and only infrared light is transmitted.

The aperture of the diaphragm 106 that does not overlap the visible cut filter portion 610 is an area where the infrared cut filter portion 611 overlaps and the infrared light is blocked and only visible light is transmitted. Since the range through which only infrared light passes gradually increases from the state (a) to the state (e), the illuminance ratio of the subject due to the visible light component also gradually decreases, and halation is suppressed. However, as a result, the entire captured image gradually becomes darker. Here, the state (a) corresponds to the day mode, and color photographing is performed. The state (b) to the state (e) correspond to the halation suppression mode, and monochrome photography is performed. As can be seen from FIG. 12, if the area (area A) of the aperture of the diaphragm 106 in each state and the area (area B) of the visible cut filter portion 610 overlapping the aperture of the diaphragm 106 are known, the first and second areas can be obtained. The brightness in the state (e) for each insertion amount of the visible cut filter portion 610 can be predicted by using the equation (1) described in the fifth embodiment.

Next, a method of detecting that the headlight 614 is turned off in the halation suppression mode will be described. In the state (c) of FIG. 12, since halation is suppressed to a sufficient degree for recognizing the license plate 613, it is possible to stop the visible cut filter portion 610 and perform photographing in this state. By photographing in this state (c), the license plate can be recognized even when the vehicle turns off the headlight 614. In the state (c), the range in which the infrared cut filter portion 611 overlaps the aperture opening 606, that is, the visible light transmitting region remains, so that the brightness of the image decreases when the headlight 614 is turned off. .. By detecting this decrease in brightness, it can be recognized that the headlight 614 has been turned off, and it can be determined that the visible cut filter portion 610 is no longer needed. Therefore, it is possible to quickly remove the visible cut filter portion 610, insert the infrared cut filter portion 611, and perform color photographing in the day mode.

Although the description has been given here using the state (c) of FIG. 12, it is clear that the same detection can be performed in any state other than the state (e). Further, in the above description, the case where the brightness of the subject is used has been described, but the magnitude of halation may be used instead of the brightness. In order to recognize the magnitude of halation, the control unit 103 may perform image recognition, recognize a circular shape having a brightness equal to or higher than a certain threshold, and use information such as a diameter or a radius thereof.

Next, how to use the method of predicting the luminance of the subject for each insertion amount of the visible cut filter portion 610 for determining the switching between the halation suppression mode and the day mode will be described.

The flow chart for switching from the day mode to the halation suppression mode is the same as the flow chart of FIG. 10 in the fifth embodiment, and therefore its explanation is omitted. There are two differences from the fifth embodiment, and one is that the shooting mode in S11 is color shooting in the day mode. The other is a configuration in which the visible light cut filter portion 610 and the infrared light cut filter portion 611 are connected in a row, and therefore, the infrared light cut filter portion 611 is removed by the amount corresponding to the insertion of the visible light cut filter portion 610. It is to be a normal operation.

As described above, according to the sixth embodiment, while the halation of the headlight is suppressed and the object to be photographed is photographed, even when the headlight is turned off, the turning off can be detected, and the color photography can be switched to the day mode. Will be possible.

<Seventh Embodiment>
Next, a seventh embodiment of the present invention will be described. FIG. 13 shows a schematic configuration of the image pickup apparatus 700 according to the seventh embodiment. It differs from the image pickup apparatus 100 in the first embodiment in that it has a polarization filter 707 as an optical filter of the imaging optical system 701 and further has an infrared cut filter 709 fixed in the optical path. The other configuration is similar to that shown in FIG. 1, and therefore the same reference numerals are given and the description thereof is omitted here. The image pickup apparatus 700 includes a reflection suppression mode (fourth shooting mode) in which a polarizing filter 707 is inserted into the optical path for shooting, and a day mode (first shooting mode) in which the polarizing filter 707 is completely removed from the optical path for shooting. Have. Then, shooting is performed by switching between the reflection suppression mode and the day mode.

FIG. 14 shows a view of the imaging optical system 701 viewed from the incident light side (−Z side), and the relationship between the captured image and the brightness of the subject at that time. Only the diaphragm 106, the polarization filter 707, and the infrared cut filter 709 are shown and the others are omitted for the sake of clarity. The ambient light in the shooting range is light having a non-polarized component such as sunlight and incandescent lamps.

In FIG. 14, reference numeral 713 represents a car on which a driver 712, which is a subject, is riding. In FIG. 14, states (a) to (e) show that the polarization filter 707 is gradually inserted into the optical path by the control unit 103, and as the state (a) progresses to the state (e), The range where the aperture of the diaphragm 106 and the polarization filter 707 overlap with each other expands.

FIG. 14 illustrates a case where the relationship between the imaging device 700, the subject, and the light source is appropriate, and the reflection can be suppressed by the polarization filter 707. In the state (a), the driver 712 is not visible due to the reflection of the windshield, but in the state (e) where the polarization filter 707 is completely inserted, the reflection is suppressed and the driver 712 is visible. The range where the aperture of the diaphragm 106 and the polarization filter 707 overlap is an area where the light of the S-polarized component reflected by the windshield (that is, the reflection) is blocked, and the infrared cut filter 709 also blocks the infrared light. This is the area that will be used. The aperture of the diaphragm 106 that does not overlap with the polarization filter 707 is an area where all polarized light is transmitted, infrared light is blocked by the infrared cut filter 709, and only visible light is transmitted.

From the state (a) to the state (e), the range in which the S-polarized light component reflected by the windshield is blocked is gradually increased, so that the glare is gradually reduced. Further, also in the range other than the windshield, of the polarized light included in the natural light, the component parallel to the polarizer of the polarization filter 707 is gradually blocked, and as a result, the entire captured image is gradually darkened.

As can be seen from FIG. 14, if the area (area A) of the aperture of the diaphragm 106 and the area (area B) of the polarization filter 707 that overlaps the aperture in each state are known, then using equation (1), The brightness when the polarization filter 707 is completely inserted can be predicted. In the formula (1), in the seventh embodiment, the brightness a and the brightness b indicate the brightness of the subject in the state (a) and the state (b) of FIG. 14, respectively. As the brightness a and the brightness b, the brightness in the state (c) or the state (d) of FIG. 14 may be used, and in addition to the state shown in FIG. 14, a state in which the polarization filter 707 is inserted by an arbitrary amount. The brightness of may be used. In addition, the brightness of three or more points may be used for the brightness prediction. When using the brightness of three or more points, the brightness value of the subject may be predicted by the least square method or the like.

Next, a case where the relationship with the light source is changed from the state of FIG. 14 and the reflection is changed so as not to be suppressed by the polarization filter 707 will be described with reference to FIG. 15. In FIG. 15, the polarization filter 707 is gradually inserted into the optical path from the state (a) to the state (e). However, reflection on the windshield includes P-polarized light in addition to S-polarized light, so reflection is suppressed. Not done and the driver is not visible in state (e). On the other hand, the change (decrease) in luminance from the state (a) to the state (e) is gentler than that in FIG. Further, in FIG. 15, the luminance of the entire vehicle is targeted, but if the luminance of only the windshield, which is the region of interest, is handled, the difference in the gradient of the luminance change between when the glare can be suppressed and when the glare can not be suppressed becomes larger. The accuracy of is improved. In this case, the control unit 103 may recognize the windshield by an image recognition method such as pattern matching and use only the brightness inside the windshield for estimation.

Further, the estimation accuracy is improved by using a contrast value as an evaluation value in addition to the image brightness. When the reflection can be suppressed as in FIG. 14, the contrast value increases from the states (a) to (e), but the contrast value does not change in the case of FIG. By using the difference in the amount of change in the contrast value in addition to the difference in the gradient in the change in brightness, it is possible to improve the accuracy of estimating whether or not the glare can be suppressed. From the above, it is possible to distinguish between the case where the reflection can be suppressed by the polarization filter and the case where the reflection cannot be suppressed because the estimated brightness value of the subject when the polarization filter is completely inserted is different. Specifically, an arbitrary luminance value between the estimated luminance values in each of the states (e) of FIGS. 14 and 15 may be used as a threshold, and the threshold for switching between the glare suppression mode and the day mode may be set.

FIG. 16 is a flowchart showing the flow when switching between the day mode and the reflection suppression mode in the seventh embodiment. First, in S31, shooting in the day mode is started, and in S32, it is determined whether or not the brightness of the subject is equal to or higher than a predetermined eighth threshold value, that is, whether or not there is a glare. When it is determined that the brightness of the subject is equal to or higher than the eighth threshold value, the process proceeds to S33, and the stepwise insertion of the polarization filter 707 is started (reflection suppression mode). Here, the brightness of the subject is acquired while gradually inserting the polarization filter 707 using a plurality of frames, and the brightness value of the subject when the polarization filter 707 is completely inserted in the optical path is estimated.

In S34, when it is determined that the estimated brightness is less than the ninth threshold value, the process proceeds to S36, the polarization filter 707 is completely inserted, and the glare suppression mode is entered. Here, the insertion speed of the polarization filter 707 may be increased in the frame after the luminance of the subject is determined to be less than the ninth threshold value in S34. The faster the insertion speed of the polarization filter 707, the more the frames with low recognition due to glare can be reduced. Further, in order to facilitate the prediction of the brightness, the insertion speed is set to the frame so that the insertion amount of the polarization filter 707 per frame (that is, the area change amount of the polarization filter 707 occupying the opening of the diaphragm 106) becomes constant. You may adjust every. Further, when the aperture 106 changes due to AE, it may be possible to prevent the luminance change due to the aperture change for each frame from being mixed with the luminance change due to the insertion of the polarization filter 707. Specifically, as in the third embodiment, the insertion amount of the polarization filter 707 for each frame is adjusted in the direction in which the change in brightness due to the change in the diaphragm 106 for each frame is canceled. As a result, the luminance change for each frame becomes linear, and thus the prediction using the equation (1) can be easily performed.

On the other hand, in S34, when the brightness value of the subject when the polarization filter 707 is completely inserted is equal to or higher than the ninth threshold value, it is determined that the reflection cannot be suppressed, the process proceeds to S35, and the polarization filter 707 is moved from the optical path. Evacuate and return to day mode shooting. Next, in S37, it is determined whether or not the current brightness value of the subject has changed to the tenth threshold value or more. If the brightness of the subject changes to the tenth threshold or more, it is determined that the positional relationship between the light source and the subject has changed, and it is possible that the glare can be suppressed. Start the automatic insertion. If the tenth threshold value is set to be small, the polarization filter can be inserted more quickly if the reflection is suppressed to a value that can be suppressed. On the contrary, when a larger value is set to the tenth threshold value, resistance to disturbance and the like is increased, unnecessary insertion of the polarization filter 707 is reduced, and image quality is improved.

If the brightness of the subject changes to less than the eleventh threshold value during shooting in the reflection suppressing mode in S36 (YES in S38), the positional relationship between the light source and the subject may change, and the reflection may have disappeared. The judgment is made, and the process proceeds to S39. Here, the eleventh threshold value is between the luminance of the subject when the image is taken with the polarization filter 707 inserted and the subject luminance when the image is taken with the polarization filter 707 removed in a situation where no reflection occurs. Set to the value of.

In step S39, the polarization filter 707 is gradually removed, and the brightness of the subject is acquired for each frame. In S40, when the estimated brightness value of the subject when the polarization filter 707 is completely removed is less than the eighth threshold value, the process returns to S31, and the polarization filter 707 is completely removed to perform day mode shooting. On the other hand, in S40, when the estimated brightness value is equal to or higher than the eighth threshold value, the process returns to S36, and the polarization filter 707 is completely inserted to return to the image capturing in the reflection suppressing mode.

As described above, according to the seventh embodiment, it is possible to determine whether or not the reflection that can be suppressed by the polarization filter has occurred, from the images of a plurality of frames in which the polarization filter is gradually inserted and removed. It is possible to suppress the deterioration of. In addition, shooting is performed in the reflection suppression mode only when there is reflection that can be suppressed by the polarization filter, and when there is reflection that cannot be suppressed, switch to shooting in day mode to improve image quality. It is possible to

Considering the case where the infrared cut filter 709 of the present embodiment is removed, it is obvious that the above-described method can be applied to switching between the reflection suppression mode and the night mode.

<Eighth Embodiment>
Next, an eighth embodiment of the present invention will be described. In the eighth embodiment, an image pickup apparatus capable of switching four modes of a day mode, a night mode, a halation suppression mode, and a reflection suppression mode will be described.

The configuration of the image pickup apparatus according to the eighth embodiment is similar to that of the image pickup apparatus 100 according to the first embodiment described with reference to FIG. However, the optical filter 107 in the eighth embodiment is composed of a plurality of filter portions having different transmission characteristics, and in the following description, the optical filter 107 will be referred to in order to distinguish it from the first to sixth embodiments. It is referred to as an optical filter 807.

Next, the configuration of the optical filter 807 in the eighth embodiment will be described with reference to FIG. The optical filter 807 has a configuration in which a visible cut filter portion 810, a dummy glass portion 811, an infrared cut filter portion 812, and a polarization filter portion 813 are connected together. The dummy glass portion 811 is for adjusting the optical path difference when there is no filter. The optical filter 807 is inserted and removed in the X direction in the figure by the filter driving unit 104. When the optical filter 807 is driven in the −X direction from the state where the visible cut filter portion 810 completely overlaps the opening of the diaphragm 106, the halation suppression mode, the night mode, the day mode, and the reflection suppression mode. The shooting mode will change in the order of.

For switching between the visible cut filter portion 810 and the dummy glass portion 811, that is, for switching between the halation suppression mode and the night mode, the method described in the fifth embodiment can be used. This makes it possible to detect that the headlight is off and switch to the night mode while suppressing halation.

Moreover, the method described in the first embodiment can be applied to the switching between the dummy glass portion 811 and the infrared cut filter portion 812, that is, the switching between the night mode and the day mode. This makes it possible to switch to the day mode even from low illuminance without hunting.

Furthermore, the switching of the infrared cut filter portion 812 and the polarization filter portion 813, that is, the switching between the day mode and the reflection suppression mode is controlled by using the method of the seventh embodiment, thereby improving the image quality. It becomes possible.

In the filter configuration of FIG. 17, the transition from the halation suppression mode to the day mode, the transition from the night mode to the reflection suppression mode, and the transition between the halation suppression mode and the reflection suppression mode cannot be performed. When it is desired to switch between these shooting modes, the order of the filters from the visible cut filter portion 810 to the polarization filter portion 813 may be changed.

As described above, according to the eighth embodiment, the transition from the night mode to the day mode can be performed without hunting even at low illuminance. Further, it is possible to shift from the halation suppression mode to the night mode in accordance with the turning off of the headlight. Further, only when the polarizing filter is effective, it is possible to appropriately shift to the reflection suppressing mode. Therefore, it is possible to switch to the optimum shooting mode according to various situations of the subject.

<Ninth Embodiment>
Next, a ninth embodiment of the present invention will be described. In the ninth embodiment, another example of an image pickup apparatus capable of switching four modes of a day mode, a night mode, a halation suppression mode, and a reflection suppression mode will be described.

The configuration of the image capturing apparatus 100 according to the ninth embodiment is similar to that of the image capturing apparatus 100 according to the first embodiment described with reference to FIG. However, the optical filter 107 in the ninth embodiment is composed of a plurality of filters having different transmission characteristics, and in the following description, in order to distinguish it from the first to sixth embodiments and the seventh embodiment. The optical filter 107 is referred to as an optical filter 907.

Next, the configuration of the optical filter 907 in the ninth embodiment will be described with reference to FIG. The optical filter 907 in the ninth embodiment is a turret filter and has a visible cut filter portion 910, a dummy glass portion 911, an infrared cut filter portion 912, and a polarization filter portion 913. The optical filter 907 can be rotated in the direction of the arrow in the figure by the filter driving unit 104 to switch between four modes of day mode, night mode, halation suppression mode, and reflection suppression mode. Compared to the eighth embodiment, it is possible to switch between the halation suppression mode and the reflection suppression mode.

The method of the fifth embodiment can be applied to the switching of the visible cut filter portion 910 and the dummy glass portion 911, that is, the switching between the halation suppression mode and the night mode. This makes it possible to detect that the headlight is off and switch to the night mode while suppressing halation.

Further, the method of the first embodiment can be applied to the switching between the dummy glass portion 911 and the infrared cut filter portion 912, that is, the switching between the night mode and the day mode. This makes it possible to switch to the day mode even from low illuminance without hunting.

Further, the switching of the infrared cut filter portion 912 and the polarization filter portion 913, that is, the switching between the day mode and the reflection suppression mode is controlled by using the method of the seventh embodiment, thereby improving the image quality. It becomes possible.

Further, in switching between the polarization filter portion 913 and the visible cut filter portion 910, that is, in switching between the reflection suppression mode and the halation suppression mode, the same operation can be performed by replacing the infrared cut filter in the seventh embodiment with the visible cut filter. The method can be applied.

As described above, according to the ninth embodiment, in addition to the effect similar to that of the eighth embodiment, the number of switchable shooting modes can be increased by one by using the same type of filter as that of the eighth embodiment. it can.

<Tenth Embodiment>
Next, a tenth embodiment of the present invention will be described. In the tenth embodiment, still another example of an image pickup apparatus capable of switching four modes of a day mode, a night mode, a halation suppression mode, and a reflection suppression mode will be described.

FIG. 19 shows a schematic configuration of the image pickup apparatus 1000 according to the tenth embodiment. The image pickup apparatus 1000 shown in FIG. 19 is different from the image pickup apparatus 100 shown in FIG. 1 in that optical filters 1009 and 1010 are added to the imaging optical system 1001, and a filter driving unit for driving the optical filters 1009 and 1010. 1011 and 1012 are added respectively. Since the other configurations are the same as those in FIG.

The filter driving units 104, 1011 and 1012 are controlled by the control unit 103, and individually drive the optical filters 107, 1009 and 1010 in the arrow direction (X direction) in the figure. Here, the optical filter 107 is an infrared cut filter, the optical filter 1009 is a visible cut filter, and the optical filter 1010 is a polarization filter. Hereinafter, they are also referred to as an infrared cut filter 107, a visible cut filter 1009, and a polarization filter 1010, respectively. These filters are installed by shifting their positions in the Z direction in the figure. Although it is installed on the lens 108 side of the diaphragm 106 in FIG. 19, it may be installed on the lens 105 side of the diaphragm 106. Further, any one of the optical filters 107, 1009, and 1010 may be installed on the opposite side with the diaphragm 106 interposed therebetween. Further, each filter may further include a dummy glass for adjusting the optical path difference. In any case, it is desirable to place the infrared cut filter 107, the visible cut filter 1009, and the polarization filter 1010 near the position where the diameter of the light beam propagating between the lenses 105 and 108 is the smallest.

Next, the positional relationship between the optical filters 107, 1009, 1010 and the diaphragm 106 for each shooting mode will be described with reference to FIG. In FIG. 20, the positional relationships of the optical filters 107, 1009, and 1010 corresponding to the four shooting modes are shown from state (a) to state (d). In the state (a), all the optical filters 107, 1009, and 1010 are retracted from the optical path, and represent the night mode. The state (b) represents the day mode because the infrared cut filter 107 is inserted in the optical path. The state (c) shows the halation suppression mode in which the visible cut filter 1009 is inserted in the optical path. In the state (d), the infrared cut filter 107 and the polarization filter 1010 are inserted in the optical path, and the reflection is in the suppression mode (color photographing).

The method of the first embodiment can be applied to the switching between the state (a) and the state (b), that is, the switching between the night mode and the day mode. The method of the fifth embodiment can be applied to the switching between the state (a) and the state (c), that is, the switching between the night mode and the halation suppression mode. The method of the seventh embodiment can be applied to the switching between the state (b) and the state (d), that is, the switching between the day mode and the reflection suppression mode.

As described above, according to the tenth embodiment, the transition from the night mode to the day mode can be performed without hunting even at low illuminance. Further, it is possible to shift from the halation suppression mode to the day mode or the night mode in accordance with the turning off of the headlight. Therefore, it is possible to switch to the optimum shooting mode according to the situation of the subject. Further, only when the polarizing filter is effective, it is possible to appropriately shift to the reflection suppressing mode. Further, since the filters are arranged in parallel and are individually inserted/extracted, the evacuation space of the filters can be made smaller than that of the seventh embodiment, and as a result, the size of the image pickup apparatus can be made smaller.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

For example, in the above-described first to tenth embodiments, the control unit in the image pickup apparatus is configured to control the filter drive unit, but from an external control apparatus, for example, via a network, wired/wireless communication, or the like. Thus, the image pickup apparatus may be controlled.

<Other Embodiments>
The present invention also supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100, 700, 1000: image pickup device, 101, 701, 1001: image forming optical system, 102: image pickup element, 103: control unit, 104, 1011, 1012: filter drive unit 105, 108: lens, 106: diaphragm, 107 , 607, 707, 709, 807, 907, 1009, 1010: optical filter, 610, 810, 910: visible cut filter portion, 811, 911: dummy glass portion, 611, 812, 912: infrared cut filter portion, 813 , 913: Polarizing filter part

Claims (18)

Control for controlling an image pickup apparatus having an image pickup device for photoelectrically converting light incident through an image forming optical system including an optical filter to output an image signal, and a driving unit for changing the position of the optical filter. A device,
Based on the brightness of the image obtained by the image sensor while controlling the drive means so as to insert the optical filter stepwise into the opening of the imaging optical system while inserting the optical filter stepwise. A control device comprising: a control unit that determines a position of the optical filter and controls the drive unit so that the position of the optical filter becomes the determined position. The optical filter includes a first filter that cuts infrared light,
The control means starts stepwise insertion of the first filter into the opening when the brightness of the image is equal to or higher than a predetermined first threshold value, and opens the first filter through the opening. The first filter based on the brightness of the image obtained by the image pickup device while being gradually inserted into the part, and the ratio of the first filter to the area of the opening. Estimating the brightness of the image when completely inserted into the opening, when the estimated brightness value is greater than or equal to a predetermined second threshold value, the first filter is completely inserted into the opening, The control device according to claim 1, wherein when it is less than the second threshold value, the first filter is controlled so as to be completely removed from the opening. The optical filter includes a second filter that cuts visible light,
The control means starts stepwise insertion of the second filter into the opening when the brightness of the image is equal to or higher than a predetermined third threshold value, and opens the second filter by the opening. Based on the brightness of the image obtained by the image pickup device while being gradually inserted into the part, the brightness when the brightness changes to a value smaller than a predetermined fourth threshold value smaller than the third threshold value. The control device according to claim 1, wherein the position of the filter is determined as the position of the optical filter. The control unit controls the brightness of an image obtained by the image pickup device while inserting the second filter into the opening stepwise, and further, the ratio of the second filter to the area of the opening section. And estimating the position of the second filter when the brightness of the image changes to a value smaller than the fourth threshold value, and determining the estimated position as the position of the optical filter. The control device according to claim 3, wherein the control device is a control device. When the brightness of the image obtained by the image sensor changes to less than a predetermined fifth threshold value with the second filter inserted in the determined position, the control means changes the second The control device according to claim 3 or 4, wherein the filter is controlled so as to be completely removed from the opening. The optical filter includes a third filter which is a polarization filter,
The control means starts stepwise insertion of the third filter into the opening when the brightness of the image is equal to or higher than a predetermined sixth threshold value, and opens the third filter by the opening. The third filter based on the brightness of the image obtained by the image pickup device while being gradually inserted into the part and the ratio of the third filter to the area of the opening. Estimating the brightness of the image when completely inserted into the opening, when the estimated brightness value is less than a predetermined seventh threshold value, the third filter is completely inserted into the opening, The control device according to claim 1, wherein when the value is equal to or more than the seventh threshold value, the third filter is controlled so as to be completely removed from the opening. When the brightness of the image obtained by the image pickup device with the third filter completely inserted in the opening is less than a predetermined eighth threshold value, the control means sets the third threshold value. The stepwise removal of the filter from the opening is started, and with respect to the brightness of the image obtained by the image sensor and the area of the opening while removing the third filter stepwise from the opening. Based on the ratio occupied by the third filter, the brightness of the image when the third filter is completely removed from the opening is estimated, and the estimated brightness value is set to a predetermined ninth value. When the value is less than a threshold value, the third filter is completely removed from the opening, and when the value is equal to or higher than the ninth threshold value, the third filter is controlled to be completely inserted into the opening. The control device according to claim 6, wherein: The control unit further controls the exposure of the imaging device so that the brightness of an image obtained by inserting the optical filter into the opening stepwise is kept constant. 7. The control device according to any one of 7. The control unit controls the position of the optical filter so as to keep the brightness of the image constant when the area of the opening changes due to the control of the exposure. Control device. The optical filter includes a first filter that cuts infrared light,
When the brightness of the image is greater than or equal to a predetermined first threshold value, the control unit captures an image in which the first filter is inserted stepwise into the opening, and sets the first filter to the The brightness of the image obtained by the image pickup device is controlled while alternately controlling the photographing of the image completely removed from the opening, and the first filter is gradually inserted into the opening, and further, Based on the ratio of the first filter to the area of the opening, and the brightness of the image when the first filter is completely inserted in the opening, the estimated brightness value is The first filter is completely inserted into the opening when it is equal to or more than a predetermined second threshold, and the first filter is completely inserted from the opening when it is less than the second threshold. Control to remove
An image captured by inserting the first filter into the opening stepwise, and an image captured before or after the first filter is completely removed from the opening. The control device according to claim 1. The optical filter further includes a first filter that cuts infrared light,
At least one of said 1st filter and said 2nd filter is inserted in the said opening part, The control apparatus in any one of Claim 3 thru|or 5 characterized by the above-mentioned. The optical filter has a configuration in which a first filter that cuts infrared light, a second filter that cuts visible light, and a third filter that is a polarization filter are arranged in a first direction. Then
The control device according to any one of claims 1 to 11, wherein the driving unit changes the position of the optical filter in the first direction. The optical filter is a turret filter having a first filter that cuts infrared light, a second filter that cuts visible light, and a third filter that is a polarization filter,
The control device according to any one of claims 1 to 11, wherein the driving unit changes a position by rotating the optical filter. The optical filter comprises a plurality of filters including a first filter that cuts infrared light, a second filter that cuts visible light, and a third filter that is a polarization filter. The control device according to any one of 1 to 11. An image sensor that photoelectrically converts light incident through an imaging optical system including an optical filter to output an image signal,
Drive means for changing the position of the optical filter,
An image pickup apparatus comprising: the control device according to claim 1. A method for controlling an image pickup apparatus, comprising: an image pickup element that photoelectrically converts light incident through an image forming optical system including an optical filter to output an image signal; and a driving unit that changes a position of the optical filter. There
Control means controls the driving means so as to insert the optical filter stepwise into the opening of the imaging optical system;
A step in which the control means determines the position of the optical filter based on the brightness of an image obtained by the image sensor while inserting the optical filter stepwise;
The control means controls the driving means so that the position of the optical filter becomes the determined position. A program for causing a computer to function as each unit of the control device according to any one of claims 1 to 14. A computer-readable storage medium storing the program according to claim 17.