JP6360816B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method.

  Conventionally, in a technical field such as a surveillance camera, it has been desired to provide a good image in any environment from high illuminance to low illuminance. Therefore, as a background art in this technical field, for example, there is JP 2011-233983 A (Patent Document 1).

  Patent Document 1 discloses an imaging apparatus that can obtain a good color image even under low illuminance such as at night, and can simultaneously obtain a clear contrast image as in the case of using an infrared illumination device. Have been described. Specifically, Patent Document 1 discloses a solid-state imaging device, an infrared LED, a light emission control unit, a color image signal by visible light, and an infrared in synchronization with a non-light emission period and a light emission period of the infrared LED. An image pickup apparatus including a signal processing unit that extracts an image signal by light is described. In Patent Document 1, the solid-state imaging device includes a pixel that receives green visible light and infrared light, a pixel that receives red visible light and infrared light, and blue visible light and infrared light. It describes that it has a unit array including pixels that receive light and pixels that receive infrared light. Furthermore, Patent Document 1 describes that a solid-state imaging device has an imaging region in which the unit array is two-dimensionally arrayed. If the technique described in Patent Document 1 is used, a good color image can be obtained even at night by using an imaging device capable of simultaneously receiving visible light and infrared light.

JP 2011-233983 A

  In the above-mentioned Patent Document 1, sufficient consideration is given to shooting in a low-light environment such as at night. However, in Patent Document 1, for example, photographing in an environment in which a light source such as sunlight, halogen light, or a sodium lamp is used, that is, an environment in which a light source containing a large amount of infrared light is used is sufficiently considered. Not done. Therefore, heretofore, in the present technical field, in an imaging apparatus capable of simultaneously receiving visible light and infrared light, an optical condition (imaging) at the time of shooting is determined according to the optical imaging environment of the subject (hereinafter referred to as the subject environment). There is a need for the development of technology that can automatically transition the (condition) to the optimum condition.

  The present invention has been made in order to meet the above-mentioned demands, and an object of the present invention is to provide an imaging apparatus capable of simultaneously receiving visible light and infrared light, with optical conditions at the time of photographing depending on the subject environment. It is to provide a technique capable of automatically shifting to an optimum condition.

  In order to solve the above-described problems of the prior art, in the present invention, an imaging unit capable of simultaneously receiving visible light and infrared light, a first optical filter, a visible light signal generation unit, an infrared light signal generation unit, An imaging device including the control unit and the optical filter control unit and an imaging method therefor are provided. In the present invention, the optical filter control unit passes the first optical filter that transmits only visible light on the first position on the optical path of the incident light or on the optical path based on the predetermined control signal output from the control unit. It arrange | positions in 2nd positions other than this position.

  According to the present invention configured as described above, in an imaging apparatus capable of simultaneously receiving visible light and infrared light, it is possible to automatically change the optical condition during photographing to an optimum condition according to the subject environment.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows the arrangement | sequence of the unit pixel in the imaging device which concerns on 1st Embodiment. It is a figure which shows the sensitivity characteristic of each unit pixel in the imaging device which concerns on 1st Embodiment. It is a figure which shows the sensitivity characteristic corresponding to each signal output from the visible light signal generation part and infrared light signal generation part of the imaging device which concerns on 1st Embodiment. It is a functional block diagram of the control part in the imaging device concerning a 1st embodiment. It is a figure for demonstrating the content of the light source determination process performed in the imaging device which concerns on 1st Embodiment. 6 is a flowchart illustrating a procedure of optical filter movement control processing (optical condition switching processing) in the imaging apparatus according to the first embodiment. It is a figure for demonstrating the effect of the imaging device which concerns on 1st Embodiment. It is a block diagram which shows the structure of the imaging device which concerns on the 2nd Embodiment of this invention. It is a figure which shows the sensitivity characteristic of each unit pixel in a general purpose image sensor. 6 is a functional block diagram of a control unit in an imaging apparatus according to Modification 1. FIG.

  Hereinafter, the contents of an imaging apparatus according to various embodiments of the present invention and an imaging technique using the imaging apparatus will be specifically described with reference to the drawings. Note that the imaging apparatus of the present invention can be applied to various uses, and can be applied to, for example, a chassis camera configured by attaching an imaging apparatus described later to a chassis member.

<First Embodiment>
[Configuration of imaging device]
FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the imaging device 1 includes an imaging unit 10, an optical filter 2, a visible light signal generation unit 3, an infrared light signal generation unit 4, an optical filter control unit 5, and an infrared irradiation unit 6. A visible light signal processing unit 7, an infrared light signal processing unit 8, a signal synthesis unit 9, and a control unit 20.

[Imaging section]
The imaging unit 10 includes a lens 11 and an imaging element 12. Although not illustrated, the imaging unit 10 includes a diaphragm mechanism and a shutter mechanism for adjusting the amount of light incident on the imaging element 12. The lens 11 forms an optical image by forming an optical signal incident from the subject (hereinafter referred to as a subject signal) on the light receiving surface of the image sensor 12.

  The imaging device 12 is configured by an imaging device (an imaging device capable of receiving visible light and infrared light simultaneously) having optical sensitivity to both visible light and infrared light. The imaging device 12 can be configured by a solid-state imaging device such as a complementary metal-oxide semiconductor (CMOS) type or a charge-coupled device (CCD) type, for example. The image sensor 12 converts the optical image (subject signal) formed by the lens 11 into an electrical signal, and outputs the converted electrical signal to the visible light signal generation unit 3 and the infrared light signal generation unit 4. .

  FIG. 2 is a schematic configuration diagram of the image sensor 12. A plurality of unit pixels are two-dimensionally arranged in the imaging region 12a (light receiving surface) of the imaging device 12.

  The plurality of unit pixels can receive the first unit pixel 31 capable of receiving red component light and infrared light contained in visible light, and the green component light and infrared light contained in visible light. 2nd unit pixel 32 is included. The plurality of unit pixels include a third unit pixel 33 that can receive blue component light and infrared light included in visible light, and a fourth unit pixel 34 that can receive only infrared light. It is. Although not shown, the first unit pixel 31 is provided with a filter that transmits red component light and infrared light in visible light, and the second unit pixel 32 has green component light in visible light. And a filter that transmits infrared light. The third unit pixel 33 is provided with a filter that transmits blue component light and infrared light in visible light, and the fourth unit pixel 34 is provided with a filter that transmits only infrared light. .

  In the imaging region 12a, a pixel group 30a (hereinafter referred to as a basic pixel group 30a) in which the first unit pixel 31 to the fourth unit pixel 34 are arranged adjacent to each other in a 2 × 2 column manner is two-dimensional. Arranged in a shape.

  Further, as another configuration of the basic pixel group, the configuration of the basic pixel group 30b illustrated in FIG. 2 may be used. The basic pixel group 30b includes a first unit pixel 35 capable of receiving red component light and infrared light in visible light, and a second unit pixel 35 capable of receiving green component light and infrared light in visible light. A unit pixel 36 is included. The basic pixel group 30b can receive the blue component light in the visible light signal and the third unit pixel 37 that can receive the infrared light, and the light in the entire visible light band and the infrared light. A fourth unit pixel 38 is included. The first unit pixel 35 to the fourth unit pixel 38 are arranged adjacent to each other in the form of 2 rows × 2 columns. Although not shown, the fourth unit pixel 38 is provided with a filter that transmits visible light (light in the entire visible light band) and infrared light.

  In addition, the positional relationship and / or arrangement mode of the first unit pixel to the fourth unit pixel in the basic pixel group is not limited to the example illustrated in FIG. 2, and may be appropriately set according to the type of the image sensor, for example. it can.

  3A, 3B, 3C, and 3D respectively show the third unit pixel 33, the second unit pixel 32, the first unit pixel 31, and the fourth unit pixel 34 that form the basic pixel group 30a (see FIG. 2). It is a figure which shows a sensitivity (photosensitive) characteristic. FIG. 3E is a diagram illustrating sensitivity characteristics of the fourth unit pixel 38 constituting the basic pixel group 30b (see FIG. 2). The sensitivity characteristic shown in each figure is a characteristic that represents a change in sensitivity of each unit pixel with respect to a change in wavelength of incident light. The wavelength bands indicated by “R”, “G”, “B”, “W”, and “IR” in each figure are respectively red component light, green component light, blue component light, visible light, and infrared. Indicates the wavelength band of light.

  The sensitivity characteristic of the third unit pixel 33 constituting the basic pixel group 30a is a characteristic having sensitivity in both wavelength bands of blue component light and infrared light, and the sensitivity characteristic of the second unit pixel 32 is green component light. And a characteristic having sensitivity in both wavelength bands of infrared light and infrared light. The sensitivity characteristic of the first unit pixel 31 constituting the basic pixel group 30a is a characteristic having sensitivity in both wavelength bands of red component light and infrared light, and the sensitivity characteristic of the fourth unit pixel 34 is infrared light. It has a characteristic having sensitivity only in the wavelength band. Further, the sensitivity characteristics of the fourth unit pixel 38 constituting the basic pixel group 30b are characteristics having sensitivity in both visible light and infrared light wavelength bands. Note that the sensitivity characteristics of the first unit pixel 35 to the third unit pixel 37 constituting the basic pixel group 30b are the same as the sensitivity characteristics shown in FIGS. 3C to 3A, respectively.

[Visible light signal generator and infrared light signal generator]
The visible light signal generation unit 3 extracts only a visible light component signal (hereinafter referred to as a visible light signal) included in the electrical signal input from the imaging unit 10. When the basic pixel group is the basic pixel group 30a described above (see FIG. 2), the visible light signal generation unit 3 outputs the red component signal, the green component signal, and the blue component signal included in the visible light signal. Extract each separately. When the basic pixel group is the basic pixel group 30b described above (see FIG. 2), the visible light signal generation unit 3 includes a red component signal, a green component signal, a blue component signal, and the like included in the visible light signal. Each signal in the entire visible light band is extracted separately. Then, the visible light signal generation unit 3 outputs the extracted various color component signals to the visible light signal processing unit 7 and the control unit 20.

  The infrared light signal generation unit 4 extracts only an infrared light component signal (hereinafter referred to as an infrared light signal) included in the electrical signal input from the imaging unit 10. Then, the infrared light signal generation unit 4 outputs the extracted infrared light signal to the infrared light signal processing unit 8 and the control unit 20.

When the basic pixel group is configured as the basic pixel group 30a (see FIG. 2), the visible light signal generation unit 3 uses the following formula to calculate a red component signal (R signal), a green component signal (G signal), and a blue color. Component signals (B signals) are extracted. In the infrared light signal generation unit 4, the signal obtained by the fourth unit pixel 34 is extracted as an infrared light signal (IR signal). Note that “R + IR”, “G + IR”, “B + IR”, and “IR” in the following equations are obtained from the first unit pixel 31, the second unit pixel 32, the third unit pixel 33, and the fourth unit pixel 34, respectively. Represents a signal.
R signal = (R + IR) −IR
G signal = (G + IR)-IR
B signal = (B + IR)-IR
IR signal = signal obtained by the fourth unit pixel 34

On the other hand, when the configuration of the basic pixel group is the basic pixel group 30b described above (see FIG. 2), the visible light signal generation unit 3 uses the following formula to calculate the R signal, the G signal, the B signal, and the signal of the entire visible light band. Each quantity (W signal) is extracted. Further, the infrared light signal generation unit 4 extracts an IR signal by the following equation. Note that “R + IR”, “G + IR”, “B + IR”, and “W + IR” in the following formulas are obtained from the first unit pixel 35, the second unit pixel 36, the third unit pixel 37, and the fourth unit pixel 38, respectively. Represents a signal. Coefficients α1 to α4 in the following formulas are coefficients for appropriately canceling unnecessary signals (signals other than the color component signal to be extracted). For example, α1 is a coefficient for appropriately canceling signals other than the R signal.
R signal = α1 × (W + IR) − {(G + IR) + (B + IR)}
G signal = α2 × (W + IR) − {(R + IR) + (B + IR)}
B signal = α3 × (W + IR) − {(R + IR) + (G + IR)}
W signal = (W + IR)-IR
IR signal = {(R + IR) + (G + IR) + (B + IR)} − α4 × (W + IR)

  4A to 4E are diagrams showing sensitivity characteristics corresponding to the B signal, the G signal, the R signal, the IR signal, and the W signal obtained by the extraction processing, respectively. The sensitivity characteristic corresponding to the B signal obtained by the extraction process of the visible light signal generation unit 3 is a characteristic having sensitivity only in the wavelength band of blue light, and the sensitivity characteristic corresponding to the G signal is the wavelength band of green light. It becomes a characteristic having sensitivity only at. The sensitivity characteristic corresponding to the R signal obtained by the extraction process of the visible light signal generation unit 3 is a characteristic having sensitivity only in the wavelength band of red light. The sensitivity characteristic corresponding to the IR signal obtained by the extraction process of the infrared light signal generation unit 4 is a characteristic having sensitivity only in the wavelength band of infrared light. In addition, the sensitivity characteristic corresponding to the W signal obtained by the extraction process of the visible light signal generation unit 3 is such that a predetermined sensitivity can be obtained over the entire wavelength band of visible light.

  The above-described extraction processing of various color component signals and infrared light signals is executed for each frame of image processing. Further, it is used in signal ratio calculation processing, light source determination processing, optical filter 2 insertion necessity determination processing, and signal processing described later, which are performed based on the extracted various color component signals and infrared light signals. Various parameter setting processes and the like are also executed for each frame of image processing.

[Visible light signal processor, infrared light signal processor, and signal synthesizer]
The visible light signal processing unit 7 performs various signal processing such as white balance processing, noise reduction processing, and gain correction processing on the various color component signals input from the visible light signal generation unit 3. This signal processing is performed based on signals (various signal processing parameters) input from the control unit 20. Then, the visible light signal processing unit 7 outputs a signal subjected to these various signal processings to the signal synthesis unit 9.

  The infrared light signal processing unit 8 performs, for example, noise reduction on the infrared light signal input from the infrared light signal generation unit 4 based on signals (various signal processing parameters) input from the control unit 20. Various signal processing such as processing and gain correction processing are performed. Then, the infrared signal processing unit 8 outputs a signal subjected to these various signal processings to the signal synthesis unit 9.

  Based on the signals (various signal processing parameters) input from the control unit 20, the signal synthesis unit 9 receives the signal input from the visible light signal processing unit 7 and the signal input from the infrared light signal processing unit 8. Is synthesized. Then, the signal synthesis unit 9 outputs the synthesized signal as a video signal to a display device such as an external monitor, for example.

[Optical filter]
The optical filter 2 (first optical filter) is configured by a filter (bandpass filter) that transmits only the optical signal of the visible light component included in the subject signal incident on the imaging unit 10.

  Although not shown, in the present embodiment, a moving mechanism unit that can move the position of the optical filter 2 is also provided. The moving mechanism unit moves the optical filter 2 to a position on the optical path between the lens 11 and the image sensor 12 based on the drive control signal input from the optical filter control unit 5, or is disposed on the optical path. The optical filter 2 is moved to a position other than the optical path. Moreover, the moving mechanism unit may be provided integrally with the optical filter 2 or may be provided separately from the optical filter 2.

[Optical filter controller]
The optical filter control unit 5 drives (moves) the optical filter 2 based on a control signal input from the control unit 20 (a signal ratio calculation unit 22 described later).

  For example, when a control signal for inserting the optical filter 2 into the optical path of incident light (hereinafter referred to as an insertion control signal) is input from the control unit 20, the optical filter control unit 5 causes the optical filter 2 to be placed on the optical path. A drive control signal for arranging at a predetermined position is output to the optical filter 2. By this processing, the operation of the insertion function of the optical filter 2 is started. When the optical filter 2 is already arranged at a predetermined position (insertion position) on the optical path during this process, the optical filter control unit 5 maintains the optical filter 2 at the insertion position based on the insertion control signal. The optical filter 2 is driven and controlled as described above.

  On the other hand, when a control signal for removing the optical filter 2 from the optical path of the incident light (hereinafter referred to as a removal control signal) is input from the control unit 20, the optical filter control unit 5 removes the optical filter 2 from other than the optical path. A drive control signal for arranging at a specific position is output to the optical filter 2. By this process, the operation of the removal function of the optical filter 2 is started. If the optical filter 2 is already placed at a specific position (removal position) other than the optical path during this processing, the optical filter control unit 5 maintains the optical filter 2 at the removal position based on the removal control signal. The optical filter 2 is driven and controlled as described above.

  In this embodiment, when the subject environment is an environment with extremely high illuminance (hereinafter referred to as ultra-high illuminance), regardless of the light source determination result described later (determination result regarding whether or not the optical filter 2 is inserted), the optical The filter 2 is disposed at an insertion position on the optical path between the lens 11 and the image sensor 12.

[Infrared irradiation part]
The infrared irradiation unit 6 includes an infrared LED (Light Emitting Diode: not shown), and turns on the infrared LED based on a control signal input from the control unit 20 to irradiate the subject with infrared light.

  In the present embodiment, when the subject environment is extremely low illuminance (hereinafter referred to as ultra-low illuminance), infrared rays are used regardless of the light source determination result (determination result regarding whether or not the optical filter 2 is inserted), which will be described later. The irradiation unit 6 turns on the infrared LED and irradiates the subject with infrared light. In addition, this infrared light irradiation process is performed based on the infrared irradiation amount calculated by the infrared irradiation amount control part 23 mentioned later. Moreover, when the infrared irradiation operation by the infrared irradiation unit 6 is performed, even if the operation of the insertion function of the optical filter 2 is determined by the determination process of the control unit 20, the determination result is ignored.

[Control unit]
FIG. 5 is a functional block diagram showing the internal configuration of the control unit 20. As shown in FIG. 5, the control unit 20 includes an exposure control unit 21, a signal ratio calculation unit 22, an infrared irradiation amount control unit 23, a light source determination unit 24, and a signal processing set value calculation unit 25. .

  The exposure control unit 21 receives the signal amount of each color component signal included in the visible light signal input from the visible light signal generation unit 3 and the signal amount of the infrared light signal input from the infrared light signal generation unit 4. Based on the above, an exposure control signal is generated, and the exposure control signal is output to the imaging unit 10. In the present embodiment, the diaphragm mechanism and shutter mechanism (not shown) in the imaging unit 10 are controlled based on the exposure control signal, and the light amount (exposure condition) of the light incident on the imaging element 12 is appropriately adjusted. .

  The signal ratio calculation unit 22 receives the signal amount of each color component signal included in the visible light signal input from the visible light signal generation unit 3 and the signal of the infrared light signal input from the infrared light signal generation unit 4. Based on the quantity, the signal ratio of each light component is calculated. Specifically, the signal ratio calculation unit 22 determines the ratio of the signal amount of red component light (R_Ratio) and the ratio of the signal amount of green component light (G_Ratio) to the signal amount of the entire visible light band (visible light signal amount). ) And the signal amount ratio (B_Ratio) of the blue component light. Further, the signal ratio calculation unit 22 calculates the ratio (IR_Ratio) of the signal amount of infrared light to the total signal amount (total signal amount of visible light and infrared light). The method for calculating the various signal ratios will be described in detail later. Then, the signal ratio calculation unit 22 outputs the calculated various signal ratios to the infrared irradiation amount control unit 23, the light source determination unit 24, and the signal processing set value calculation unit 25.

  The signal ratio calculation unit 22 receives the light source determination result (light source information) from the light source determination unit 24 and the infrared irradiation amount calculation result from the infrared irradiation amount control unit 23. Based on these pieces of information, the signal ratio calculation unit 22 determines whether or not the operation of the insertion function of the optical filter 2 is necessary (whether or not the operation of the removal function of the optical filter 2 is necessary). Then, the signal ratio calculation unit 22 outputs a predetermined control signal (insertion control signal or removal control signal) corresponding to the determination result to the optical filter control unit 5.

  The infrared irradiation amount control unit 23 calculates the irradiation amount of infrared rays irradiated to the subject based on various signal ratios input from the signal ratio calculation unit 22. The infrared irradiation amount control unit 23 outputs the calculation result (infrared irradiation amount) to the infrared irradiation unit 6 and the signal ratio calculation unit 22.

  The light source determination unit 24 determines the type of light source currently used in the subject environment based on various signal ratios input from the signal ratio calculation unit 22. In the present embodiment, the light source determination unit 24 determines whether or not the current light source is a light source to be determined (for example, a light source that includes a large amount of infrared rays such as sunlight, halogen light, and sodium lamp). Specifically, the light source determination unit 24 determines whether or not the values of various signal ratios input from the signal ratio calculation unit 22 are values that can be taken under the subject environment in which the light source to be determined is used. . This light source determination method will be described in detail later.

  Then, the light source determination unit 24 updates the light source information based on the determination result, and outputs the light source information to the signal ratio calculation unit 22 and the signal processing set value calculation unit 25. If the light source information after the determination process is the same as that before the determination process, the light source determination unit 24 does not perform the light source information update process.

  The signal processing set value calculation unit 25 calculates optimum values of various signal processing parameters (setting parameters) used in various signal processing performed by the visible light signal processing unit 7 and the infrared light signal processing unit 8. This calculation process is performed based on various signal ratios input from the signal ratio calculation unit 22, infrared irradiation amount, position information of the optical filter 2, and light source information input from the light source determination unit 24. Then, the signal processing set value calculation unit 25 outputs the calculated various signal processing parameters to the visible light signal processing unit 7 and the infrared light signal processing unit 8.

  The signal processing set value calculation unit 25 is used in the synthesis process performed by the signal synthesis unit 9, for example, a signal processing parameter such as a matrix setting value for generating a gain value, a luminance signal, and a color signal of each signal. (Setting parameter) is calculated. Then, the signal processing set value calculation unit 25 outputs the calculated signal processing parameters to the signal synthesis unit 9.

  As shown in FIG. 1, the control unit 20 includes a memory unit 20a in which various programs and various data necessary for various processes are stored. The memory unit 20a can be composed of, for example, a ROM (Read Only Memory) or the like. Note that the memory unit 20 a may be provided outside the control unit 20. Although not shown, the control unit 20 also includes a working memory for temporarily storing various processing data and the like when executing various processing. This working memory can be composed of, for example, a RAM (Random Access Memory) or the like.

  Examples of various data stored in the memory unit 20a (storage unit) include determination data such as a signal ratio threshold and a light source spectral characteristic, which will be described later, used in the light source determination process. Furthermore, examples of various data stored in the memory unit 20a include various signal processing parameters used in various signal processing of the visible light signal processing unit 7, the infrared light signal processing unit 8, and the signal synthesis unit 9. . Note that these data may be stored in the memory unit 20a in advance, for example, at the manufacturing stage of the imaging apparatus 1, or various data may be stored later by a predetermined operation by a user (operator) of the imaging apparatus 1 or the like. You may make it the structure which can be added to the memory part 20a. When the latter configuration is provided, various kinds of data (determination data, signal processing parameters, etc.) optimum for the user (operator) or the like can be added to the memory unit 20a according to the subject environment. In this case, for example, the determination accuracy of the light source and the reproducibility of the color image can be further improved.

[Signal ratio calculation method]
Next, a method for calculating various signal ratios (R_Ratio (first signal ratio), G_Ratio (second signal ratio), B_Ratio (third signal ratio), and IR_Ratio (fourth signal ratio)) will be described. First, a method for calculating various signal ratios when the optical filter 2 is not disposed on the optical path between the lens 11 and the image sensor 12 will be described.

  In this case, first, for each basic pixel group, the signal ratio calculation unit 22 has a ratio of the signal amount of the red component light (r_ratio) and a ratio of the signal amount of the green component light (g_ratio) to the signal amount of the entire visible light band. And the ratio (b_ratio) of the signal amount of the blue component light is calculated. Further, the signal ratio calculation unit 22 calculates the ratio (ir_ratio) of the signal amount of infrared light to the total signal amount for each basic pixel group. In this calculation process, when the basic pixel group is configured as the basic pixel group 30a (see FIG. 2), the signal amount of the entire visible light band is the signal amount of the R signal and the signal of the G signal in the basic pixel group. And the total amount of the B signal. In this case, the total signal amount is the sum of the signal amount of the R signal, the signal amount of the G signal, the signal amount of the B signal, and the signal amount of the IR signal in the basic pixel group. On the other hand, when the basic pixel group is configured as the basic pixel group 30b (see FIG. 2), the signal amount of the entire visible light band is the signal amount of the W signal in the basic pixel group, and the total signal amount is the basic pixel. The sum of the signal amount of the W signal and the signal amount of the IR signal in the group.

  Next, the signal ratio calculation unit 22 obtains average values of r_ratio, g_ratio, b_ratio, and ir_ratio over the entire imaging region 12a calculated for each basic pixel group. Then, the signal ratio calculation unit 22 sets the calculated average values of r_ratio, g_ratio, b_ratio, and ir_ratio as R_Ratio, G_Ratio, B_Ratio, and IR_Ratio, respectively. The method for calculating the signal ratio is not limited to this. For example, the average value of the signal amount of each color component signal and infrared light signal is obtained over the entire imaging region 12a, and the average value of each signal amount is used. The signal ratio may be calculated.

  Next, a method for calculating various signal ratios when the optical filter 2 is disposed on the optical path between the lens 11 and the image sensor 12 will be described.

  In this case, the signal ratio calculation unit 22 calculates the signal ratio (R_Ratio, G_Ratio, B_Ratio) of each color component light in the same manner as when the optical filter 2 is not disposed on the optical path between the lens 11 and the image sensor 12. To do. However, in this case, since the actual measurement value of the infrared light signal cannot be obtained, the signal ratio (IR_Ratio) of the infrared light is calculated as follows.

  First, the signal ratio calculation unit 22 compares the calculated signal ratio of each color component light with the spectral characteristics of various light sources stored in the memory unit 20 a in the control unit 20. Next, the signal ratio calculation unit 22 estimates the type of the light source based on the comparison result between the two. At this time, the signal ratio calculation unit 22 estimates the type of the light source by selecting, for example, the spectrum characteristic having the most similar magnitude and ratio between R_Ratio, G_Ratio, and B_Ratio from the spectral characteristics of various light sources. Next, the signal ratio calculation unit 22 estimates the signal ratio (IR_Ratio) of infrared light based on the estimated spectrum characteristics of the light source and the calculated signal ratio of each color component light.

  For example, when a halogen lamp is used as the light source, the relationship “B_Ratio <G_Ratio ≦ R_Ratio” is established between R_Ratio, G_Ratio, and B_Ratio. Therefore, the signal ratio calculation unit 22 determines whether this relationship is established among R_Ratio calculated based on the actual measurement value of the visible light signal, G_Ratio, and B_Ratio, and the light source is a halogen lamp. It is determined whether or not. When it is determined that the light source is a halogen lamp, the signal ratio calculation unit 22 estimates IR_Ratio based on the spectral characteristics of the halogen lamp.

  In addition, the relationship among R_Ratio, G_Ratio, and B_Ratio in the spectral characteristics of the light source may vary slightly depending on the sensitivity characteristics of the image sensor 12 to be used. For example, when the light source is a halogen lamp, when the applied voltage is small, R_Ratio tends to increase, and G_Ratio and B_Ratio tend to decrease. Therefore, when estimating IR_Ratio, the relationship between R_Ratio, G_Ratio, and B_Ratio may be considered, and IR_Ratio may be corrected according to the difference between R_Ratio and G_Ratio and B_Ratio. For example, the correction may be performed so that the estimated value of IR_Ratio increases as the difference between R_Ratio and G_Ratio and B_Ratio increases.

[Light source determination method]
Next, a light source determination method executed by the light source determination unit 24 will be described. In the present embodiment, the light source determination unit 24 compares the magnitude relationship between each signal ratio and the corresponding threshold value. Specifically, the light source determination unit 24 compares R_Ratio with a corresponding threshold value (R_Ratio_th), and compares G_Ratio with a corresponding threshold value (G_Ratio_th). Further, the light source determination unit 24 compares B_Ratio with a corresponding threshold value (B_Ratio_th), and compares IR_Ratio with a corresponding threshold value (IR_Ratio_th).

  In this comparison process, the light source determination unit 24 determines that the calculated signal ratio values are within a range that can be taken in the subject environment where the light source to be determined is used, with the corresponding threshold as a boundary (a range larger than the threshold). Or a small range value). Next, the light source determination unit 24 determines whether or not the light source used in the subject environment is a determination target light source based on the determination (comparison) result. Then, the light source determination unit 24 updates or maintains the light source type information (light source information) based on the determination result.

  In the present embodiment, the discrimination (comparison) result of each signal ratio is represented by flag information (“High” or “Low”). Specifically, the comparison result between R_Ratio and R_Ratio_th is represented by “Flag_R”, and the comparison result between G_Ratio and G_Ratio_th is represented by “Flag_G”. Further, the comparison result between B_Ratio and B_Ratio_th is represented by “Flag_B”, and the comparison result between IR_Ratio and IR_Ratio_th is represented by “Flag_IR”.

  When the value of the signal ratio is a value within a range that can be taken under the subject environment where the light source to be determined is used with the corresponding threshold as a boundary, the corresponding flag information is “High”. Shall. Therefore, when the light source to be determined is used in the current subject environment, Flag_R, Flag_G, Flag_B, and Flag_IR are all “High”. On the other hand, when the light source to be determined is not used in the current subject environment, at least one of Flag_R, Flag_G, Flag_B, and Flag_IR is “Low”.

  In the present embodiment, the threshold used in the signal ratio comparison process is varied based on the optical information at the time of the determination process (optical information at the end of the process of the previous frame). That is, the threshold when the current optical information (optical information before update) is light source information corresponding to the light source to be determined, and the threshold when the current optical information is light source information corresponding to a light source other than the determination target Set to a different value. In addition, this invention is not limited to this, The threshold value of a signal ratio may be the same irrespective of the present optical information.

  6A to 6D are diagrams showing the relationship between each signal ratio value, the corresponding threshold value, and the corresponding flag information when a halogen lamp is used as the light source to be determined. It is a figure which shows the control image of the determination process of the necessity of the action | operation of 2 insertion functions. A thick solid line in each of FIGS. 6A to 6D shows a transition mode of flag information with respect to a change in the value of each signal ratio when the current light source information is light source information other than a halogen lamp. On the other hand, the thick broken line in each figure of Drawing 6A-Drawing 6D shows the transition mode of flag information to the change of the value of each signal ratio when the current light source information is the light source information of the halogen lamp.

  That is, for example, when the light source is a halogen lamp in the determination process of the previous frame, or when the insertion function of the optical filter 2 is currently operating, the thick broken line in each of FIGS. The determination process is performed with reference to the transition mode of the flag information shown. In other cases, the determination process is performed with reference to the transition state of the flag information indicated by the thick broken line in each of FIGS. 6A to 6D. Thus, by changing the threshold value of each signal ratio according to the current light source information and providing hysteresis in the transition mode of the flag information, the light source information is not erroneously updated due to the influence of noise or the like. it can.

Then, the light source determination unit 24 determines that the light source is a halogen lamp when all the flag information is “High” as a result of comparison processing between each signal ratio and the corresponding threshold value. In the example shown in FIGS. 6A to 6D, when the following relationship (1) or (2) is established, the light source determination unit 24 determines that the light source is a halogen lamp.
(1) R_Ratio> R_Ratio_th1, G_Ratio <G_Ratio_th1, B_Ratio <B_Ratio_th1, and IR_Ratio> IR_Ratio_th1
(2) R_Ratio> R_Ratio_th2, G_Ratio <G_Ratio_th2, B_Ratio <B_Ratio_th2, and IR_Ratio> IR_Ratio_th2

  On the other hand, the light source determination unit 24 determines that the light source is not a halogen lamp when at least one flag information is “Low” by the comparison process. Even in this case, when all the signal ratios for which the flag information is “Low” are very close to the corresponding threshold values (values within the range of the threshold value ± δ), the light source The determination unit 24 determines that the light source is a halogen lamp.

  In this embodiment, a plurality of types of light sources to be determined, that is, subject environments to be considered can be set. In this case, the above-described signal ratio threshold, light source spectral characteristics, various signal processing parameters used in various signal processing, and the like are also set separately for each light source to be determined (for each subject environment), and the memory unit 20a. Stored in

[Optical filter insertion / removal function switching control]
FIG. 7 is a flowchart illustrating the procedure of the switching control processing (movement control processing of the optical filter 2) of the insertion / removal function of the optical filter 2 in the imaging device 1 of the present embodiment.

  First, the visible light signal generation unit 3 extracts (calculates) each color component signal (R signal, G signal, B signal, W signal) included in the visible light signal from the input electrical signal for each basic unit group. (S1). In the processing of S1, the infrared light signal generation unit 4 extracts (calculates) an infrared light signal (IR signal) from the input electrical signal for each basic unit group. Next, the signal ratio calculation unit 22 acquires the signal amount of each color component signal and infrared light signal extracted for each basic unit group by the visible light signal generation unit 3 and the infrared light signal generation unit 4 (S2).

  Next, the signal ratio calculation unit 22 determines whether or not the optical filter 2 is inserted at a predetermined position (insertion position) on the optical path between the lens 11 and the image sensor 12 (S3). In this process, the signal ratio calculation unit 22 inserts the optical filter 2 based on the type of drive control signal currently output from the optical filter control unit 5 to the optical filter 2 (position information of the optical filter 2). It is determined whether or not.

  In the process of S3, when the signal ratio calculation unit 22 determines that the optical filter 2 is not inserted (when S3 is NO), the signal ratio calculation unit 22 calculates the signal ratio of each color component light and infrared light. Is calculated (S4). Specifically, the signal ratio calculation unit 22 calculates R_Ratio, G_Ratio, B_Ratio, and IR_Ratio based on the various signal amounts acquired in S2.

  On the other hand, in the process of S3, when the signal ratio calculation unit 22 determines that the optical filter 2 is inserted (when S3 is YES), the signal ratio calculation unit 22 calculates each signal ratio of each color component light. Calculate (S5). Specifically, the signal ratio calculation unit 22 calculates R_Ratio, G_Ratio, and B_Ratio based on the various signal amounts acquired in S2.

  Next, the signal ratio calculation unit 22 estimates the type of light source based on the signal ratio of each color component light calculated in S5 and the spectral characteristics of various light sources, and estimates the signal ratio (IR_Ratio) of infrared light. A value is calculated (S6).

  After the processing of S4 or S6, the light source determination unit 24 compares each calculated signal ratio with the corresponding threshold value based on the current light source information (S7). Next, the light source determination unit 24 sets flag information corresponding to the comparison result of S7 for each signal ratio (S8).

  Next, the light source determination unit 24 performs light source determination based on the set flag information of each signal ratio (S9). In this process, when all the flag information is “High”, the light source determination unit 24 determines that the light source is a light source to be determined (a light source containing a lot of infrared rays), and otherwise. The light source determination unit 24 determines that the light source is a light source other than the determination target. Even when at least one flag information is “Low”, when all signal ratios for which the flag information is “Low” are very close to the corresponding threshold values, The light source determination unit 24 determines that the light source is a light source to be determined.

  Next, the light source determination unit 24 performs light source information update processing (or maintenance processing) based on the determination result of S9 (S10). In this process, when the light source information before the determination process of S9 is different from the light source information after the determination process, the light source determination unit 24 performs a light source information update process.

  Next, the infrared irradiation amount control unit 23 calculates the irradiation amount of infrared rays irradiated to the subject based on the various signal ratios input from the signal ratio calculation unit 22 (S11). Note that the process of S11 can be executed at an arbitrary timing as long as it is after the process of S4 or S6 and before the process of S12 described later. Moreover, the process of S11 may be performed in parallel with the process of S7-S10 which the light source determination part 24 performs.

  Next, the signal ratio calculation unit 22 determines whether or not the subject environment is an ultra-high illuminance environment (S12). In this processing, the signal ratio calculation unit 22 determines whether or not the subject environment is an ultra-high illuminance environment with reference to, for example, the content of the exposure control signal output from the exposure control unit 21 to the imaging unit 10. May be. Further, in this process, the signal ratio calculation unit 22 determines whether or not the subject environment is an ultra-high illuminance environment, for example, by determining whether or not the amount of incident light exceeds a predetermined threshold. May be.

  In the process of S12, when the signal ratio calculation unit 22 determines that the subject environment is an ultra-high illuminance environment (when S12 is YES), the signal ratio calculation unit 22 performs the process of S15 described later.

  On the other hand, in the processing of S12, when the signal ratio calculation unit 22 determines that the subject environment is not an ultra-high illuminance environment (when S12 is NO), the signal ratio calculation unit 22 determines that the subject environment is an ultra-low illuminance. It is determined whether or not the environment is present (S13). In this process, the signal ratio calculation unit 22 determines whether or not the subject environment is an ultra-low illuminance environment with reference to, for example, the contents of the exposure control signal output from the exposure control unit 21 to the imaging unit 10. May be. Further, in this process, the signal ratio calculation unit 22 determines whether the subject environment is an environment with extremely low illuminance, for example, by determining whether the amount of incident light is less than a specific threshold value. May be.

  In the process of S13, when the signal ratio calculation unit 22 determines that the subject environment is an environment with extremely low illuminance (when S13 is YES), the imaging apparatus 1 performs a process of S17 described later.

  On the other hand, in the process of S13, when the signal ratio calculation unit 22 determines that the subject environment is not an environment with extremely low illuminance (when S13 is NO), the signal ratio calculation unit 22 has an infrared irradiation amount of zero. And it is discriminate | determined whether an infrared-light component is large (S14). In this process, when the light source information determined in the process of S10 is light source information corresponding to a light source to be determined (a light source that includes a large amount of infrared rays), the signal ratio calculation unit 22 uses red light included in the incident light. It is determined that the external light component is large. The present invention is not limited to this, and the signal ratio calculation unit 22 determines whether or not the infrared light component included in the incident light is large based on the information indicating the amount of incident infrared light. May be.

  In the process of S14, when the signal ratio calculation unit 22 determines that the determination condition of S14 is not satisfied (when S14 is NO), the imaging device 1 performs a process of S17 described later. On the other hand, in the process of S14, when the signal ratio calculation unit 22 determines that the determination condition of S14 is satisfied (when S14 is YES determination) or when S12 is YES determination, the signal ratio calculation unit 22 inserts The control signal is output to the optical filter control unit 5 (S15).

  Next, the optical filter control unit 5 drives and controls the optical filter 2 based on the input insertion control signal, and controls the optical filter 2 to a predetermined position (insertion position: first position) on the optical path between the lens 11 and the image sensor 12. (S16). Note that when the optical filter 2 is not disposed at the insertion position at the end of the processing of the previous frame, in this processing, the optical filter control unit 5 performs a process of moving the optical filter 2 to the insertion position. On the other hand, when the optical filter 2 is disposed at the insertion position at the end of the processing of the previous frame, in this processing, the optical filter control unit 5 performs processing for maintaining the state where the optical filter 2 is disposed at the insertion position. . After the process of S16, the imaging apparatus 1 ends the insertion / removal function switching control process of the optical filter 2 (optical condition switching process at the time of photographing).

  Here, returning to the processing of S13 and S14 again, if S13 is YES or S14 is NO, the infrared irradiation unit 6 irradiates the subject with infrared rays with the infrared irradiation amount calculated in S11. (S17). Note that when the infrared irradiation amount calculated in S11 is zero, the subject is not irradiated with infrared rays in the processing of S17.

  Next, the signal ratio calculation unit 22 outputs a removal control signal to the optical filter control unit 5 (S18).

  Next, the optical filter control unit 5 drives and controls the optical filter 2 based on the input removal control signal, and the optical filter 2 is moved to a specific position (removal position) other than the position on the optical path between the lens 11 and the image sensor 12. : Second position) (S19). When the optical filter 2 is arranged at the insertion position at the end of the process of the previous frame, in this process, the optical filter control unit 5 performs a process of moving the optical filter 2 to the removal position. On the other hand, when the optical filter 2 is disposed at the removal position at the end of the processing of the previous frame, in this process, the optical filter control unit 5 performs a process of maintaining the state where the optical filter 2 is disposed at the removal position. . Then, after the process of S19, the imaging apparatus 1 ends the switching control process for the insertion / removal function of the optical filter 2 (the optical condition switching process at the time of photographing).

[effect]
In the imaging apparatus 1 according to the present embodiment, the above-described insertion / removal function of the optical filter 2 is provided, and both these functions are appropriately switched according to the subject environment. Therefore, according to the imaging apparatus 1 of the present embodiment, the optical condition (imaging condition) at the time of shooting can be automatically changed to the optimum condition according to the subject environment. More specifically, the following effects can be obtained.

  8A and 8B are diagrams illustrating the relationship between the amount of light incident from the subject (the amount of light of the subject) and the value of the signal (output signal) output from the image sensor 12. Consider a case where the subject environment is a high illuminance environment and the output signal of the image sensor 12 is saturated with respect to the amount of light of the subject. In this environment, for example, when the amount of infrared light included in the amount of light of the subject is twice or more than the amount of visible light, as shown in FIG. 8A, the amount of light of the subject that is actually subject to signal processing ( The amount of infrared light included in the amount of light excluding the saturation amount is also twice or more the amount of visible light. In this case, the amount of visible light that is actually subject to signal processing is reduced (the output signal of the visible light component output from the image sensor 12 is reduced).

  Normally, a luminance signal is generated based on both a visible light signal and an infrared light signal, but a signal having color information (color signal) is generated based only on a visible light signal. Therefore, for example, when the amount of infrared light included in the amount of light of the subject is large, the color signal is small with respect to the luminance signal, so that a light color image is generated.

  In this case, for example, in order to compensate for an insufficient signal amount with respect to the color signal (visible light signal), for example, the color signal is amplified by performing gain correction processing or the like, thereby darkening the color of the image. However, in this case, noise is also amplified in the process of the gain correction process. That is, when light including an infrared ray amount more than necessary is incident, unnecessary noise is generated in the video signal. Such a situation is likely to occur when, for example, a light source such as sunlight, halogen light, or a sodium lamp that contains a large amount of infrared light is used (except in a low-light environment).

  On the other hand, in the present embodiment, when a light source containing a large amount of infrared light is used and the subject environment is high illuminance, the insertion function of the optical filter 2 is activated (on the optical path between the lens 11 and the image sensor 12). The optical filter 2 is disposed on the optical filter 2 to block the infrared light component contained in the incident light. In this case, only the visible light component is incident on the image sensor 12. Therefore, the change characteristic of the output signal of the image sensor 12 with respect to the amount of light of the subject is as shown in FIG. 8B, and the signal corresponding to the amount of light of the subject that is actually subject to signal processing is the light amount of the visible light component. Only the corresponding visible light signal is obtained. In this case, since the visible light signal output from the image sensor 12 becomes large, it is possible to generate both the luminance signal and the color signal without performing gain correction processing.

  Note that the above-described problem is, for example, that the first unit pixel constituting the basic pixel group is a unit pixel that can receive only red light, and the second unit pixel is a unit pixel that can receive only green light. This may also occur when the third unit pixel is a unit pixel that can receive only blue light. Therefore, the insertion / removal function of the optical filter 2 of the present embodiment can be similarly applied to an imaging apparatus having a basic pixel group having such a configuration.

<Second Embodiment>
[Configuration of imaging device]
FIG. 9 is a block diagram illustrating a configuration of an imaging apparatus according to the second embodiment. In addition, in the imaging device 40 of this embodiment shown in FIG. 9, the same code | symbol is attached | subjected and shown to the structure similar to the structure of the imaging device 1 (refer FIG. 1) of the said 1st Embodiment, and those structures are shown. Description of is omitted.

  As is apparent from a comparison between FIG. 9 and FIG. 1, the imaging device 40 of the present embodiment has a configuration in which one optical filter is further added to the configuration of the imaging device 1 of the first embodiment. That is, the imaging device 40 of this embodiment includes two optical filters 41 and 42 (a first optical filter 41 and a second optical filter 42).

  Similar to the optical filter 2 of the first embodiment, the first optical filter 41 is a filter that transmits only an optical signal in the visible light band. The second optical filter 42 is a filter that transmits an optical signal in the visible light band and an optical signal in the infrared light band. Further, the second optical filter 42 has a frequency characteristic that blocks an optical signal in the near-infrared light band existing on the visible light band side of the infrared light band. Although not shown, the imaging device 40 also includes a moving mechanism unit that switches the arrangement positions of the first optical filter 41 and the second optical filter 42 based on the drive control signal output from the optical filter control unit 5.

[Optical filter switching control]
In the switching control of the optical filter of the imaging device 40, when a light source with a large amount of infrared rays is used in the subject environment, the first optical filter 41 is positioned at a predetermined position on the optical path of incident light ( The drive (movement) is controlled so as to be arranged at the (insertion position). At this time, the second optical filter 42 is controlled to be driven (moved) so as to be disposed at a specific position (removal position) other than the position on the optical path. On the other hand, in other cases, the second optical filter 42 is driven and controlled to be disposed at the insertion position, and the first optical filter 41 is driven and controlled to be disposed at the removal position.

  In addition, when the subject environment is an ultra-high illuminance environment, the first optical filter 41 is disposed at the insertion position and the second optical filter 42 is disposed at the removal position regardless of the determination result of the light source. The first optical filter 41 and the second optical filter 42 are driven and controlled. On the other hand, when the subject environment is an extremely low illuminance environment, the second optical filter 42 is disposed at the insertion position and the first optical filter 41 is disposed at the removal position regardless of the determination result of the light source. The first optical filter 41 and the second optical filter 42 are driven and controlled.

  The driving (moving) operation of the first optical filter 41 and the second optical filter 42 described above is controlled by the optical filter control unit 5. The light source determination process in the optical filter switching control according to the present embodiment is performed in the same manner as in the first embodiment.

  Further, the optical filter switching control (movement control) flow of the present embodiment is the same control flow as that of the first embodiment shown in FIG. If an insertion control signal is output from the signal ratio calculation unit 22 to the optical filter control unit 5 in the present embodiment, the first optical filter 41 is disposed at the insertion position in the process of S16, and the second optical filter A drive control process is performed so as to place 42 at the removal position. On the other hand, when the removal control signal is output from the signal ratio calculation unit 22 to the optical filter control unit 5 in the present embodiment, the first optical filter 41 is disposed at the removal position in the process of S19, and the second optical filter A drive control process is performed so as to place 42 at the insertion position.

[effect]
Also in the imaging apparatus 40 and the optical filter switching control (movement control) method of the present embodiment described above, the optical conditions at the time of shooting are automatically set to the optimum conditions according to the subject environment, as in the first embodiment. Transitions can be made. In the present embodiment, the following effects are also obtained.

  Among various general-purpose (low-cost) image sensors that can receive both visible light and infrared light, the near-infrared light band existing on the visible light band side of the infrared light band in the unit pixel is used. However, there are image sensors that are sensitive to some extent. FIG. 10A shows sensitivity characteristics of a third unit pixel that can receive blue component light and infrared light, and forms a basic pixel group in such a general-purpose image sensor, and FIG. 10B shows green component light and infrared light. It is a sensitivity characteristic of the 2nd unit pixel which can receive light. FIG. 10C shows sensitivity characteristics of the first unit pixel that can receive red component light and infrared light, which constitutes a basic pixel group in a general-purpose image sensor, and FIG. 10D can receive only infrared light. This is the sensitivity characteristic of the fourth unit pixel.

  In such a general-purpose imaging device, the first to fourth unit pixels have a certain degree of sensitivity with respect to the near-infrared light band (the “NIR” band in FIGS. 10A to 10D). Further, the sensitivity characteristics of the near-infrared light band in the first to third unit pixels are different from each other. When a general-purpose imaging device having such sensitivity characteristics is employed in the imaging device 1 of the first embodiment, for example, an image obtained when the insertion function of the optical filter 2 is not operated is light in the near infrared light band. Under such circumstances, the image may become reddish and color reproducibility may be reduced.

  On the other hand, in the imaging device 40 of the present embodiment, when the insertion function of the second optical filter 42 is not in operation, the second optical filter 42 that blocks light in the near-infrared band is light between the lens 11 and the imaging device 12. It is arranged at the insertion position on the road. Therefore, in the imaging device 40 of the present embodiment, even if a general-purpose imaging device having sensitivity characteristics as shown in FIGS. 10A to 10D is adopted, the imaging device 40 is not affected by light in the near-infrared light band. Accurate color images can be obtained, and color reproducibility can be improved. In the present embodiment, a low-priced general-purpose image sensor can also be adopted, so that the cost of the image pickup apparatus 40 can be reduced.

<Various modifications>
As described above, the imaging apparatus and the imaging method according to the various embodiments of the present invention have been described. However, the present invention is not limited to these, and so long as it does not depart from the gist of the present invention described in the claims. Various modifications and application examples can be employed.

[Modification 1]
In the various embodiments described above, when the optical filter 2 is disposed on the optical path between the lens 11 and the image sensor 12, the infrared light signal ratio (IR_Ratio) is changed to the visible light signal ratio (R, G, B_Ratio). ) Has been described based on an example of estimation. However, the present invention is not limited to this. For example, when the optical filter 2 is disposed on the optical path between the lens 11 and the image sensor 12, a component that separately measures the amount of infrared light (infrared amount) included in the incident light from the subject may be provided. Good.

  FIG. 11 is a functional block diagram illustrating an internal configuration of the control unit 50 of the imaging apparatus according to the first modification. In the control unit 50 of this example shown in FIG. 11, the same reference numerals are given to the same components as those of the control unit 20 of the first embodiment shown in FIG. Is omitted.

  As is apparent from a comparison between FIG. 11 and FIG. 5, the control unit 50 of this example has a configuration in which an infrared light receiving unit 51 is further provided in the configuration of the control unit 20 of the first embodiment. The infrared light receiving unit 51 can be composed of a light receiving element such as a photodiode for receiving infrared light, for example. The infrared light receiving unit 51 may be provided outside the control unit 50.

  In the imaging apparatus of this example, even when the optical filter 2 is disposed on the optical path between the lens 11 and the imaging element 12, the amount of infrared light included in the incident light from the subject is directly acquired. Therefore, the accuracy of the light source determination process can be improved.

[Modification 2]
In the various embodiments and the first modification, the example in which the common signal ratio threshold value (determination parameter) is used in the determination process of the switching control of the insertion / removal function of the optical filter 2 and the determination process of the light source has been described. The present invention is not limited to this. For example, a signal ratio threshold value used for determination of switching control of the insertion / removal function of the optical filter 2 may be provided separately from a signal ratio threshold value used for light source determination, and each determination process may be performed separately. In this case, the former threshold value may be different from or equal to the latter threshold value depending on, for example, the type of light source to be determined (subject environment).

  In the above-described various embodiments and Modification 1 described above, in the light source determination process, an example in which it is determined whether or not the light source is a light source that includes a large amount of infrared light (a determination target light source) has been described. Is not limited to this. For example, the type of the light source itself may be determined using threshold values of signal ratios and spectral characteristics of various light sources. In this case, necessary signal ratio threshold values and spectral characteristic data of various light sources are stored in advance in the memory unit in the control unit, or are additionally written in the memory unit by the user.

[Modification 3]
In the various embodiments and various modifications described above, the example in which the signal ratio of each light component in the entire imaging region 12a is calculated and various determination processes are performed using the signal ratio has been described, but the present invention is not limited to this. For example, the imaging region 12a may be divided into a plurality of regions (divided regions), a signal ratio may be calculated for each divided region, and various determination processes may be performed using the signal ratio. In this case, the determination result obtained for each divided region may be statistically processed, and the determination result dominant in the plurality of divided regions may be used as the determination result for the entire imaging region 12a (the entire subject).

  In this example, only the determination results obtained in some of the divided areas may be statistically processed to obtain the determination result for the entire imaging area 12a (the entire subject). If this method is adopted, for example, if the subject environment is a special environment where a part of the subject is always irradiated with light from a specific light source, a divided region including a part of the subject is determined. It can be selectively excluded from processing targets. As a result, the influence of a specific light source can be suppressed in the light source determination process. Therefore, if the imaging method of this example is adopted, for example, it is possible to more appropriately cope with various subject environments (imaging device use environments), various needs of users, and the like.

[Modification 4]
In the various embodiments and the various modifications described above, the example in which all signal ratios of the red component light, the green component light, and the blue component light are used in the light source determination process has been described, but the present invention is not limited thereto. For example, when a solid-state image sensor such as a CMOS type or a CCD type is used as the image sensor 12, the sensitivity to green component light is hardly affected by a change in the light source (type of light source). Therefore, the signal ratio of the green component light may not be used in the light source determination process.

[Modification 5]
In the various embodiments and the various modifications described above, the example in which the infrared signal ratio (IR_Ratio) is estimated using the spectral characteristics of the light source during the operation of the optical filter insertion function has been described. It is not limited to this. For example, during the operation of the optical filter insertion function, a function for calculating IR_Ratio using an estimation formula (empirical formula) may be provided in the control unit. In this case, a function (learning function) for automatically optimizing the coefficients of various parameters used in the estimation formula according to the use environment of the imaging apparatus and the subject environment may be added to the control unit. Good. By providing these functions, it is possible to more appropriately cope with a wide variety of subject environments (use environments of the imaging device).

  In addition, as a parameter prescribed | regulated by this estimation formula (empirical formula), each signal ratio etc. immediately before and immediately after insertion to the optical path of an optical filter can be used, for example. In addition, this estimation formula may be provided separately for each subject environment with a high occurrence frequency, for example. Furthermore, a conventionally known learning algorithm can be applied as a learning algorithm used in the learning function.

[Modification 6]
In the various embodiments and the various modifications described above, the example in which the light source determination process (optical filter insertion / removal function switching control process) is performed for each frame has been described, but the present invention is not limited thereto. For example, the light source determination process may be executed every predetermined plural frame periods. In this case, the processing load on the control unit can be reduced.

[Modification 7]
In the various embodiments and the various modifications described above, the example in which the insertion position of the optical filter for blocking infrared light is set to the position on the optical path between the lens 11 and the image sensor 12 has been described. It is not limited to. The insertion position of the optical filter can be set to any position as long as the infrared light incident on the image sensor 12 can be blocked. For example, the insertion position of the optical filter may be set to a position immediately before the light incident surface of the lens 11.

[Other variations]
In the various embodiments and the various modifications described above, the configuration of the imaging apparatus has been described in detail and specifically in order to explain the present invention in an easy-to-understand manner. However, the present invention is not necessarily limited to those having all the configurations described. It is not something. For example, the imaging apparatus of the present invention is a camera module including at least the imaging unit 10, the optical filter 2, the visible light signal generation unit 3, the infrared light signal generation unit 4, the optical filter control unit 5, and the control unit 20 described above. Can be configured. In this case, the other signal processing unit may be configured by a module separate from the camera module.

  In addition, each configuration, function, processing unit, and the like related to the signal control (electrical signal processing control) may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. At this time, some or all of the components, functions, processing units, and the like related to the signal control may be integrated as appropriate.

  Furthermore, each configuration, function, processing unit, and the like related to the signal control may be realized by a processor interpreting and executing a program that realizes each function. That is, each configuration, function, processing unit, and the like related to the signal control may be realized by software. Information (data) such as programs, tables, and files for realizing each function can be stored not only in the above-described built-in memory unit but also in a recording device such as a hard disk or an SSD (Solid State Drive). . Information (data) such as programs, tables, and files for realizing each function can also be stored in a recording medium such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, a DVD (Digital Versatile Disc), or the like. .

  DESCRIPTION OF SYMBOLS 1,40 ... Imaging device, 2 ... Optical filter, 3 ... Visible light signal generation part, 4 ... Infrared light signal generation part, 5 ... Optical filter control part, 6 ... Infrared irradiation part, 7 ... Visible light signal processing part, DESCRIPTION OF SYMBOLS 8 ... Infrared light signal processing part, 9 ... Signal synthetic | combination part, 10 ... Imaging part, 11 ... Lens, 12 ... Imaging element, 20, 50 ... Control part, 20a ... Memory part, 21 ... Exposure control part, 22 ... Signal Ratio calculation unit, 23 ... infrared irradiation amount control unit, 24 ... light source determination unit, 25 ... signal processing set value calculation unit, 30a, 30b ... basic pixel group, 31, 35 ... first unit pixel, 32, 36 ... second Unit pixel, 33, 37 ... third unit pixel, 34, 38 ... fourth unit pixel, 41 ... first optical filter, 42 ... second optical filter

Claims (11)

A lens that forms an optical image by forming incident light from a subject, and an imaging device that can simultaneously receive visible light and infrared light, and that converts an optical signal incident through the lens into an electrical signal An imaging unit having
A first optical filter that transmits only visible light and is movable between a first position on the optical path of the incident light and a second position other than the position on the optical path;
A visible light signal generation unit that calculates only a signal of a visible light component included in the electrical signal;
An infrared light signal generation unit for calculating only the signal of the infrared light component included in the electrical signal;
A control unit that outputs a predetermined control signal based on signals calculated by the visible light signal generation unit and the infrared light signal generation unit;
An optical filter control unit that arranges the first optical filter at the first position or the second position based on the predetermined control signal;
An imaging apparatus comprising: The imaging device includes a first unit pixel capable of receiving red component light and infrared light, a second unit pixel capable of receiving green component light and infrared light, and blue light included in the incident light. A third unit pixel capable of receiving component light and the infrared light; a fourth unit pixel capable of receiving only the infrared light or receiving light in the entire visible light band and the infrared light; Have
In the imaging region of the imaging device, a basic pixel group in which the first unit pixel, the second unit pixel, the third unit pixel, and the fourth unit pixel are arranged in a predetermined manner is arranged in a two-dimensional manner. The imaging apparatus according to claim 1. The visible light signal generation unit is configured to generate the red component light and the green component based on signals obtained from the first unit pixel, the second unit pixel, the third unit pixel, and the fourth unit pixel, respectively. Generating a color component signal corresponding to each of the light and the blue component light,
The infrared light signal generation unit is based on a signal obtained from the fourth unit pixel or a signal obtained from the first unit pixel, the second unit pixel, the third unit pixel, and the fourth unit pixel. The imaging device according to claim 2, wherein an infrared light signal corresponding to the infrared light is generated. further,
An infrared irradiator that irradiates the subject with infrared;
A visible light signal processing unit that performs predetermined signal processing on each color component signal generated by the visible light signal generation unit;
An infrared light signal processing unit that performs specific signal processing on the infrared light signal generated by the infrared light signal generation unit;
The signal subjected to the predetermined signal processing by the visible light signal processing unit and the signal subjected to the specific signal processing by the infrared light signal processing unit are combined, and the combined signal is a video signal. And a signal synthesis unit that outputs as
The controller is
Based on each color component signal generated by the visible light signal generation unit and the infrared light signal generated by the infrared light signal generation unit, an exposure control signal for controlling an exposure condition in the imaging unit An exposure control unit that outputs to the imaging unit;
Based on each color component signal generated by the visible light signal generation unit and the infrared light signal generated by the infrared light signal generation unit, the signal amount of the red component light with respect to the signal amount of the visible light The first signal ratio, the second signal ratio of the green component light signal amount, the third signal ratio of the blue component light signal amount, and the infrared light relative to the total signal amount of the visible light and the infrared light A signal ratio calculator that calculates a fourth signal ratio of the signal amount of
Based on the first signal ratio, the second signal ratio, the third signal ratio, and the fourth signal ratio calculated by the signal ratio calculation unit, an infrared irradiation amount to the subject is calculated, and the calculation is performed. An infrared irradiation amount control unit that outputs the infrared irradiation amount to the infrared irradiation unit,
A light source for determining a light source of light radiated to the subject based on the first signal ratio, the second signal ratio, the third signal ratio, and the fourth signal ratio calculated by the signal ratio calculation unit A determination unit;
The first signal ratio, the second signal ratio, the third signal ratio and the fourth signal ratio calculated by the signal ratio calculation unit, the infrared irradiation amount calculated by the infrared irradiation amount control unit, Settings necessary for signal processing executed by the visible light signal processing unit, the infrared light signal processing unit, and the signal combining unit based on the arrangement position of the first optical filter and the determination result of the light source determination unit The imaging apparatus according to claim 3, further comprising: a signal processing set value calculation unit that calculates a parameter. The signal ratio calculation unit is configured to move the first optical filter to the first position and the second based on the infrared irradiation amount calculated by the infrared irradiation amount control unit and the determination result of the light source determination unit. The imaging apparatus according to claim 4, wherein it is determined in which position it is arranged, and a control signal corresponding to the determination result is output to the optical filter control unit as the predetermined control signal. The signal ratio calculation unit uses the first signal ratio, the second signal ratio, and the third signal ratio when the first optical filter is disposed at the first position. The imaging device according to claim 4, wherein an estimated value of the signal ratio is calculated. further,
When the first optical filter is disposed at the first position, the infrared light included in the incident light is received, and an electrical signal corresponding to the received infrared light is received in the signal ratio calculation unit. The imaging device according to claim 4, further comprising: an infrared light receiving unit that outputs to an infrared ray. The light source determination unit performs a determination process of the light source using predetermined determination data,
The signal ratio calculation unit calculates an estimated value of the fourth signal ratio based on the first signal ratio, the second signal ratio, the third signal ratio, and predetermined light source estimation data,
further,
The plurality of predetermined determination data and the plurality of predetermined light source estimation data set for each of a plurality of subject imaging environments are stored, and at least one of another determination data and another light source estimation data is externally stored. The imaging device according to claim 4, further comprising a storage unit that can be stored from the storage unit. further,
The imaging apparatus according to claim 4, further comprising a storage unit configured to store a plurality of the setting parameters set for each of the imaging environments of a plurality of subjects, and to store another setting parameter from the outside. further,
Visible light and infrared light are transmitted, near-infrared light having a wavelength closer to the visible light side than the wavelength band of the infrared light is blocked, and between the first position and the second position. A movable second optical filter;
The optical filter control unit arranges the first optical filter at one of the first position and the second position based on the predetermined control signal, and places the second optical filter in the first position. The imaging device according to claim 1, wherein the imaging device is disposed at the other of the second positions. A lens that forms an optical image by forming incident light from a subject, and an imaging device that can simultaneously receive visible light and infrared light, and that converts an optical signal incident through the lens into an electrical signal The visible light signal of an imaging device comprising: an imaging unit having: an optical filter that transmits only visible light; a visible light signal generation unit; an infrared light signal generation unit; a control unit; and an optical filter control unit. The generation unit calculates only a signal of a visible light component included in the electrical signal;
The infrared light signal generation unit calculates only an infrared light component signal included in the electrical signal;
The control unit outputs a predetermined control signal based on the signals calculated by the visible light signal generation unit and the infrared light signal generation unit;
The optical filter control unit disposing the optical filter at a first position on the optical path of the incident light or a second position other than the position on the optical path based on the predetermined control signal;
An imaging method including:

Images