JP2015126537A - Imaging device, method of controlling the same, and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of switching a video between a first mode and a second mode without generating disturbance in the video.SOLUTION: A video processing part 5 generates a first video signal, and a video output part 6 generates a second video signal. An imaging part 3, in a first mode, performs light exposure correspondingly to one frame time period of the second video signal, and in a second mode, performs light exposure on imaging conditions different from each other for each section obtained by dividing the one frame time period of the second video signal into a plurality of sections. The video processing part 5, in the first mode, generates a frame of the first video signal on the basis of an imaging signal read out in response to the light exposure corresponding to the one frame time period, and in the second mode, generates the frame of the first video signal on the basis of an imaging signal read out in response to the light exposure of each section. The video output part 6 uses the second video signal as a common signal system in the first mode and the second mode to continuously output the frame of the second video signal.

Description

  The present invention relates to an imaging apparatus, an imaging apparatus control method, and a control program.

Conventionally, in order to image a subject in an environment where there is almost no visible light such as at night, for example,
A method of irradiating a subject with infrared light using an infrared projector and imaging infrared light reflected from the subject is used. This method is an effective imaging method when a light that emits visible light cannot be used.

However, an image obtained by imaging a subject by this imaging method is a monochrome image.
In monochrome images, it may be difficult to identify an object. If a color image can be captured even in an environment where there is no visible light, the object identification can be improved. For example, in a surveillance camera, it is desired to capture a color image even in an environment without visible light in order to improve the object identification.

Patent Document 1 describes an imaging apparatus that can capture a color image even in an environment without visible light. In the imaging device described in Patent Document 1, an infrared projector is also used. If the technique described in Patent Document 1 is installed in the surveillance camera, it becomes possible to convert the subject into a color image and improve the object identification.

JP 2011-50049 A

It is desired that a single imaging apparatus can perform both normal imaging in an environment where visible light exists and night vision imaging using an infrared projector in an environment where visible light is low. The mode in which the imaging device performs normal imaging without using an infrared projector is the normal mode, and the mode in which the imaging device performs imaging in the state where infrared light is projected by the infrared projector is the infrared light projection mode I will call it.

When the imaging apparatus is equipped with two imaging modes of a normal mode and an infrared light projection mode, infrared light projection is performed from the normal mode (first mode) to the infrared light projection mode (second mode). Light mode (
It is preferable that the video is switched from the second mode) to the normal mode (first mode) without any video disturbance.

The first aspect of the present invention images a subject without projecting infrared light in an environment where visible light exists.
An imaging apparatus capable of switching between images without causing disturbance of the image, and a second mode in which an object is imaged while projecting infrared light in an environment with little visible light. It is an object to provide a control method and a control program.

The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue In the second mode in which each image is captured in the third section in which the third infrared light is projected, the imaging unit that images the subject, and the imaging signal output from the imaging unit An image output unit that generates and outputs a video signal of a predetermined signal system based on the exposure time in the first section when the second mode is switched to the first mode. It is characterized in that it is a time shorter than the maximum exposure time in one section To provide an image device.
The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue In the second mode in which each image is captured in the third section in which the third infrared light is projected, the imaging step for imaging the subject and the imaging signal output from the imaging unit A video output step of generating and outputting a video signal of a predetermined signal system based on the exposure time in the first section when switching from the second mode to the first mode is the second mode. Must be less than the maximum exposure time in one section A control method of an imaging apparatus according to claim.
The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue In the second mode in which each image is captured in the third section in which the third infrared light is projected, the imaging step for imaging the subject and the imaging signal output from the imaging unit An image output step of generating and outputting a video signal of a predetermined signal system based on the computer, and the exposure time in the first section when switching from the second mode to the first mode is the second time Less than the maximum exposure time in one section Providing a control program of an image pickup apparatus, characterized in that the a time.

According to the imaging apparatus, the imaging apparatus control method, and the control program of the present invention, the first mode in which an object is imaged without projecting infrared light in an environment in which visible light exists, and an environment in which visible light is low Under the second mode, the image can be switched to the second mode in which the subject is imaged while projecting the infrared light without causing any disturbance of the image.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to an embodiment. It is a figure which shows an example of the arrangement | sequence of the color filter in the color filter used for the imaging device of one Embodiment. It is a characteristic view which shows the spectral sensitivity characteristic of the wavelength of three primary color lights and relative sensitivity in the imaging part which comprises the imaging device of one Embodiment. It is a characteristic view showing the relationship between the wavelength and the relative detection rate when the reflectance of the three primary colors from a predetermined substance is multiplied by the light receiving sensitivity of silicon. It is a block diagram which shows the specific structural example of the image pick-up element 31 in FIG. It is a figure for demonstrating the operation | movement of exposure of the image pick-up element 31 shown in FIG. 5, and the read-out of an image pick-up signal. It is a block diagram which shows the specific structural example of the front signal process part 52 in FIG. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in normal mode. It is a figure which shows schematically the relationship between exposure and the flame | frame of a video signal when the imaging device of one Embodiment is operate | moving in intermediate | middle mode and night vision mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in intermediate | middle 1st mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in intermediate | middle 1st mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in intermediate | middle 2nd mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in intermediate | middle 2nd mode. It is a figure for demonstrating the addition process of the surrounding pixel when the imaging device of one Embodiment is operate | moving in night vision mode. It is a figure which shows the flame | frame in which the addition process of the surrounding pixel was performed. It is a figure for demonstrating the front signal process when the imaging device of one Embodiment is operate | moving in night vision 1st mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in night vision 1st mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in night vision 2nd mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in night vision 2nd mode. It is a figure for demonstrating the example of the mode switching in the imaging device of one Embodiment. It is a figure which shows the state of each part when the imaging device of one Embodiment is set to each mode. It is a partial block diagram which shows the 1st modification of the imaging device of one Embodiment. It is a partial block diagram which shows the 2nd modification of the imaging device of one Embodiment. It is a partial block diagram which shows the 3rd modification of the imaging device of one Embodiment. It is a flowchart which shows the video signal processing method performed with the imaging device of one Embodiment. It is a flowchart which shows the specific process of the normal mode shown to step S3 in FIG. It is a flowchart which shows the specific process of the intermediate | middle mode of step S4 in FIG. It is a flowchart which shows the specific process of the night vision mode of step S5 in FIG. It is a flowchart which shows the process which a video signal processing program memorize | stored in the imaging device of one Embodiment performs a computer. It is a figure which shows the sequence of the 1st example of the mode switching method between normal mode and infrared light projection mode. It is a figure which shows the sequence of the 1st example of the mode switching method between normal mode and infrared light projection mode. It is a figure which shows the sequence of the 2nd example of the mode switching method between normal mode and infrared light projection mode. It is a figure which shows the sequence of the 2nd example of the mode switching method between normal mode and infrared light projection mode. It is a figure which shows the sequence of the 3rd example of the mode switching method with normal mode and infrared light projection mode. It is a figure which shows the sequence of the 3rd example of the mode switching method with normal mode and infrared light projection mode.

Hereinafter, an imaging device, an imaging device control method, and a control program according to an embodiment will be described with reference to the accompanying drawings.

<Configuration of imaging device>
First, the overall configuration of an imaging apparatus according to an embodiment will be described with reference to FIG. An imaging apparatus according to an embodiment shown in FIG. 1 includes a normal mode suitable for an environment where there is sufficient visible light such as daytime, a night vision mode suitable for an environment where there is almost no visible light such as nighttime, and a visible mode. This is an imaging device capable of imaging in three modes, an intermediate mode suitable for an environment where light is slightly present.

Both the night vision mode and the intermediate mode are infrared light projection modes in which imaging is performed while projecting infrared light in an environment with little visible light. The infrared light projection mode may be only the night vision mode.
In this embodiment, as a preferable configuration, an imaging device capable of imaging in three modes including the intermediate mode is taken as an example.

In FIG. 1, the light indicated by the alternate long and short dash line reflected from the subject is collected by the optical lens 1. Here, the optical lens 1 has visible light in an environment in which visible light is sufficiently present, and infrared light in which an object reflects infrared light emitted from an infrared projector 9 described below in an environment with little visible light. Is incident.

Under an environment where there is a slight amount of visible light, the optical lens 1 is incident with light in which visible light and infrared light emitted from the infrared projector 9 are reflected by the subject.

Although only one optical lens 1 is shown in FIG. 1 for the sake of simplicity, the imaging apparatus actually includes a plurality of optical lenses.

An optical filter 2 is provided between the optical lens 1 and the imaging unit 3. The optical filter 2 has two parts, an infrared cut filter 21 and a dummy glass 22. The optical filter 2 is connected between the optical lens 1 and the imaging unit 3 by the drive unit 8.
1 and the state where the dummy glass 22 is inserted between the optical lens 1 and the imaging unit 3.

The imaging unit 3 includes an imaging element 31 in which a plurality of light receiving elements (pixels) are arranged in the horizontal direction and the vertical direction.
And a color filter 32 in which any one of red (R), green (G), and blue (B) color filters is arranged corresponding to each light receiving element. The image sensor 31 is a CCD (Charge
Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) may be used.

As an example, as shown in FIG. 2, the color filter 32 has R, G, and B color filters arranged in an array called a Bayer array. The Bayer array is an example of a predetermined array of R, G, and B color filters. In FIG. 2, the G color filter sandwiched between the R color filters in each row is Gr, and the G color filter sandwiched between the B color filters is Gb.

In the Bayer array, a horizontal row in which R color filters and Gr color filters are alternately arranged and a horizontal row in which B color filters and Gb color filters are alternately arranged are vertical. They are arranged alternately in the direction.

FIG. 3 shows spectral sensitivity characteristics of the wavelengths of R light, G light, and B light and relative sensitivity in the imaging unit 3. The relative sensitivity is normalized to 1 at the maximum value. When the imaging apparatus is operated in the normal mode, it is necessary to cut infrared light having a wavelength of 700 nm or more in order to capture a good color image by visible light.

Therefore, the drive unit 8 drives the optical filter 2 so that the infrared cut filter 21 is inserted between the optical lens 1 and the imaging unit 3 based on control by the control unit 7.

As can be seen from FIG. 3, the imaging unit 3 has sensitivity even in an infrared light region having a wavelength of 700 nm or more. Therefore, when the imaging device is operated in the intermediate mode or the night vision mode, the driving unit 8 removes the infrared cut filter 21 between the optical lens 1 and the imaging unit 3 based on the control by the control unit 7 and the dummy glass. The optical filter 2 is driven so that 22 is inserted.

In a state where the dummy glass 22 is inserted between the optical lens 1 and the imaging unit 3, the wavelength is 700n.
Infrared light of m or more is not cut. Therefore, the imaging apparatus can obtain R, G, and B color information by using the sensitivity of the portion surrounded by the dashed ellipse in FIG. The reason why the dummy glass 22 is inserted is to make the optical path length the same as the optical path length when the infrared cut filter 21 is inserted.

The infrared projector 9 includes projectors 91, 92, and 93 that project infrared light having wavelengths IR1, IR2, and IR3, respectively. In the intermediate mode or the night vision mode, the light projecting control unit 71 in the control unit 7 performs control so as to selectively project infrared light with wavelengths IR1 to IR3 from the light projecting units 91 to 93 in a time division manner. .

Incidentally, a silicon wafer is used for the image sensor 31. FIG. 4 shows the relationship between the wavelength and the relative detection rate when the reflectance at each wavelength is multiplied by the light receiving sensitivity of silicon when a material exhibiting each color of R, G, and B is irradiated with white light. Yes. Also in FIG. 4, the maximum value of the relative detection rate is normalized to 1.

As shown in FIG. 4, in the infrared light region, for example, the reflected light at a wavelength of 780 nm is highly correlated with the reflected light of a material exhibiting an R color, and the reflected light at a wavelength of 870 nm is a reflection of a material exhibiting a B color. Correlation with light is high, and reflected light at a wavelength of 940 nm is highly correlated with reflected light of a material exhibiting G color.

Therefore, in the present embodiment, the wavelength IR of the infrared light that is projected by the light projecting units 91, 92, and 93.
1, IR2 and IR3 are set to 780 nm, 940 nm, and 870 nm. These wavelengths are
It is an example of the wavelengths IR1 to IR3, and may be other than 780 nm, 940 nm, and 870 nm.

The light projecting unit 91 irradiates the subject with infrared light having the wavelength IR1, and assigns an image signal obtained by imaging the light reflected from the subject to the R signal. The light projecting unit 93 irradiates the subject with infrared light having a wavelength IR2, and assigns a video signal obtained by imaging the light reflected from the subject to the G signal. The light projecting unit 92 has a wavelength IR3.
The infrared signal is irradiated onto the subject, and a video signal obtained by imaging the light reflected from the subject is assigned to the B signal.

In this way, in principle, even in the intermediate mode or the night vision mode, it is possible to reproduce the same color as when the subject is imaged in an environment where visible light exists in the normal mode.

Although the color image is different from the actual color of the subject, the wavelength IR1 at 780 nm
May be assigned to the R light, the wavelength IR3 of 870 nm to the G light, and the wavelength IR2 of 940 nm to the B light. The wavelengths IR1, IR2, and IR3 can be arbitrarily assigned to the R light, G light, and B light.

In the present embodiment, the wavelengths IR1, IR2, IR that best reproduce the color of the subject.
3 is assigned to R light, G light, and B light, respectively.

The control unit 7 controls imaging in the imaging unit 3, each unit in the video processing unit 5, and the video output unit 6.

Taking a case where the image sensor 31 is a CCD as an example, a schematic configuration of the image sensor 31 and how the controller 7 controls the image sensor 3 will be described.

As shown in FIG. 5, the image sensor 31 includes a plurality of light receiving elements P arranged in the horizontal direction and the vertical direction.
x. The imaging device 31 includes a vertical transfer register 3vr corresponding to each column of the light receiving elements Px arranged in the vertical direction, and a horizontal transfer register 3hr connected to each vertical transfer register 3vr.

The R, G, and B color filters of the color filter 32 described in FIG. 2 are associated with each of the plurality of light receiving elements Px arranged in the horizontal direction and the vertical direction. Actually, as shown in FIG. 5, the light receiving elements Px adjacent to each other are arranged in a state separated from each other in the horizontal direction and the vertical direction, but in FIG. It is shown in the state.

6A shows the exposure of the light receiving element Px of the image sensor 31. FIG. Light receiving element P
Let the maximum exposure time of x be tExmax. The maximum exposure time tExmax corresponds to one frame period.
The exposure time of the light receiving element Px increases or decreases according to the shutter speed, with the maximum exposure time tExmax being the maximum.

For example, a sampling pulse Ps1 for discharging charges accumulated by exposure at intervals of one horizontal period is supplied to the imaging element 31. The control unit 7 extracts the sampling pulse P at a predetermined timing.
When the supply of s1 is stopped, the exposure Ex1, Ex2... of the time indicated by hatching after the supply of the extraction pulse Ps1 is stopped.

The image sensor 31 has a readout pulse Ps2 at the timing when the maximum exposure time tExmax is reached.
Is supplied. When the readout pulse Ps2 is supplied to the image sensor 31, the charges accumulated in the light receiving elements Px in the respective columns in FIG. 5 are collectively transferred to the vertical transfer register 3vr.

The vertical transfer register 3vr sequentially transfers the charges transferred from the light receiving element Px to the horizontal transfer register 3hr by a vertical transfer clock. The horizontal transfer register 3hr sequentially transfers the charges transferred from the respective vertical transfer registers 3vr by a horizontal transfer clock.
The charges transferred by the horizontal transfer register 3hr are amplified by the output amplifier 3ap, converted into a voltage signal, and output.

In FIG. 6, as shown in FIG. 6B, the image sensor 31 reads out the electric charge obtained by the exposure Ex1, Ex2,... As a voltage signal over one frame period after the read pulse Ps2 is supplied. The frames F1, F2,... Of the imaging signal are output. The imaging signals of the frames F1, F2,... Are supplied to the A / D converter 4.

The imaging signal input to the A / D converter 4 is A / D converted and input to the video processing unit 5.
The imaging unit 3 and the A / D converter 4 may be integrated.

The control unit 7 is a writing / reading control unit 7 that controls writing of video data to a frame buffer 50 provided in the video processing unit 5 and reading of video data from the frame buffer 50.
A mode switching unit 72 that switches between 0, normal mode, intermediate mode, and night-vision mode is provided.
How the video data is written and read by the write / read control unit 70 will be described later.

The mode switching unit 72 appropriately switches the operation in the video processing unit 5 as described later in correspondence with the normal mode, the intermediate mode, and the night vision mode. The video processing unit 5 and the control unit 7 may be integrated.

The video processing unit 5 includes a frame buffer 50, switches 51 and 53, and a previous signal processing unit 52.
And a demosaic processing unit 54. The frame buffer 50 includes memories 50a to 50f each having a capacity of one frame. As indicated by the broken line, the frame buffer 50 is
In addition to the memories 50a to 50f having a total capacity of 6 frames, there may be memories 50g to 50i each having a capacity of 1 frame.

The switches 51 and 53 may be physical switches, or may be conceptual switches for switching between operation and non-operation of the previous signal processing unit 52. A video signal is input from the video processing unit 5 to the control unit 7 in order to detect the brightness of the video being captured.

As illustrated in FIG. 7, the previous signal processing unit 52 includes a surrounding pixel addition unit 521, a same-position pixel addition unit 522, and a synthesis unit 523.

The video processing unit 5 generates R, G, and B primary color data and supplies them to the video output unit 6. The video output unit 6 outputs the three primary color data in a predetermined format to a display unit not shown.

The video output unit 6 may output the R, G, and B signals as they are, or may convert the R, G, and B signals into luminance signals and color signals (or color difference signals) and output them. The video output unit 6 may output a composite video signal. The video output unit 6 may output a digital video signal or may output a video signal converted into an analog signal by a D / A converter.

In this embodiment, the video output unit 6 includes an NTSC encoder 61 and outputs an NTSC interlace video signal based on control by the control unit 7. The video output unit 6 may include a PAL encoder for generating a PAL video signal instead of the NTSC encoder 61. HDTV (High
-definition television) system, and the video output unit 6 can be applied to various systems.

Hereinafter, specific operations in the normal mode, the intermediate mode, and the night vision mode will be described.

<Normal mode>
In the normal mode, the control unit 7 causes the driving unit 8 to insert the infrared cut filter 21 between the optical lens 1 and the imaging unit 3. The light projection control unit 71 turns off infrared light projection by the infrared projector 9.

The image signal picked up by the image pickup unit 3 is converted into video data which is a digital signal by the A / D converter 4 and input to the video processing unit 5. In the normal mode, the mode switching unit 7
2 controls the switches 51 and 53 to be connected to the terminal Tb.

As described in FIG. 6, the frame F1 of the video signal is obtained based on the exposure Ex1.
A frame F2 of the video signal is obtained based on the exposure Ex2. The same applies to the subsequent exposure. The frame frequency of the video signal is set to 30 frames / second, for example.

The frame frequency of the video signal may be set as appropriate, such as 30 frames / second or 60 frames / second for the NTSC system and 25 frames / second or 50 frames / second for the PAL system. The frame frequency of the video signal is 24 frames /
It may be seconds.

The video data of each frame output from the A / D converter 4 is temporarily held in the frame buffer 50. The video data read from the frame buffer 50 is transferred to the switch 51.
, 53 to the demosaic processing unit 54. The demosaic processing unit 54 performs demosaic processing on the input video data of each frame. The video processing unit 5 performs various video processing such as white balance correction and gain correction in addition to demosaic processing, and outputs R, G, and B primary color data.

The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG.
(A) shows an arbitrary frame Fm of video data. The frame Fm is a frame composed of pixels in the effective video period. The number of pixels of the video data is, for example, V
In the GA standard, there are 640 pixels horizontally and 480 pixels vertically. Here, for simplification, the number of pixels of the frame Fm is significantly reduced, and the frame Fm is conceptually illustrated.

The video data generated using the Bayer-array imaging unit 3 is R, G,
This data is a mixture of B pixel data. The demosaic processing unit 54 generates R interpolation pixel data Ri obtained by calculating R pixel data at a pixel position where no R pixel data is present using surrounding R pixel data. The demosaic processing unit 54 generates an R frame FmR in which all the pixels in one frame shown in FIG. 8B are made up of R pixel data.

The demosaic processing unit 54 generates G interpolation pixel data Gi obtained by calculating G pixel data at a pixel position where no G pixel data exists using surrounding G pixel data. The demosaic processing unit 54 generates a G frame FmG in which all pixels of one frame shown in FIG.

The demosaic processing unit 54 generates B interpolated pixel data Bi obtained by calculating B pixel data at a pixel position where no B pixel data exists using surrounding B pixel data. The demosaic processing unit 54 generates a B frame FmB in which all the pixels in one frame shown in FIG.

The demosaic processing unit 54 may use at least R pixel data when interpolating R pixel data, may use at least G pixel data when interpolating G pixel data, and B pixel data. When interpolation is performed, at least B pixel data may be used. In order to improve interpolation accuracy, the demosaic processing unit 54 may use pixel data of another color different from the color of the interpolation pixel data to be generated when interpolating R, G, and B pixel data. Good.

Since there are pixels outside the effective video period in the imaging unit 3, R, G, and B pixel data can be interpolated even in pixels located at the upper, lower, left, and right ends of the frame Fm.

The R frame FmR, G frame FmG, and B frame FmB generated by the demosaic processing unit 54 are output as R, G, and B primary color data. In FIG. 8, the R, G, B pixel data is described in units of frames in order to facilitate understanding, but actually, the R, G, B pixel data is sequentially output for each pixel.

<Intermediate mode: Intermediate first mode>
In the intermediate mode (intermediate first mode and intermediate second mode described later), the control unit 7 causes the drive unit 8 to insert the dummy glass 22 between the optical lens 1 and the imaging unit 3. The light projection controller 71 turns on the infrared light projection by the infrared light projector 9. The mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Ta.

FIG. 9A shows a state of infrared light projection by the infrared projector 9. The control unit 7 divides one frame period of the normal mode into 1/3, and controls so as to project infrared light in the order of the light projecting units 91, 92, 93, for example.

In the example shown in FIG. 9A, in the first 1/3 period of one frame, the wavelength IR1 (78
0 nm) infrared light is irradiated on the subject. In the next 1/3 period of one frame, the wavelength IR
2 (940 nm) infrared light is irradiated on the subject. In the last 1/3 period of one frame, the subject is irradiated with infrared light having a wavelength IR3 (870 nm). The order of projecting infrared light with wavelengths IR1 to IR3 is arbitrary.

As shown in FIG. 9B, at the timing when infrared light having a wavelength IR1 is projected, the imaging unit 3 performs exposure Ex1R having a high correlation with the R light. At the timing when the infrared light having the wavelength IR2 is projected, the imaging unit 3 performs the exposure Ex1G having a high correlation with the G light. At the timing when the infrared light having the wavelength IR3 is projected, the imaging unit 3 performs exposure Ex1B having a high correlation with the B light.

However, in the intermediate mode, since imaging is performed in an environment where there is a small amount of visible light, visible light and infrared light projected from the infrared projector 9 are mixed. Therefore, in the intermediate mode, the exposures Ex1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B... Are exposures that combine exposure with visible light and exposure with infrared light.

As shown in FIG. 9 (c), the exposure Ex1R, Ex1G, and Ex1B are used to determine the exposure Ex
A frame F1IR1 corresponding to 1R, a frame F1IR3 corresponding to exposure Ex1G, and a frame F1IR2 corresponding to exposure Ex1B are obtained.

Also, based on the exposure Ex2R, Ex2G, Ex2B, a frame corresponding to the exposure Ex2R after a predetermined time
A frame F2IR3 corresponding to F2IR1, exposure Ex2G, and a frame F2IR2 corresponding to exposure Ex2B are obtained. The same applies to exposures Ex3R, Ex3G, and Ex3B.

The frame frequency of the imaging signal in (c) of FIG. 9 is 90 frames / second. In the intermediate mode, one frame of the video signal in the normal mode is time-divided and infrared light of wavelengths IR1 to IR3 is projected. In order to output a video signal in the same format as in the normal mode, ( The frame frequency of the imaging signal in c) is three times the frame frequency in the normal mode.

As will be described later, one frame of a video signal having a frame frequency of 30 frames / second shown in (d) of FIG. 9 is generated based on the imaging signals of three frames shown in (c) of FIG. Here, the description will be made as a progressive video signal rather than an interlace video signal. For example, a frame F1IR is generated based on the frames F1IR1, F1IR2, and F1IR3, and a frame F2IR is generated based on the frames F2IR1, F2IR2, and F2IR3.

The operation in the intermediate mode for generating the video signal of each frame of FIG. 9D based on the image pickup signals of 3 frames of FIG. 9C will be specifically described.

The video data of each frame corresponding to the imaging signal shown in FIG. 9C output from the A / D converter 4 is temporarily held in the frame buffer 50. The video data read from the frame buffer 50 is input to the previous signal processing unit 52 via the switch 51.

The previous signal processing in the previous signal processing unit 52 will be described with reference to FIG. (
a) shows an arbitrary frame FmIR1 of the video data generated at the timing when the infrared light having the wavelength IR1 is projected. The pixel data of R, B, Gr, and Gb in the frame FmIR1 are
A subscript 1 is attached to indicate that the infrared light having the wavelength IR1 is generated in a projected state.

FIG. 10B shows an arbitrary frame FmIR2 of video data generated at the timing when infrared light having a wavelength IR2 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR2 is attached with a subscript 2 indicating that it is generated in a state where infrared light having a wavelength IR2 is projected.

FIG. 10C shows an arbitrary frame FmIR3 of the video data generated at the timing when the infrared light having the wavelength IR3 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR3 has a subscript 3 indicating that it is generated in a state where infrared light having the wavelength IR3 is projected.

Since the frame FmIR1 shown in FIG. 10A is video data generated in a state where infrared light having a wavelength IR1 having a high correlation with the R light is projected, the R pixel data is projected. The pixel data corresponding to the infrared light and the pixel data B and G are pixel data not corresponding to the projected infrared light. The hatching attached to the B, Gr, and Gb pixel data means that the pixel data does not correspond to the projected infrared light.

The frame FmIR2 shown in FIG. 10B is video data generated in a state where infrared light having a wavelength IR2 having a high correlation with the G light is projected, so that the G pixel data is projected. The pixel data corresponding to the infrared light, and the R and B pixel data are pixel data not corresponding to the projected infrared light. The hatching attached to the R and B pixel data means that the pixel data does not correspond to the projected infrared light.

The frame FmIR3 shown in FIG. 10C is video data generated in a state where infrared light having a wavelength IR3 having a high correlation with the B light is projected, so that the B pixel data is projected. The pixel data corresponding to the infrared light, and the R and G pixel data are pixel data not corresponding to the projected infrared light. The hatching attached to the R, Gr, and Gb pixel data means pixel data that does not correspond to the projected infrared light.

The same-position pixel addition unit 522 in the previous signal processing unit 52 has R, Gr,
Gb and B pixel data are individually added according to the following equations (1) to (3) to obtain pixel data R1.
23, Gr123, Gb123, B123 are generated. In the intermediate mode, the surrounding pixel addition unit 521 in the previous signal processing unit 52 does not operate.

R123 = ka × R1 + kb × R2 + kc × R3 (1)
G123 = kd × G1 + ke × G2 + kf × G3 (2)
B123 = kg × B1 + kh × B2 + ki × B3 (3)

In the equations (1) to (3), R1, G1, and B1 are R, G, and B pixel data in the frame FmIR1, R2, G2, and B2 are R, G, and B pixel data in the frame FmIR2, R3, G3, B3 is R, G, B pixel data in the frame FmIR3. ka to ki are predetermined coefficients. Formula (2
G123 in) is Gr123 or Gb123.

At this time, the same-position pixel addition unit 522 does not include hatched R, Gr, Gb,
The pixel data of R, Gr, Gb, and B at the same pixel position with hatching are added to the pixel data of B, respectively.

That is, the same-position pixel adding unit 522 calculates R in the frame FmIR1 based on the equation (1).
The pixel data R123 is generated by adding the R pixel data at the same pixel position in the frames FmIR2 and FmIR3 to the pixel data. That is, red pixel data R123 is generated using only pixel data in a region corresponding to the red color filter in the light receiving element.

The same-position pixel adding unit 522 calculates Gr, G in the frame FmIR2 based on Expression (2).
The pixel data G123 is generated by adding the pixel data Gr and Gb at the same pixel position in the frames FmIR1 and FmIR3 to the pixel data b. That is, green pixel data G123 is generated using only pixel data in a region corresponding to the green color filter in the light receiving element.

The same-position pixel addition unit 522 generates pixel data B123 by adding the B pixel data at the same pixel position in the frames FmIR1 and FmIR2 to the B pixel data in the frame FmIR3 based on the equation (3). That is, blue pixel data B123 is generated using only pixel data in a region corresponding to the blue color filter in the light receiving element.

Based on the pixel data R123, Gr123, Gb123, and B123 generated at each pixel position, the combining unit 523 in the previous signal processing unit 52 generates a frame FmIR123 of the combined video signal shown in (d) of FIG. .

Specifically, the synthesis unit 523 generates the pixel data R123 in the frame FmIR1 and the frame FmI.
The pixel data Gr123 and Gb123 in R2 and the pixel data B123 in the frame FmIR3 are selected and combined. Thereby, the synthesis unit 523 generates a frame FmIR123 of the synthesized video signal.

As described above, the synthesis unit 523 generates the frame FmIR123 in which the pixel data R123, Gr123, Gb123, and B123 are arranged so as to have the same arrangement as the arrangement of the color filters in the color filter 32.

In the intermediate first mode, video data of the frame FmIR123 is generated using pixel data that is not hatched and pixel data that is hatched.

The same-position pixel adding unit 522 adds pixel data at the same pixel position to each other.
For the following reason. Since the imaging is performed in an environment where visible light is present in the intermediate mode, the hatched pixel data includes respective color components based on exposure with visible light. Therefore, the sensitivity of each color can be increased by adding pixel data at the same pixel position.

If the visible light and infrared light are mixed and there is a relatively large amount of visible light, exposure with visible light becomes dominant. In this case, the video data of the frame FmIR123 mainly includes components based on the video signal exposed with visible light. If visible light and infrared light are mixed and there is a relatively large amount of infrared light, exposure with infrared light becomes dominant. In this case, the video data of the frame FmIR123 mainly includes components based on the video signal exposed with infrared light.

When the visible light is relatively small, the relationship between the coefficients ka, kb, and kc in equation (1) is expressed as ka
> kb, kc, and in equation (2), the relationship between the coefficients kd, ke, kf is kf> kd, ke, and equation (3
), The magnitude relationship between the coefficients kg, kh and ki should be kh> kg and ki. This is the wavelength I
This is because R1 has a high correlation with R light, wavelength IR2 has a high correlation with G light, and wavelength IR3 has a high correlation with B light.

In this way, the R pixel data is R pixel data in the frame FmIR1, the G pixel data is G pixel data in the frame FmIR2, and the B pixel data is frame FmI.
The pixel data of B in R3 can be mainly used.

The video data of the frame FmIR123 output from the previous signal processing unit 52 is input to the demosaic processing unit 54 via the switch 53. The demosaic processing unit 54 performs demosaic processing on the video data of the input frame FmIR123, as in the normal mode. The video processor 5
In addition to demosaic processing, various video processing such as white balance correction and gain correction are performed,
Outputs the three primary color data of R, G and B.

The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 11A shows the frame FmIR123. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data,
R interpolation pixel data R123i is generated. The demosaic processing unit 54 generates an R frame FmIR123R in which all the pixels of one frame shown in FIG.

The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding G pixel data, and generates the G interpolation pixel data G123i. The demosaic processing unit 54 uses G pixel data in which all pixels in one frame shown in FIG.
A frame FmIR123G is generated.

The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B123i. In the demosaic processing unit 54, all pixels in one frame shown in (d) of FIG.
A frame FmIR123B is generated.

As can be understood by comparing the operation of the demosaic processing unit 54 shown in FIG. 8 in the normal mode with the operation of the demosaic processing unit 54 shown in FIG. 11 in the intermediate mode, they are substantially the same. The operation of the demosaic processing unit 54 does not change regardless of whether it is the normal mode or the intermediate mode.

In the normal mode, the front signal processing unit 52 is disabled, and in the intermediate mode, the surrounding pixel adding unit 52 is operated.
Except for 1, the previous signal processing unit 52 may be operated. In the normal mode and the intermediate mode, the demosaic processing unit 54 in the video processing unit 5 and the signal processing unit such as white balance correction and gain correction can be shared.

<Intermediate mode: Intermediate second mode>
The operation in the intermediate second mode will be described with reference to FIGS. In the operation in the intermediate second mode, the description of the same part as the operation in the intermediate first mode is omitted. FIG.
Frames FmIR1, FmIR2, and FmIR3 of (a) to (c) of 2 are the same as the frames FmIR1, FmIR2, and FmIR3 of (a) to (c) of FIG.

The synthesizer 523 has R1 which is R pixel data in the frame FmIR1, Gr2 and Gb2 which are G pixel data in the frame FmIR2, and B which is B pixel data in the frame FmIR3.
Select 3 and synthesize. As a result, the synthesis unit 523 generates a frame FmIR123 ′ of the synthesized video signal shown in FIG.

That is, the frame FmIR123 ′ is video data obtained by collecting R, Gr, Gb, and B pixel data not hatched in the frames FmIR1, FmIR3, and FmIR2 into one frame.

That is, in the frame FmIR123 ′, the pixel data for red using only the pixel data in the region corresponding to the red color filter in the state where the infrared light having the wavelength IR1 is projected, and the infrared light having the wavelength IR2 are projected. Only the pixel data for the green color using only the pixel data of the region corresponding to the green color filter in the selected state, and the pixel data of the region corresponding to the blue color filter in the state of projecting the infrared light having the wavelength IR3 are used. The pixel data is for blue.

In this way, the synthesis unit 523 generates a frame FmIR123 ′ in which the pixel data R1, Gr2, Gb2, and B3 are arranged so as to be the same arrangement as the arrangement of the color filters in the color filter 32.

In the intermediate second mode, the same-position pixel adding unit 522 sets the coefficient ka in Equation (1) to 1, the coefficients kb and kc to 0, the coefficient ke in Equation (2) to 1, the coefficients kd and kf to 0, In equation (3), the coefficient ki is 1 and the coefficients kg and kh are 0.

As a result, R pixel data in the frame FmIR1 and Gr in the frame FmIR2 are obtained.
, Gb pixel data and the B pixel data in the frame FmIR3 have the same values.

Therefore, the synthesizing unit 523 performs the R pixel data in the frame FmIR1, the Gr and Gb pixel data in the frame FmIR2, and the frame FmI, as in the operation in the intermediate first mode.
If the B pixel data in R3 is selected, the frame FmIR123 ′ can be generated.

In the intermediate second mode, the pre-signal processing unit 52 generates pixel data (hatched) in a state in which infrared light for generating pixel data having the same color as the pixel data is projected. Video data of the frame FmIR123 ′ is generated using only (non-pixel data).

According to the intermediate second mode, although sensitivity and color reproducibility are lower than in the intermediate first mode,
The arithmetic processing can be simplified and the frame memory can be reduced.

The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 13A shows a frame FmIR123 ′. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R1i. The demosaic processing unit 54 generates an R frame FmIR123′R in which all the pixels of one frame shown in FIG. 13B are made up of R pixel data.

The demosaic processing unit 54 calculates G pixel data at a pixel position where no G pixel data exists using the surrounding G pixel data, and generates G interpolation pixel data G2i. The demosaic processing unit 54 uses G pixel data in which all the pixels of one frame shown in FIG.
A frame FmIR123'G is generated.

The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B3i. The demosaic processing unit 54 uses the B pixel data in which all the pixels in one frame shown in FIG.
A frame FmIR123'B is generated.

As described above, in the intermediate mode, the pixel data for red is generated from the pixel data obtained from the region corresponding to the red color filter in the light receiving element, and obtained from the region corresponding to the green color filter in the light receiving element. Green pixel data is generated from the pixel data, and blue pixel data is generated from the pixel data obtained from the region corresponding to the blue color filter in the light receiving element.

<Night vision mode: first night vision mode>
In the night vision mode (first night vision mode and second night vision mode described later), as in the intermediate mode,
The control unit 7 causes the drive unit 8 to insert the dummy glass 22 between the optical lens 1 and the imaging unit 3. The light projection controller 71 turns on the infrared light projection by the infrared light projector 9. The mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Ta.

The schematic operation in the night vision mode is the same as that in FIG. However, in the night vision mode, since the imaging is performed in an environment where there is almost no visible light, the exposures Ex1R and Ex1B in FIG.
1G, Ex1B, Ex2R, Ex2G, Ex2B... Assume exposure using only infrared light.

In an environment where there is almost no visible light and only infrared light is present, the color filter 32
Since there is no difference in the characteristics of the respective color filters, the imaging unit 3 can be regarded as a monochrome imaging device.

Therefore, in the night vision mode, the surrounding pixel addition unit 521 in the previous signal processing unit 52 adds pixel data located in the periphery to each pixel data in order to improve the sensitivity of infrared light.

Specifically, as illustrated in FIG. 14A, when the R pixel is the target pixel, the surrounding pixel adding unit 521 includes the G and B positioned around the R pixel data of the target pixel. The pixel data of 8 pixels are added.

In other words, in the intermediate mode, red pixel data was generated from the pixel data obtained from the region corresponding to the red color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. Pixel data for red is generated from the pixel data obtained from the above. In the example of FIG. 14, pixel data obtained from an area corresponding to nine pixels including the target pixel is used for each color.

As shown in FIG. 14B, when the G pixel is the target pixel, the surrounding pixel adder 521
Adds the pixel data of G and B pixels located in the periphery to the G pixel data of the target pixel. The G pixel in FIG. 14B is a Gr or Gb pixel.

In other words, in the intermediate mode, the pixel data for green was generated from the pixel data obtained from the area corresponding to the green color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. The pixel data for green is generated from the pixel data obtained from the above.

As shown in FIG. 14C, when the pixel B is the target pixel, the surrounding pixel adding unit 521
Adds the pixel data of 8 pixels of R and G located around the B pixel data of the target pixel.

In other words, in the intermediate mode, blue pixel data was generated from the pixel data obtained from the region corresponding to the blue color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. The pixel data for blue is generated from the pixel data obtained from the above.

The surrounding pixel adding unit 521 may simply add 9 pixels of the pixel data of the target pixel and the surrounding 8 pixel pixel data, or may apply a predetermined weight to the surrounding 8 pixel pixel data. May be added to the pixel data of the pixel of interest.

By the way, there is an image pickup device that can read out a plurality of pixels called binning as a single pixel. When an image sensor having a binning function is used as the image sensor 31, an addition process by an image sensor having a binning function may be performed instead of the addition process by the surrounding pixel addition unit 521. Binning by the image sensor is substantially equivalent to addition processing by the surrounding pixel addition unit 521.

Frames FmIR1, FmIR3, and FmIR2 of (a) to (c) of FIG. 15 are represented by (a) to (c) of FIG.
This is the same as the frames FmIR1, FmIR3, and FmIR2. In (d) to (f) of FIG. 15, R1ad
, Gr1ad, Gb1ad, B1ad, R2ad, Gr2ad, Gb2ad, B2ad, R3ad, Gr3ad, Gb3ad, and B3ad are obtained by adding the surrounding pixel data of R, Gr, Gb, and B to the pixel data of R, Gr, Gb, and B, respectively. Pixel data.

The surrounding pixel addition unit 521 performs the addition process shown in FIG. 14 on the respective pixel data of the frames FmIR1, FmIR3, and FmIR2, so that the frames Fm shown in (d) to (f) of FIG.
Generate IR1ad, FmIR2ad, and FmIR3ad.

The frames FmIR1ad, FmIR2ad, and FmIR3ad in FIGS. 16A to 16C are shown in FIGS.
This is the same as the frames FmIR1ad, FmIR2ad, and FmIR3ad in (f).

The same-position pixel addition unit 522 adds the pixel data of R2ad and R3ad at the same pixel position in the frames FmIR2ad and FmIR3ad to the pixel data of R1ad in the frame FmIR1ad based on Expression (1), as in the first intermediate mode. Thus, pixel data R123ad is generated.

The same-position pixel addition unit 522 calculates Gr2ad in the frame FmIR2ad based on the equation (2),
Gb2ad pixel data includes Gr1ad and Gb1ad at the same pixel position in frames FmIR1ad and FmIR3ad
, Gr3ad and Gb3ad are added to generate pixel data Gr123ad and Gb123ad.

Based on Expression (3), the same-position pixel addition unit 522 adds the pixel data of B1ad and B2ad at the same pixel position in the frames FmIR1ad and FmIR2ad to the pixel data of B3ad in the frame FmIR3ad to generate pixel data B123ad To do.

The synthesizer 523 performs pixel data R123a in the frame FmIR1ad as in the first intermediate mode.
The pixel data Gr123ad and Gb123ad in the frame FmIR2ad and the pixel data B123ad in the frame FmIR3ad are selected and combined. As a result, the combining unit 523 displays the (d
The frame FmIR123ad of the synthesized video signal shown in FIG.

The combining unit 523 generates a frame FmIR123ad in which the pixel data R123ad, Gr123ad, Gb123ad, and B123ad are arranged so as to be the same arrangement as the arrangement of the color filters in the color filter 32.

FIG. 17A shows a frame FmIR123ad. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R123adi. The demosaic processing unit 54 is shown in FIG.
The R frame FmIR123adR in which all the pixels in one frame are made up of R pixel data is generated.

The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding G pixel data, and generates the G interpolation pixel data G123adi. The demosaic processing unit 54 generates a G frame FmIR123adG in which all the pixels in one frame shown in FIG.

The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B123adi. The demosaic processing unit 54 generates a B frame FmIR123adB in which all the pixels of one frame shown in FIG.

The first intermediate mode is different from the first night-vision mode in that the former does not operate the surrounding pixel adding unit 521 while the latter operates the surrounding pixel adding unit 521. Mode switching unit 72
In the night vision mode, the surrounding pixel adding unit 521 may be operated.

The operation of the demosaic processing unit 54 in the night vision mode is substantially the same as the operation of the demosaic processing unit 54 in the normal mode and the intermediate mode. In the normal mode, the intermediate mode, and the night vision mode, the demosaic processing unit 54 in the video processing unit 5 and a signal processing unit such as white balance correction and gain correction can be shared.

<Night Vision Mode: Night Vision Second Mode>
The operation in the night vision second mode will be described with reference to FIGS. In the operation in the night vision second mode, the description of the same part as the operation in the night vision first mode is omitted. FIG.
The frames FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG. 8 are shown in (a) to (c) of FIG.
This is the same as FmIR1ad, FmIR2ad, and FmIR3ad.

The synthesizer 523 generates R1ad, which is R pixel data in the frame FmIR1ad, and the frame FmI.
Gr2ad and Gb2ad which are G pixel data in R2 and B3ad which is B pixel data in the frame FmIR3 are selected and combined. Thereby, the synthesis unit 523 generates a frame FmIR123′ad of the synthesized video signal shown in FIG.

The synthesizing unit 523 generates a frame FmIR123′ad in which the pixel data R1ad, Gr2ad, Gb2ad, and B3ad are arranged so that the arrangement is the same as the arrangement of the color filters in the color filter 32.

As described with reference to FIG. 14, the pixel data R1ad for red in the frame FmIR123'ad is obtained from a wider area than the area used for generating the pixel data for red in the intermediate mode. It is generated from pixel data.

Further, the pixel data Gr2ad for green in the frame FmIR123'ad is generated from pixel data obtained from an area wider than the area used for generating the pixel data for green in the intermediate mode. Yes.

Further, the blue pixel data B3ad in the frame FmIR123'ad is generated from pixel data obtained from an area larger than the area used to generate the blue pixel data in the intermediate mode. Yes.

In the second night vision mode, the same-position pixel addition unit 522 performs the equation (1) as in the second intermediate mode.
) Is set to 1, the coefficients kb and kc are set to 0, the coefficient ke in equation (2) is set to 1, and the coefficients kd and kf
Is 0, the coefficient ki in equation (3) is 1, and the coefficients kg and kh are 0.

Thereby, the pixel data of R1ad in the frame FmIR1ad, the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad, and the pixel data of B3ad in the frame FmIR3ad have the same values.

Therefore, the synthesizing unit 523 selects the pixel data of R1ad in the frame FmIR1ad, the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad, and the pixel data of B3ad in the frame FmIR3ad, as in the operation in the first night vision mode. A frame FmIR123'ad can be generated.

The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 19A shows a frame FmIR123′ad. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R1ad pixel data, and generates the R interpolation pixel data R1adi. The demosaic processing unit 54 is shown in FIG.
The R frame FmIR123′adR in which all the pixels of one frame shown in FIG.

The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding Gr2ad and Gb2ad pixel data, and generates the G interpolation pixel data G2adi. The demosaic processing unit 54 performs interpolation to generate a G frame FmIR123′adG in which all pixels of one frame shown in FIG. 19C are made up of G pixel data.

The demosaic processing unit 54 generates B interpolation pixel data B3adi obtained by calculating B pixel data at a pixel position where no B pixel data exists using surrounding B3ad pixel data. In the demosaic processing unit 54, all pixels in one frame shown in FIG.
A frame FmIR123'adB is generated.

The intermediate second mode and the night vision second mode are different in that the former does not operate the surrounding pixel adding unit 521 while the latter operates the surrounding pixel adding unit 521.

Further, in the intermediate mode, pixel data for each color is generated from each pixel data obtained from the region corresponding to each color in the light receiving element, but in the night vision mode, the surrounding pixels are added. It can also be said that pixel data for each color is generated from pixel data obtained from a region wider than each region for generating pixel data for each color.

<Example of mode switching>
An example of mode switching by the mode switching unit 72 will be described with reference to FIG. (A) of FIG.
As an example, schematically shows how the brightness of the surrounding environment changes as time elapses from the daytime period to the nighttime period.

As shown in FIG. 20 (a), the brightness decreases as time elapses from daytime to evening, and it becomes almost dark after time t3. The brightness shown in FIG. 20A substantially shows the amount of visible light, and there is almost no visible light after time t3.

The control unit 7 can determine the brightness of the surrounding environment based on the luminance level of the video signal (video data) input from the video processing unit 5. As shown in FIG. 20B, the mode switching unit 72 sets the normal mode when the brightness is equal to or higher than a predetermined threshold value Th1 (first threshold value), and sets the predetermined threshold value Th2 (less than the threshold value Th1). When the second threshold) or higher, intermediate mode, threshold Th2
When it is less than that, the night vision mode is set.

The imaging apparatus according to the present embodiment is in the normal mode until time t1 when the brightness becomes the threshold value Th1, and at time t1.
From 1 to the time t2 when the brightness becomes the threshold value Th2, the intermediate mode, and after the time t2, the night vision mode,
The mode is switched automatically. In FIG. 20B, the intermediate mode may be either the intermediate first mode or the intermediate second mode, and the night vision mode may be either the night vision first mode or the night vision second mode.

In FIG. 20A, the brightness immediately before the time t3 when almost no visible light is present is set as the threshold Th2, but the brightness at the time t3 may be set as the threshold Th2.

As shown in FIG. 20 (c), the mode switching unit 72 has a period of time t1 in which the visible light is relatively high in the intermediate mode period, and is on the time t2 side in the intermediate first mode. The period may be set to the intermediate second mode. In FIG. 20C, the night vision mode may be either the night vision first mode or the night vision second mode.

In the imaging apparatus according to the present embodiment, the light projection control unit 71 controls on / off of the infrared projector 9, and the mode switching unit 72 switches the operation / non-operation of each unit in the video processing unit 5. Can be realized.

As shown in FIG. 21, in the normal mode, the infrared projector 9 is off, the surrounding pixel adding unit 521, the same position pixel adding unit 522 and the synthesizing unit 523 are not operating, and the demosaic processing unit 54 is operating.

In the intermediate first mode, the infrared projector 9 is on, the surrounding pixel adding unit 521 is not operating, and the same-position pixel adding unit 522, the combining unit 523, and the demosaic processing unit 54 are operating. In the intermediate second mode, the infrared projector 9 is on, and the same pixel addition unit 522 as the surrounding pixel addition unit 521 is used.
Are inoperative, and the combining unit 523 and the demosaic processing unit 54 are in operation.

As described above, the operation and non-operation in the same-position pixel adding unit 522 are expressed by the equations (1) to (3).
) Can be easily switched by appropriately setting the values of the coefficients ka to ki.

In the first night-vision mode, the infrared projector 9 is on, and the surrounding pixel addition unit 521, the same-position pixel addition unit 522, the synthesis unit 523, and the demosaic processing unit 54 are all in an operating state. Night vision 2
In the mode, the infrared projector 9 is on, the same-position pixel adding unit 522 is not operating, and the surrounding pixel adding unit 521, the combining unit 523, and the demosaic processing unit 54 are operating.

By the way, the surrounding pixel addition unit 521 uses a coefficient exceeding the zero (0) as a coefficient to multiply the surrounding pixel data in the calculation formula for adding the surrounding pixel data to the pixel data of the target pixel.
For example, in the case of 1), the surrounding pixel addition processing can be set to the operation state.

Further, the surrounding pixel adding unit 521 can set the surrounding pixel addition processing to the non-operating state by setting the coefficient by which the surrounding pixel data is multiplied to 0 in the calculation formula.

The operation and non-operation of the surrounding pixel addition unit 521 can be easily switched by appropriately setting the coefficient value.

<First Modification of Imaging Device>
The method by which the control unit 7 detects the brightness of the surrounding environment is not limited to the method based on the luminance level of the video signal.

As shown in FIG. 22, the brightness sensor 11 may detect the brightness of the surrounding environment.
In FIG. 20, the brightness of the surrounding environment may be determined based on both the luminance level of the video signal and the brightness detected by the brightness sensor 11.

<Second Modification of Imaging Device>
The control unit 7 does not directly detect the brightness of the surrounding environment, but roughly assumes the brightness of the surrounding environment based on the time (date) and time (time zone) in one year, and the mode switching unit 72 may be switched to each mode.

As shown in FIG. 23, in the mode setting table 12, any one of the normal mode, the intermediate mode, and the night vision mode is set corresponding to the combination of the date and the time zone. A clock 73 in the control unit 7 manages the date and time. The control unit 7 refers to the date and time indicated by the clock 73 and reads the mode set from the mode setting table 12.

The light projection control unit 71 and the mode switching unit 72 control the imaging device so that the mode read from the mode setting table 12 is set.

<Third Modification of Imaging Device>
As illustrated in FIG. 24, the imaging device may be controlled such that the user manually selects a mode using the operation unit 13 and the light projection control unit 71 and the mode switching unit 72 are in the selected mode.
The operation unit 13 may be an operation button provided on the housing of the imaging apparatus or a remote controller.

<Video signal processing method>
A video signal processing method executed by the imaging apparatus shown in FIG. 1 will be described again with reference to FIG.

In FIG. 25, when the imaging apparatus starts operation, the control unit 7 determines in step S1 whether or not the brightness of the surrounding environment is equal to or greater than a threshold value Th1. If it is equal to or greater than the threshold Th1 (YES), the control unit 7 causes the process in the normal mode to be executed in step S3. If it is not equal to or greater than the threshold value Th1 (NO), the control unit 7 determines whether or not the brightness of the surrounding environment is equal to or greater than the threshold value Th2 in step S2.

If it is equal to or greater than the threshold value Th2 (YES), the control unit 7 causes the process in the intermediate mode to be executed in step S4. If it is not greater than or equal to the threshold Th2 (NO), the control unit 7 causes the process in the night-vision mode to be executed in step S5.

Control part 7 returns processing to Step S1 after Steps S3-S5, and repeats after Step S1.

FIG. 26 shows specific processing in the normal mode in step S3. In FIG. 26, the control part 7 (light projection control part 71) turns off the infrared light projector 9 in step S31. Control unit 7
In Step S32, the infrared cut filter 21 is inserted. In step S33, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Tb. The order of steps S31 to S33 is arbitrary and may be simultaneous.

In step S34, the control unit 7 causes the imaging unit 3 to image the subject. In step S <b> 35, the control unit 7 causes the demosaic processing unit 54 to perform demosaic processing on the frames constituting the video signal generated by the imaging unit 3 imaging the subject.
To control.

FIG. 27 shows specific processing in the intermediate mode in step S4. In FIG. 27, the control unit 7 (light projection control unit 71) receives wavelengths IR1 to IR from the light projection units 91 to 93 in step S41.
The infrared projector 9 is turned on so that the infrared light of 3 is projected in a time-sharing manner.

The control part 7 inserts the dummy glass 22 in step S42. In step S43, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Ta. The order of steps S41 to S43 is arbitrary and may be simultaneous.

In step S44, the control unit 7 causes the imaging unit 3 to image the subject. The imaging unit 3 projects infrared light with a wavelength IR1 associated with R, infrared light with a wavelength IR2 associated with G, and infrared light with a wavelength IR3 associated with B, respectively. Take a picture of the subject in the

In step S45, the control unit 7 (mode switching unit 72) controls the pre-signal processing unit 52 so that the surrounding pixel addition unit 521 is deactivated and the synthesis unit 523 is operated to generate a synthesized video signal.

Frames constituting a video signal generated when the imaging unit 3 images the subject in a state where infrared light of wavelengths IR1, IR2, and IR3 are respectively projected are a first frame and a second frame.
And the third frame.

The synthesizing unit 523 converts the three primary color pixel data based on the R pixel data in the first frame, the G pixel data in the second frame, and the B pixel data in the third frame into color The filters 32 are arranged in the same arrangement as the color filters. Synthesizer 52
3 generates a composite video signal by combining the first to third frames into one frame in this way.

In step S46, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frame of the composite video signal.

The demosaic processing unit 54 performs demosaic processing for generating an R frame, a G frame, and a B frame based on the frame of the composite video signal, and sequentially generates the demosaiced three primary color frames. .

The demosaic processing unit 54 can generate an R frame by interpolating the R pixel data at a pixel position where the R pixel data does not exist. The demosaic processing unit 54
A G frame can be generated by interpolating G pixel data at pixel positions where no G pixel data exists. The demosaic processing unit 54 can generate the B frame by interpolating the B pixel data at the pixel position where the B pixel data does not exist.

When the intermediate first mode is set, the same-position pixel adding unit 522 is operated in step S45, and when the intermediate second mode is set, the same-position pixel adding unit 52 is set in step S45.
2 may be set as non-operation.

FIG. 28 shows specific processing in the night-vision mode in step S5. In FIG. 28, the control unit 7 (light projection control unit 71) receives wavelengths IR1 to IR from the light projection units 91 to 93 in step S51.
The infrared projector 9 is turned on so that the infrared light of 3 is projected in a time-sharing manner.

The control part 7 inserts the dummy glass 22 in step S52. In step S53, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Ta. The order of steps S51 to S53 is arbitrary and may be simultaneous.

In step S54, the control unit 7 causes the imaging unit 3 to image the subject. Control unit 7 (
In step S55, the mode switching unit 72) controls the previous signal processing unit 52 to operate the surrounding pixel adding unit 521 and the combining unit 523 to generate a combined video signal.

In step S56, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frame of the composite video signal.

When the night vision first mode is set, the same position pixel addition unit 522 is operated in step S55, and when the night vision second mode is set, the same position pixel addition unit 52 is set in step S55.
2 may be set as non-operation.

<Video signal processing program>
In FIG. 1, the control unit 7 or an integrated part of the video processing unit 5 and the control unit 7 is configured by a computer (microcomputer), and the video signal processing program (computer program) is executed by the computer. It is also possible to realize the same operation as that of the imaging apparatus of this embodiment. The image output unit 6 may be configured by a computer.

An example of a procedure executed by the computer when the control in the intermediate mode which is step S4 in FIG. 23 is configured by the video signal processing program will be described with reference to FIG. FIG.
The process which a video signal processing program performs a computer is shown.

In FIG. 29, the video signal processing program loads R to the computer in step S401.
, G, and B, the step of controlling the infrared projector 9 so as to project infrared light of wavelengths IR1, IR2, and IR3 respectively.

The step shown in step S401 may be executed outside the video signal processing program.
In FIG. 29, the step of inserting the dummy glass 22 is omitted. Dummy glass 22
The step of inserting may also be executed outside the video signal processing program.

In step S402, the video signal processing program captures the first frame of the video signal generated by the imaging unit 3 imaging the subject in a state where infrared light having the wavelength IR1 is projected on the computer. A step of acquiring pixel data to be configured is executed.

In step S403, the video signal processing program captures the second frame of the video signal generated when the imaging unit 3 images the subject in a state where infrared light having the wavelength IR2 is projected on the computer. A step of acquiring pixel data to be configured is executed.

In step S404, the video signal processing program captures a third frame of the video signal generated when the imaging unit 3 images the subject in a state where infrared light having the wavelength IR3 is projected on the computer. A step of acquiring pixel data to be configured is executed. Step S402
The order of ~ S404 is arbitrary.

In step S405, the video signal processing program causes the computer to arrange the R, G, and B pixel data so as to be in the same arrangement as the arrangement of the color filters in the color filter 32, and combine the synthesized video into one frame. A step of generating a signal is executed.

In the intermediate mode, the video signal processing program does not cause the computer to execute the step of adding surrounding pixels in step S405.

In step S406, the video signal processing program causes the computer to execute a step of performing demosaic processing on the frame of the composite video signal to generate R, G, and B frames.

Although illustration is omitted, when the control in the night vision mode, which is step S5 in FIG. 25, is configured by the video signal processing program, the step of adding surrounding pixels is executed in the computer in step S405 in FIG. You can do it.

The video signal processing program may be a computer program recorded on a computer-readable recording medium. The video signal processing program may be provided in a state where it is recorded on a recording medium, or may be provided via a network such as the Internet so that the computer can download the video signal processing program. The computer-readable recording medium may be any non-temporary arbitrary recording medium such as a CD-ROM or a DVD-ROM.

In the imaging apparatus of the present embodiment configured as shown in FIG. 1, for example, a plurality of units may be provided as necessary, and the intermediate mode and the night vision mode may be executed simultaneously. In that case, the video output unit 6 may output both the video signal generated in the intermediate mode and the video signal generated in the night vision mode.

The mode switching unit 72 switches between a state in which the video output unit 6 outputs the video signal generated in the intermediate mode and a state in which the video output unit 6 outputs the video signal generated in the night vision mode. May be. At this time, as described above, switching may be performed according to the brightness of the surrounding environment, the time, or the like. The video processing unit 5 (video processing device) may be separated from other units.

Furthermore, without using the intermediate mode, switching from normal mode to night vision mode,
You may make it switch from night vision mode to normal mode.

In the normal mode and the night vision mode, in the state where the visible light and the infrared light for night vision imaging are mixed, the color video signal is not better than the intermediate mode. However, even in the normal mode or the night vision mode, it is possible to take an image in a state where visible light and infrared light for night vision are mixed, and the ambient brightness changes (for example, all day with a surveillance camera). Even in the case of shooting), it is possible to achieve an effect of enabling imaging with the same imaging device.

Furthermore, the normal mode may be switched to the intermediate mode, or the intermediate mode may be switched to the normal mode without using the night vision mode. In that case, the night vision mode may not be installed.

The normal mode and intermediate mode do not provide better color video signals than the night vision mode in an environment with little visible light, but imaging in an environment with little visible light is possible even in the intermediate mode. Even when the ambient brightness changes (for example, when shooting all day with a surveillance camera), it is possible to achieve an effect of enabling imaging with the same imaging device.
In some places where there is an electric light, the night vision mode may not be used.

Next, a specific mode switching method for switching the image in each mode between the normal mode and the infrared light projection mode without causing image disturbance will be described.

<First Example of Mode Switching Method>
30A and 30B show a first example of the mode switching method. 30A and 30B show the operation of each part of the imaging apparatus when the video output unit 6 outputs an interlace video signal by the NTSC encoder 61.

30A and 30B, the control unit 7 switches the imaging apparatus from the infrared light projection mode to the normal mode at time t1, and from the normal mode to the infrared light projection mode at time t2. The sequence of cases is shown. The infrared light projection mode may be either an intermediate mode or a night vision mode.

30A and 30B show the sequence divided into two. Note that FIG. 30A and FIG. 30B partially overlap.

30A and 30B, (a) is a state of infrared light projection by the infrared projector 9,
(B) shows exposure of the imaging unit 3. The exposure of the imaging unit 3 shown in FIGS. 30A and 30B (b) shows the maximum exposure time tExmax described in FIG. 6A, and the actual exposure time varies depending on the shutter speed. However, in the infrared light projection mode, the maximum exposure time tExmax is likely to be the actual exposure time.

30A and 30B, (c) shows a frame generated by an imaging signal read from the imaging unit 3 based on each exposure of (b). As shown in FIG. 30A, based on the exposures Ex1R, Ex1G, and Ex1B, the frame F1R corresponding to the exposure Ex1R and the exposure Ex1G
A frame F1G corresponding to the above and a frame F1B corresponding to the exposure Ex1B are obtained. Frame F1R, F1G
, F1B correspond to the frames F1IR1, F1IR2, and F1IR3 in FIG.

Also, based on exposure Ex2R, Ex2G, Ex2B, frame F2R corresponding to exposure Ex2R, exposure Ex2G
A frame F2G corresponding to, and a frame F2B corresponding to exposure Ex2B are obtained. Frame F2R, F2G
, F2B correspond to the frames F2IR1, F2IR2, and F2IR3 in FIG. The maximum exposure time tExmax of exposures Ex1R to Ex2B is 1/90 seconds.

In the period of the infrared light projection mode before time t1, the imaging signal output from the imaging unit 3 is 9
It is a progressive method of 0 frame / second. The progressive method of 90 frames / second is referred to as 90P. The imaging device generates a 60 field / second interlace video signal based on the 90P imaging signal, as will be described later. 60 fields /
The second interlace method will be referred to as 60i.

The light projection control unit 71 turns off the infrared light projection by the infrared projector 9 at time t1. The control unit 7 operates the imaging device in the normal mode after time t1.

If the normal mode period is 30 frames / second, the interlace method is 60 fields / second. Therefore, in the normal mode, the control unit 7 controls the imaging unit 3 so that the imaging unit 3 outputs a 60P imaging signal so that the video output unit 6 finally generates and outputs a 60i video signal. .

Specifically, the imaging unit 3 sets the maximum exposure time tExmax to 1/60 seconds, and alternately repeats exposure for odd fields and exposure for even fields. 30B of FIG. 30A and FIG. 30B
, The subscript o indicates exposure for odd fields, and the subscript e indicates exposure for even fields.

At the timing of the first exposure after time t1, a frame F2B having a maximum exposure time tExmax of 1/90 second is generated as shown in (c) of FIG. 30A. Therefore, in the first exposure after time t1, the maximum exposure time tExmax is not set to 1/60 seconds, but the maximum exposure time tExmax.
Is the exposure Ex3o for odd fields with 1/90 seconds.

As illustrated in FIG. 30A, in the first example, the imaging unit 3 performs time t when the exposure Ex3o ends.
Up to 11, it operates to output 90P image signals. The imaging unit 3 starts from the time t11, 6
It operates to output an image signal of 0P.

Since the exposure Ex3e for the even field, which is the first exposure after the time t11, is a pair exposure with the exposure Ex3o for the odd field, the maximum exposure time tExmax is not set to 1/60 seconds, but the exposure Ex3o The maximum exposure time tExmax is preferably 1/90 second.

Therefore, the control unit 7 causes the imaging unit 3 to discharge the charge accumulated by the extraction pulse Ps1 as described with reference to FIG. 6 only for a period of 1/180 seconds from the beginning of the maximum exposure time tExmax of 1/60 seconds. Control. As a result, a temporal gap of 1/180 seconds is formed between the exposure Ex3o and the exposure Ex3e, and the substantial maximum exposure time of the exposure Ex3e is 1/90 seconds.

By reading out the charges accumulated by the exposures Ex3o and Ex3e with a maximum exposure time of 1/90 seconds in a period of 1/60 seconds, a frame F3o corresponding to the exposure Ex3o and a frame F3e corresponding to the exposure Ex3e are obtained.

After the exposure Ex3e, the frames F4o, F4e, F5 are read out by reading out the charges accumulated by the exposures Ex4o, Ex4e, Ex5o, Ex5e whose maximum exposure time tExmax is 1/60 seconds in a period of 1/60 seconds.
o, F5e is obtained.

As shown in FIG. 30B, the light projection control unit 71 turns on the infrared light projection by the infrared light projector 9 again at time t2. The control unit 7 causes the imaging device to operate in the infrared light projection mode after time t2.

The frame F5e based on the exposure Ex5e occurs in a 1/60 second period straddling the time t2. In the infrared light projection mode after time t2, the maximum exposure time tExmax needs to be 1/90 seconds. Therefore, in the exposure Ex6R following the exposure Ex5e, the control unit 7 sets the maximum exposure time tExmax of 1/60 seconds.
The imaging unit 3 is controlled to discharge the electric charge accumulated by the extraction pulse Ps1 only for a period of 1/180 seconds from the beginning of.

As a result, a time gap of 1/180 seconds is formed between the exposure Ex5e and the exposure Ex6R, and the substantial maximum exposure time of the exposure Ex6R becomes 1/90 seconds.

Based on the exposures Ex6R, Ex6G, and Ex6B, a frame F6R corresponding to the exposure Ex6R, a frame F6G corresponding to the exposure Ex6G, and a frame F6B corresponding to the exposure Ex6B are obtained. Frame F6R, F6G, F6B
Since one frame is formed by the above, the exposure times of the exposures Ex6R, Ex6G, and Ex6B must be the same. By setting the maximum exposure time of exposure Ex6R to 1/90 second, exposure Ex6R,
Ex6G and Ex6B can have the same exposure time.

Based on the exposures Ex7R, Ex7G, and Ex7B, a frame F7R corresponding to the exposure Ex7R, a frame F7G corresponding to the exposure Ex7G, and a frame F7B corresponding to the exposure Ex7B are obtained. The same applies to exposure after Ex8R.

As illustrated in FIG. 30B, in the first example, the imaging unit 3 determines that the time t at which the exposure Ex6R ends.
It operates to output 60P image signals up to 21 and operates to output 90P image signals after time t21.

As described above, in the first example shown in FIGS. 30A and 30B, the maximum exposure time tExmax is 1/6 in the first exposure after time t1 when the infrared light projection mode is switched to the normal mode.
The exposure Ex3o is set to 1/90 seconds instead of 0 seconds.

In addition, the charge for a period of 1/180 seconds from the beginning of the maximum exposure time tExmax of 1/60 seconds straddling the time t2 when switching from the normal mode to the infrared light projection mode is discharged, and the first after the time t2 The maximum exposure time tExmax of exposure is 1/90 seconds.

Thereby, when the frame of the imaging signal generated by the imaging signal read from the imaging unit 3 is switched from the infrared light projection mode to the normal mode, as shown in (c) of FIG. 30A and FIG. 30B, Both are continuous when switching from the normal mode to the infrared light projection mode.

30A and 30B, (d) shows the writing of the frame of the video data output from the A / D converter 4 to the frame buffer 50, and (e) shows the reading of the frame from the frame buffer 50. Symbols of the memories 50a to 50f to be written or read are shown in the respective sections (d) and (e) of FIGS. 30A and 30B.

The frame of the video data output from the A / D converter 4 is also shown in (c of FIG. 30A and FIG. 30B.
) Of the image pickup signal shown in FIG. 4) and F1R, F1G, F1B,. Frame F1R, F1G
, F1B are written in the memories 50a, 50b, 50c, respectively.

When the writing of the frame F1B to the memory 50c is completed, from the memories 50a to 50c,
Frames F1R, F1G, and F1B are simultaneously read out in a 1/60 second period. Even in the subsequent 1/60 second period, the frames F1R, F1G, and F1B are simultaneously read from the memories 50a to 50c. That is, the frames F1R, F1G, and F1B are continuously read out in two 1/60 second periods.

The frames F2R, F2G, and F2B are written in the memories 50d, 50e, and 50f in correspondence with the period for reading the frames F1R, F1G, and F1B, respectively. Frames F2R, F2G, F2B are also frame F
When the writing to the 2B memory 50f is completed, it is continuously read out in two 1/60 second periods.

The reason for reading out one set of frames in the infrared light projection mode twice is to generate odd fields in one set of read frames and generate even fields in the other set of frames.

The frames F3o and F3e are written in the memories 50a and 50b. Frame F3e memory 50
When the writing to b is completed, the frames F3o and F3e are read from the memories 50a and 50b. The frames F4o and F4e are written in the memories 50d and 50e. Frame F4e memory 5
When the writing to 0e is completed, the frames F4o and F4e are read from the memories 50d and 50e.
The

The frames F5o and F5e are written in the memories 50a and 50b. Memory F of frame F5e
When the writing to b is completed, the frames F5o and F5e are read from the memories 50a and 50b.

The frames F6R, F6G, and F6B are written in the memories 50d, 50e, and 50f, respectively. The frames F6R, F6G, and F6B are continuously read out in two 1/60 second periods when the writing of the frame F6B into the memory 50f is completed.

The frames F7R, F7G, and F7B are written in the memories 50a, 50b, and 50c, respectively. Although not shown in FIG. 30B, the frames F7R, F7G, and F7B are stored in the memory 50f of the frame F7B.
When writing to is completed, it is continuously read out in two 1/60 second periods. Thereafter, the same operation is repeated.

30A and 30B, (f) shows the previous signal processing in the previous signal processing unit 52 described above. The previous signal processing is represented as P52. The entire frames F1R, F1G, and F1B read from the memories 50a to 50c are represented as a frame F1RGB. For example, P52 (F1RGB) is frame F1RGB
Shows that the previous signal processing P52 is applied.

As shown in (f) of FIG. 30A and FIG. 30B, the pre-signal processing P52 is performed on the frames F0RGB, F1RGB, F2RGB, F6RGB,.

30A and 30B, (g) shows the previous signal processing in the demosaic processing unit 54 described above. The demosaic process is represented as P54. For example, P54 (F1RGB) is frame F1R
It shows that the above-mentioned demosaic processing P54 is applied to GB.

As shown in FIG. 30A and FIG. 30B (g), the frame F0RGB subjected to the previous signal processing P52.
, F1RGB, F2RGB, F6RGB... Are subjected to demosaic processing P54. The demosaic processing P54 is applied to the frames F3o, F3e, F4o, F4e, F5o, and F5e that have not been subjected to the previous signal processing P52.

30A and 30B, (h) shows an interlace video signal obtained by performing progressive-interlace conversion (PI conversion) on each frame shown in (g) by the NTSC encoder 61. In FIG. 30A and FIG. 30B (h), the subscript o indicates an odd-field video signal, and the subscript e indicates an even-field video signal.

As shown in (h) of FIG. 30A, in the infrared light projection mode, interlaced video signals F0io, F0ie, F1io are based on the video data of the frames F0RGB, F1RGB, F2RGB subjected to demosaic processing P54. , F1ie, F2io, F2ie are generated and output in order. Subscript i
Indicates that the horizontal lines are thinned out by the PI conversion to form an interlaced video signal.

As shown in (h) of FIG. 30A and FIG. 30B, when the mode is switched to the normal mode, the interlace method is used based on the video data of each of the frames F3o, F3e, F4o, F4e, F5o, and F5e subjected to the demosaic process P54. Video signals F3io, F3ie, F4io, F4ie, F5io, and F5ie are sequentially generated and output.

As shown in (h) of FIG. 30B, when switching to the infrared light projection mode, demosaic processing P5
Based on the respective video data of the frame F6RGB to which 4 is applied, interlaced video signals F6io, F6ie,... Are sequentially generated and output.

As shown in (h) of FIG. 30A and FIG. 30B, according to the first example, even if the 60i video signal is switched from the infrared light projection mode to the normal mode, the normal mode changes to the infrared light. Each frame (each field) is continuously output even when switched to the projection mode.

Therefore, according to the first example, it is possible to switch the video in each mode without the video being disturbed.

In the first example shown in FIGS. 30A and 30B, during the normal mode period, (b)
The boundary of the frame generated by the exposure of the image pickup unit 3 shown in FIG. 6 coincides with the boundary of the frame (field) of the output video signal shown in (g). That is, the frame generated by the exposure of the imaging unit 3 and the frame of the output video signal are synchronized. Therefore, noise is unlikely to occur in the output video signal.

<Second Example of Mode Switching Method>
31A and 31B show a second example of the mode switching method. 31A and 31B also show the operation of each part of the imaging apparatus when the video output unit 6 outputs an interlace video signal by the NTSC encoder 61.

In the second example shown in FIGS. 31A and 31B, the description will focus on parts different from the first example shown in FIGS. 30A and 30B. 31A and 31B also show the sequence divided into two, and FIG. 31A and FIG. 31B partially overlap.

As shown in (b) of FIG. 31A and FIG. 31B, in the second example, in the normal mode period from time t1 to t2, the imaging unit 3 performs exposure with the maximum exposure time tExmax set to 1/60 seconds. . As shown in (c) of FIG. 31A and FIG. 31B, the control unit 7 performs exposure in the normal mode period.
Based on Ex3o, Ex3e, Ex4o, Ex4e, Ex5o, and Ex5e, control is performed so that the imaging signal is read out from the imaging unit 3 in 1/90 seconds to generate frames F3o, F3e, F4o, F4e, F5o, and F5e.

Since reading of the frame F3o corresponding to the exposure Ex3o is started from the time when the exposure Ex3o ends, the frame F3o starts from the time when 1/180 second has elapsed from the time when the frame F2B ends. Therefore, a time gap of 1/180 seconds is formed between the frame F2B and the frame F3o. Thereafter, similarly, a temporal gap of 1/180 seconds is formed between adjacent frames of the frames F3o, F3e, F4o, F4e, F5o, and F5e.

FIGS. 31A and 31B (d) to (h) are the same as FIGS. 30A and 30B (d) to (h). However, in FIGS. 31A and 31B (d) and (e), the frames F3o to F5e written in the frame buffer 50 in a period of 1/90 seconds are read out in a period of 1/60 seconds, respectively.

As shown in FIG. 31A and FIG. 31B (g), according to the second example, even if the 60i video signal is switched from the infrared light projection mode to the normal mode, Each frame (each field) is continuously output even when switched to the projection mode.

Therefore, according to the second example, it is possible to switch the video in each mode without causing video disturbance.

In the second example shown in FIGS. 31A and 31B, during the normal mode period, (b)
The frame generated by the exposure of the image pickup unit 3 shown in FIG. 6 and the frame of the output video signal shown in (g) are not synchronized. However, in the second example shown in FIGS. 31A and 31B, in the normal mode period, the pair of periods of the odd-numbered field frame and even-numbered field frame of the imaging unit 3 are all 1/30 seconds. To be unified.

<Third example of mode switching method>
32A and 32B show a third example of the mode switching method. 32A and 32B also show the operation of each part of the imaging apparatus when the video output unit 6 outputs an interlaced video signal by the NTSC encoder 61. FIG.

In the third example shown in FIGS. 32A and 32B, the description will focus on parts different from the first example shown in FIGS. 30A and 30B. 32A and 32B also show the sequence divided into two, and FIG. 32A and FIG. 32B partially overlap.

In order to realize the third example, as shown in FIG. 1, the frame buffer 50 is configured to include memories 50 a to 50 i having a capacity of a total of nine frames. In order to realize the third example, a configuration having a memory with a capacity of at least 7 frames may be used. Memory 50a
It is set as the structure which has ~ 50i.

As shown in (b) of FIG. 32A and FIG. 32B, in the third example, in the normal mode period from time t1 to t2, the imaging unit 3 performs exposure with the maximum exposure time tExmax set to 1/60 seconds. .

As shown in (c) of FIG. 32A and FIG. 32B, the control unit 7 performs exposure Ex during the normal mode period.
Based on 3o, Ex3e, Ex4o, Ex4e, and Ex5o, control is performed so that the charges accumulated from the imaging unit 3 are read out in 1/60 second, and frames F3o, F3e, F4o, F4e, and F5o are generated.

As shown in (c) of FIG. 32A, the control unit 7 performs the final exposure Ex in the infrared light projection mode.
Only for 2B, the charge accumulated from the imaging unit 3 is controlled to be read out in 1/60 second, and the frame F2
Generate B.

As shown in (c) of FIG. 32B, the control unit 7 performs control so that only the last exposure Ex5e in the period of the normal mode reads out the electric charge accumulated from the imaging unit 3 in 1/90 seconds, and the frame F5e is read. Generate. As a result, the maximum exposure time of the first exposure Ex6R after time t2 and the frame F5e
The period is the same.

As shown in (d) of FIG. 32A, frames F1R, F1G, and F1B are stored in memories 50a and 5a, respectively.
It is written in 0b, 50c. As shown in (e) of FIG. 32A, the memory 5 of the frame F1B
When the writing to 0c is completed, the frames F1R, F1G, and F1B are continuously read from the memories 50a to 50c in two 1/60 second periods.

The frames F2R, F2G, and F2B are written in the memories 50d, 50e, and 50f, respectively. When the writing of the frame F2B to the memory 50f is completed, the frames F2R, F2G, and F2B are also continuously read out in two 1/60 second periods.

As can be seen by comparing (d) and (e) in FIG. 32A, in the third example, for example,
Timing for writing the frames F2R, F2G, and F2B to the memories 50d to 50f and the memory 50a
The timings for reading the frames F1R, F1G, and F1B from .about.50c do not match. This is because the frame F2B is generated in a period of 1/60 seconds.

Therefore, the third example cannot be realized with the frame buffer 50 having the configuration of the memories 50a to 50f having a capacity of 6 frames in total as shown in FIG. Therefore, the third
In order to realize this example, the frame buffer 50 is configured to have the memories 50a to 50i having a capacity of a total of nine frames shown in FIG.

As shown in FIGS. 32A and 32B (d) and (e), the frames F3o and F3e are stored in the memory 50.
g, 50h. When the writing of the frame F3e to the memory 50h is completed, the frames F3o and F3e are read from the memories 50g and 50h. Frames F4o and F4e are memory 5
It is written in 0a and 50b. When the writing of the frame F4e to the memory 50b is completed,
Frames F4o and F4e are read from the memories 50a and 50b.

The frames F5o and F5e are written in the memories 50d and 50e. Memory F of frame F5e
When the writing to e is completed, the frames F5o and F5e are read from the memories 50d and 50e.

The frames F6R, F6G, and F6B are written in the memories 50g, 50h, and 50i, respectively. The frames F6R, F6G, and F6B are continuously read out in two 1/60 second periods when the writing of the frame F6B to the memory 50i is completed.

The frames F7R, F7G, and F7B are written in the memories 50a, 50b, and 50c, respectively. Although not shown in FIG. 32B, the frames F7R, F7G, and F7B are stored in the memory 50c of the frame F7B.
When writing to is completed, it is continuously read out in two 1/60 second periods. Thereafter, the same operation is repeated.

(F) to (h) in FIGS. 32A and 32B are the same as (g) to (h) in FIGS. 30A and 30B.

As shown in FIG. 32A and FIG. 32B (h), according to the third example, even if the 60i video signal is switched from the infrared light projection mode to the normal mode, Each frame (each field) is continuously output even when switched to the projection mode.

Therefore, according to the third example, it is possible to switch the video of each mode without causing the disturbance of the video.

In the third example shown in FIG. 322A and FIG. 32B, (b
The synchronization of the exposure of the imaging unit 3 shown in () matches the synchronization of the output video signal shown in (g). If the exposure synchronization of the imaging unit 3 and the synchronization of the output video signal match, it is difficult for noise to occur in the output video signal.

As described above, in order to realize the third example, it is necessary to increase the memory capacity of the frame buffer 50 compared to the case of realizing the first example. The first example is better in that the memory capacity of the frame buffer 50 is further reduced.

However, if the memory write and read controls are devised, the third example can be realized with the frame buffer 50 having a memory capacity of 7 frames. Therefore, even in the third example, the memory capacity of the frame buffer 50 does not matter so much.

<Summary of Mode Switching Methods in Imaging Device>
To summarize the first to third examples of the mode switching method described above, the imaging apparatus of the present embodiment operates as follows.

The imaging unit 3 images a subject. The video processing unit 5 generates a first video signal based on the imaging signal output from the imaging unit 3. The first video signal is a video signal obtained by performing both the previous signal processing P52 and the demosaic processing P54 or only the demosaic processing P54 on the video data output from the A / D converter 4.

The video output unit 6 (NTSC encoder 61) generates and outputs a second video signal of a predetermined signal system based on the first video signal. The predetermined signal system is, for example, an interlace system, and may be a progressive system. The second video signal is a video signal converted into a final signal system based on the first video signal.

Here, the normal mode is referred to as a first mode, and the infrared light projection mode is referred to as a second mode. In the first mode, the image capturing unit 3 captures an image of a subject by performing exposure in accordance with each frame period of the second video signal. In the second mode, the imaging unit 3 captures an image of the subject by performing exposure under different imaging conditions in each section obtained by dividing each frame period of the second video signal into a plurality of sections.

In the first mode, the video processing unit 5 generates each frame of the first video signal based on the imaging signal read corresponding to the exposure corresponding to one frame period, In the mode, each frame of the first video signal is generated based on the imaging signal read corresponding to the exposure of each section.

The video output unit 6 sets the second video signal to a common signal system (for example, 60i) in the first mode and the second mode, and continuously outputs frames of the second video signal. Therefore, the images in the first mode and the second mode can be switched to each other without causing the image disturbance. More preferably, the second video signal has the same horizontal and vertical frequencies in the first mode and the second mode.

In the second mode in which the subject is imaged while projecting infrared light in an environment where there is little visible light,
The imaging unit 3 images the subject as follows.

The imaging unit 3 images the subject by dividing each frame period of the second video signal into three sections. The imaging unit 3 has first imaging in each of the three sections as different imaging conditions.
The subject is imaged in each of a state where the infrared light is projected, a state where the second infrared light is projected, and a state where the third infrared light is projected. The first infrared light has a first wavelength associated with red. The second infrared light has a second wavelength associated with green. The third infrared light has a third wavelength associated with blue.

When the video output unit 6 outputs an interlace video signal as the second video signal, the imaging unit 3, the video processing unit 5, and the video output unit 6 may operate as follows.

In the first mode, the imaging unit 3 images the subject by dividing each frame period of the second video signal into two field periods of an odd field period and an even field period. The control unit 7 sets the maximum exposure time of the two field periods to the same time. The video processing unit 5 generates a video signal in the odd field period and a video signal in the even field period as the first video signal.

In the first mode, the video output unit 6 generates an odd-field video signal based on the odd-field video signal generated by the video processing unit 5, and generates an even-field video signal based on the even-field video signal. Generate a video signal. In the second mode, the video output unit 6
Based on each frame of the first video signal generated by the video processing unit 5, an odd-field video signal and an even-field video signal are generated.

In order for the video output unit 6 to continuously output the frames of the second video signal so that the video is not disturbed, the imaging unit 3 specifically operates as follows.

In the first example described with reference to FIG. 30A and FIG. 30B, the operation is as follows. In the exposure of the first section in which the imaging unit 3 is switched from the first mode to the second mode, the charge is discharged from the beginning of the maximum exposure time of the field period in the first mode, and the maximum exposure time is The time is the same as the maximum exposure time of one section in the second mode.

The imaging unit 3 sets the maximum exposure time to be the same as the maximum exposure time of one section in the second mode in the exposure in the first field period switched from the second mode to the first mode. In the first mode, the imaging unit 3 reads the electric charge accumulated by exposure in each field period as an imaging signal in one field period.

In the second example described with reference to FIG. 31A and FIG. 31B, the operation is as follows. In the first mode, the imaging unit 3 sets one field period as the maximum exposure time, and after the exposure of the maximum exposure time is completed, the charge accumulated by the exposure is the same as the maximum exposure time of one section in the second mode. To read out as an imaging signal.

The imaging unit 3 provides a temporal gap corresponding to the difference between one field period and the maximum exposure time of one section in the second mode before reading the imaging signal.

In the third example described with reference to FIG. 32A and FIG. 32B, the operation is as follows. In the first mode, the imaging unit 3 exposes with one field period as the maximum exposure time.

The imaging unit 3 reads out the electric charge accumulated by the exposure of the last one section in the second mode as an imaging signal in one field period.

The imaging unit 3 reads out the electric charge accumulated by the exposure in the last field period in the first mode as an imaging signal at the same time as the maximum exposure time of one section.

<Control Method of Imaging Device>
The control unit 7 controls the imaging device as follows.

The control unit 7 causes the imaging unit 3 to image the subject. The control unit 7 causes the video processing unit 5 to generate the first video signal based on the imaging signal obtained by imaging the subject by the imaging unit 3. The control unit 7 causes the video output unit 6 to generate a second video signal of a predetermined signal system based on the first video signal.

When the control unit 7 sets the imaging device to the first mode, the imaging device is controlled as follows. The control unit 7 causes the imaging unit 3 to perform exposure in accordance with each frame period of the second video signal so as to image the subject. The control unit 7 generates each frame of the first video signal based on the imaging signal read out by the video processing unit 5 corresponding to the exposure corresponding to one frame period.

When the control unit 7 sets the imaging device to the second mode, the imaging device is controlled as follows. The control unit 7 causes the imaging unit 3 to expose the subject under different imaging conditions in each section obtained by dividing each frame period of the second video signal into a plurality of sections and cause the subject to be imaged. Control unit 7
Is based on the imaging signal read by the video processing unit 5 corresponding to the exposure of each section,
Each frame of the first video signal is generated.

The control unit 7 causes the video output unit 6 to set the second video signal to a common signal system in the first mode and the second mode, and continuously output the frames of the second video signal. More preferably, the second video signal has the same horizontal and vertical frequencies in the first mode and the second mode.

<Control program for imaging apparatus>
When the above-described mode switching method is realized by control using a computer program, a control program having the following steps may be executed by a computer mounted on the imaging apparatus.

First, the computer is caused to execute a first step of imaging a subject by the imaging unit 3. Next, the computer causes the imaging unit 3 to execute a second step of generating a first video signal based on an imaging signal obtained by imaging the subject. Finally, the computer is caused to execute a third step of generating a second video signal of a predetermined signal system based on the first video signal.

When the computer sets the imaging apparatus to the first mode, the first step and the second step are as follows. The first step is a step in which the imaging unit 3 captures an image of the subject by exposing in correspondence with each frame period of the second video signal. Second
This step is a step of generating each frame of the first video signal based on the imaging signal read corresponding to the exposure corresponding to one frame period.

When the computer sets the imaging apparatus to the second mode, the first step and the second step are as follows. The first step is a step in which the imaging unit 3 exposes the subject under different imaging conditions in each section obtained by dividing each frame period of the second video signal into a plurality of sections, and images the subject. The second step is a step of generating each frame of the first video signal based on the imaging signal read corresponding to the exposure of each section.

When the computer shifts the imaging device from the first mode to the second mode; and
When shifting from the second mode to the first mode, the third step is to make the second video signal a common signal system in the first mode and the second mode, and This is a step of continuously outputting the frames. In this step, it is more preferable that the second video signal has the same horizontal and vertical frequencies in the first mode and the second mode.

The control program for the imaging apparatus may be a computer program recorded on a computer-readable recording medium. The control program may be provided in a state of being recorded on a recording medium, or may be provided via a network such as the Internet so that the control program can be downloaded to a computer. The computer-readable recording medium may be any non-temporary arbitrary recording medium such as a CD-ROM or a DVD-ROM.

<Application example>
By the way, the operation of the image pickup apparatus of the present embodiment described above can be applied not only to the infrared light projection mode in which an object is imaged while projecting infrared light, but also to the following cases.

Even in the state where infrared light is not projected in an environment where visible light exists, each frame period of the second video signal is divided into a plurality of sections, and different imaging conditions (different from each other) The present invention can be applied to the case where so-called multiple exposure is performed at a shutter speed.

In other words, the second mode is not limited to the infrared light projection mode, and for the multiple exposure, each frame period of the second video signal is divided into, for example, three sections and exposed. This can be used when one image signal is generated by synthesizing the image pickup signals of the sections.

The present invention is not limited to the embodiment described above, and various modifications can be made without departing from the scope of the present invention. In the description of the problems, objects, and effects to be solved by the above-described invention, switching between the normal mode and the infrared light projection mode is described in order to facilitate understanding of the present invention.

However, as described above, the operation of the imaging apparatus of the present embodiment can be applied to multiple exposure. Based on the description of the problems, objects, and effects to be solved by the invention, the present invention is not construed as being limited to being applied only to switching between the normal mode and the infrared light projection mode.

3 Imaging unit 5 Video processing unit 6 Video output unit

The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue In the second mode in which each image is captured in the third section in which the third infrared light is projected, the imaging unit that images the subject, and the imaging signal output from the imaging unit And a video output unit that generates and outputs a video signal of a predetermined signal system based on the second mode, and the exposure time at least in the first section when switching from the second mode to the first mode is Ri maximum exposure time or less time der in one section in the mode, before Maximum exposure time is in the other sections in the first mode, to provide an imaging apparatus characterized by longer than the maximum exposure time in the second mode.
The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue In the second mode in which each image is captured in the third section in which the third infrared light is projected, the imaging step for imaging the subject, and the imaging signal captured in the imaging step A video output step of generating and outputting a video signal of a predetermined signal system based on the exposure time at least in the first section when switching from the second mode to the first mode. Maximum exposure in one section in the mode The following Ri time der, the maximum exposure time in the other sections in the first mode, a control method of an imaging apparatus characterized by longer than the maximum exposure time in the second mode.
The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue In the second mode in which each image is captured in the third section in which the third infrared light is projected, the imaging step for imaging the subject, and the imaging signal captured in the imaging step An image output step of generating and outputting a video signal of a predetermined signal system based on the computer, and switching from the second mode to the first mode, the exposure time in at least the first section is One section in the second mode Maximum exposure time or less time der the definitive is, providing the maximum exposure time in the other sections in the first mode, the control program of the image pickup apparatus characterized by longer than the maximum exposure time in the second mode To do.

Claims (7)

The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue An imaging unit for imaging a subject in a second mode for imaging in each of a third section for imaging in a state where the third infrared light is projected;
A video output unit that generates and outputs a video signal of a predetermined signal system based on the imaging signal output from the imaging unit;
The exposure time in the first section switched from the second mode to the first mode is a time shorter than the maximum exposure time in one section in the second mode.
An imaging apparatus characterized by that. In the first mode, the imaging unit divides one frame period of the predetermined video signal into two field periods, an odd field period and an even field period, and images a subject.
The first interval is an odd field period.
The imaging apparatus according to claim 1. The maximum exposure times of the two field periods are the same time.
The imaging apparatus according to claim 2. The imaging unit
In the exposure of the first section where the first mode is switched to the second mode, charges are discharged from the beginning of the maximum exposure time of the section in the first mode, and the maximum exposure time is set to the second exposure time. The same time as the maximum exposure time of one section in the mode of
The imaging apparatus according to any one of claims 1 to 3, wherein The video output unit generates a video signal of a common signal system in the first mode and the second mode;
The imaging apparatus according to any one of claims 1 to 4, wherein The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue An imaging step of imaging a subject in a second mode of imaging in each of a third section for imaging in a state where the third infrared light is projected;
A video output step of generating and outputting a video signal of a predetermined signal system based on the imaging signal output from the imaging unit;
The exposure time in the first section switched from the second mode to the first mode is a time shorter than the maximum exposure time in one section in the second mode.
And a method of controlling the imaging apparatus. The first mode for imaging in a state where infrared light is not projected, or the first mode for imaging in a state where first infrared light having a first wavelength associated with red is projected A second section that captures an image in a state where the second infrared light having the second wavelength associated with green is projected, and a third wavelength associated with blue An imaging step of imaging a subject in a second mode of imaging in each of a third section for imaging in a state where the third infrared light is projected;
Causing the computer to execute a video output step of generating and outputting a video signal of a predetermined signal system based on the imaging signal output from the imaging unit;
The exposure time in the first section switched from the second mode to the first mode is a time shorter than the maximum exposure time in one section in the second mode.
A control program for an image pickup apparatus.

Images