JP6221911B2 - Imaging apparatus, video signal processing method, and video signal processing program - Google Patents

Description

  The present invention relates to an imaging apparatus, a video signal processing method, and a video signal processing program.

  Conventionally, in order to image a subject in an environment where there is almost no visible light such as at night, a method of irradiating the subject with infrared light using an infrared projector and imaging infrared light reflected from the subject has been used. This method is an effective imaging method when a light that emits visible light cannot be used.

  However, an image obtained by imaging a subject by this imaging method is a monochrome image. In monochrome images, it may be difficult to identify an object. If a color image can be captured even in an environment where there is no visible light, the object identification can be improved. For example, in a surveillance camera, it is desired to capture a color image even in an environment without visible light in order to improve the object identification.

  Patent Document 1 describes an imaging apparatus that can capture a color image even in an environment without visible light. In the imaging device described in Patent Document 1, an infrared projector is also used. If the technique described in Patent Document 1 is installed in the surveillance camera, it becomes possible to convert the subject into a color image and improve the object identification.

JP 2011-50049 A

  For example, a slight amount of visible light is present outdoors in the twilight hours before sunrise or after sunset, or in a state where the illumination is very dark indoors. Even though there is a small amount of visible light, there is not enough visible light to capture color images. It is necessary to irradiate.

  When infrared light for night vision imaging is projected in the presence of visible light, the visible light and the infrared light are mixed. Since the imaging apparatus described in Patent Document 1 is premised on imaging in an environment where there is no visible light, there is a problem in that a good color image cannot be captured in a state where visible light and infrared light are mixed. is there.

  SUMMARY OF THE INVENTION In view of such problems, the present invention provides an imaging device that can capture a good color image even when visible light and infrared light for night vision are mixed, and visible light and night vision imaging. It is an object of the present invention to provide a video signal processing method and a video signal processing program capable of generating a good color video signal based on a captured image even in a state in which infrared light for use is mixed.

  The present invention includes a color filter in which filter elements of three primary colors of red, green, and blue are arranged in a predetermined arrangement in a plane, and are associated with red, in order to solve the above-described problems of the related art. The first infrared light having the first wavelength is projected, the subject is imaged to generate the first frame of the video signal, and the second having the second wavelength associated with green The subject is imaged in a state where the infrared light is projected to generate a second frame of the video signal, and the third infrared light having the third wavelength associated with the blue color is projected. An imaging unit that captures an image of a subject and generates a third frame of a video signal, and red pixel data at the same pixel position in the first to third frames and a green pixel at the same pixel position. Blue pixel data at the same pixel position as the pixel data Generated by the same-position pixel addition unit and the same-position pixel addition unit that individually add and generate red first addition pixel data, green second addition pixel data, and blue third addition pixel data The three primary color pixel data of the first added pixel data, the second added pixel data, and the third added pixel data are arranged in the same arrangement as the filter elements in the color filter. Based on the frame of the synthesized video signal generated by the synthesizing unit that generates the synthesized video signal synthesized into one frame by arranging the red pixel data at the pixel position where the red pixel data does not exist Interpolated red frame, green frame interpolated with green pixel data at pixel positions where no green pixel data exists, and blue pixel data It is subjected to demosaic processing to generate a blue frame interpolation blue pixel data to a nonexistent pixel position to provide an imaging apparatus characterized by comprising a demosaic processor for generating the three primary colors of the frame.

Further, in order to solve the above-described problems of the conventional technology, the present invention provides an image pickup unit including a color filter in which filter elements of three primary colors of red, green, and blue are arranged in a predetermined arrangement in a plane. The first infrared light having the first wavelength associated with the first infrared light is projected, and the subject is imaged to generate a first frame of the video signal. In the state where the second infrared light having the second wavelength is projected, the subject is imaged to generate a second frame of the video signal, which is correlated with blue by the imaging unit A third frame of the video signal is generated by imaging the subject in a state where the third infrared light having the third wavelength is projected, and the same pixel in the first to third frames. The same pixel position as the red pixel data of the position Green pixel data and blue pixel data at the same pixel position are individually added to obtain red first addition pixel data, green second addition pixel data, and blue third addition pixel data. The pixel data of the three primary colors of the first addition pixel data, the second addition pixel data, and the third addition pixel data are generated so as to have the same arrangement as the arrangement of the filter elements in the color filter. It generates a composite video signal by combining in one frame by aligning the front based on the frame of Kigo adult video signal, and red interpolated red pixel data to the pixel position where the red pixel data is not present frame A green frame obtained by interpolating green pixel data at a pixel position where no green pixel data exists, and a blue pixel data at a pixel position where no blue pixel data exists. It is subjected to demosaic processing to generate the frame blue interpolated, and provides a video signal processing method characterized by generating three primary colors of the frame.

  Furthermore, in order to solve the above-described problems of the related art, the present invention provides a computer in which a first infrared light having a first wavelength associated with red is projected, Pixel data constituting a first frame of a video signal generated by an imaging unit including a color filter in which filter elements of three primary colors of green and blue are arranged in a plane in a predetermined arrangement image a subject. An image signal generated by the imaging unit imaging the subject in a state in which the second infrared light having the second wavelength associated with green is projected Acquiring the pixel data constituting the second frame, and imaging the subject in a state where the third infrared light having the third wavelength associated with the blue color is projected. Generate Obtaining pixel data constituting the third frame of the received video signal, red pixel data at the same pixel position and green pixel data at the same pixel position in the first to third frames, Adding blue pixel data at the same pixel position to each other to generate red first addition pixel data, green second addition pixel data, and blue third addition pixel data; The pixel data of the three primary colors of the first addition pixel data, the second addition pixel data, and the third addition pixel data are arranged so as to be the same as the arrangement of the filter elements in the color filter. Generating a composite video signal synthesized into one frame by the step, and based on the frame of the composite video signal, at a pixel position where no red pixel data exists. Interpolate color pixel data with red frame, green pixel data with green pixel data interpolated at pixel positions where no green pixel data exists, and blue pixel data interpolate at pixel positions without blue pixel data And a step of generating a frame of three primary colors by performing a demosaic process for generating the blue frame.

  According to the imaging apparatus of the present invention, it is possible to capture a good color image even when visible light and infrared light for night vision imaging are mixed. According to the video signal processing method and the video signal processing program of the present invention, even when visible light and infrared light for night vision are mixed, a good color video signal is generated based on the captured video. can do.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to an embodiment. It is a figure which shows an example of the arrangement | sequence of the filter element in the color filter used for the imaging device of one Embodiment. It is a characteristic view which shows the spectral sensitivity characteristic of the wavelength of three primary color lights and relative sensitivity in the imaging part which comprises the imaging device of one Embodiment. It is a characteristic view showing the relationship between the wavelength and the relative detection rate when the reflectance of the three primary colors from a predetermined substance is multiplied by the light receiving sensitivity of silicon. It is a block diagram which shows the specific structural example of the front signal process part 52 in FIG. It is a figure which shows the relationship between exposure and the flame | frame of a video signal when the imaging device of one Embodiment is operate | moving in normal mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in normal mode. It is a figure which shows the relationship between exposure and the flame | frame of a video signal when the imaging device of one Embodiment is operate | moving in intermediate | middle mode and night vision mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in 1st intermediate | middle mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 1st intermediate | middle mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in 2nd intermediate mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 2nd intermediate mode. It is a figure for demonstrating the addition process of the surrounding pixel when the imaging device of one Embodiment is operate | moving in night vision mode. It is a figure which shows the 1st example of the weighting in the addition process of a surrounding pixel. It is a figure which shows the 2nd example of the weighting in the addition process of a surrounding pixel. It is a figure which shows the 3rd example of the weighting in the addition process of a surrounding pixel. It is a figure which shows the flame | frame in which the addition process of the surrounding pixel was performed. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in 1st night-vision mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 1st night-vision mode. It is a figure for demonstrating the front signal process when the imaging device of one Embodiment is operate | moving in 2nd night-vision mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 2nd night-vision mode. It is a figure for demonstrating the example of the mode switching in the imaging device of one Embodiment. It is a figure which shows the state of each part when the imaging device of one Embodiment is set to each mode. It is a partial block diagram which shows the 1st modification of the imaging device of one Embodiment. It is a partial block diagram which shows the 2nd modification of the imaging device of one Embodiment. It is a partial block diagram which shows the 3rd modification of the imaging device of one Embodiment. It is a flowchart which shows the video signal processing method regarding mode switching. It is a flowchart which shows the specific process of the normal mode shown to step S3 in FIG. It is a flowchart which shows the specific process of the intermediate | middle mode of step S4 in FIG. It is a flowchart which shows the specific process of the night vision mode of step S5 in FIG. It is a flowchart which shows the process which the video signal processing program regarding mode switching performs a computer. FIG. 4 is a characteristic diagram for considering characteristics of spectral sensitivity characteristics shown in FIG. 3. It is a figure for demonstrating the exposure amount when infrared light is projected in the state where visible light is the main. It is a figure for demonstrating the exposure amount when infrared light is projected in the state where infrared light is main. It is a conceptual explanatory drawing of the determination part which implement | achieves the 1st example of the determination method of the relationship between the quantity of visible light, and the quantity of infrared light. It is a flowchart which shows the 1st example of the determination method. It is a flowchart which shows the process in which a color gain control part controls a color gain setting part based on the determination result. It is a figure for demonstrating the exposure amount when the infrared light which varied the light emission power of the infrared light of a one part wavelength in the state where infrared light is main, and is projected. It is a conceptual explanatory drawing of the determination part which implement | achieves the 2nd example of the determination method of the relationship between the quantity of visible light, and the quantity of infrared light. It is a figure for demonstrating the difference value produced | generated in a determination part when infrared light is projected in the state where visible light is main. It is a figure for demonstrating the difference value produced | generated in a determination part, when infrared light is projected in the state where infrared light is main. It is a flowchart which shows the 2nd example of the determination method. It is a flowchart which shows the video signal processing method according to determination of the relationship of light quantity. It is a flowchart which shows the process which a video signal processing program which controls operation | movement of an imaging device including determination of the relationship of light quantity makes a computer perform. It is a figure for demonstrating the group of the color gain which the imaging device of one Embodiment uses in each mode.

  Hereinafter, an imaging device, a video signal processing method, and a video signal processing program according to an embodiment will be described with reference to the accompanying drawings.

<Configuration of imaging device>
First, the overall configuration of an imaging apparatus according to an embodiment will be described with reference to FIG. An imaging apparatus according to an embodiment shown in FIG. 1 includes a normal mode suitable for an environment where there is sufficient visible light such as daytime, a night vision mode suitable for an environment where there is almost no visible light such as nighttime, and a visible mode. This is an imaging device capable of imaging in three modes, an intermediate mode suitable for an environment where light is slightly present.

  The intermediate mode is a first infrared light projection mode in which imaging is performed while projecting infrared light in an environment with little visible light. The night vision mode is a second infrared light projection mode in which imaging is performed while projecting infrared light in an environment where there is even less (almost) no visible light.

  In FIG. 1, the light indicated by the alternate long and short dash line reflected from the subject is collected by the optical lens 1. Here, the optical lens 1 has visible light in an environment in which visible light is sufficiently present, and infrared light in which an object reflects infrared light emitted from an infrared projector 9 described below in an environment in which there is almost no visible light. Is incident.

  Under an environment where there is a slight amount of visible light, the optical lens 1 is incident with light in which visible light and infrared light emitted from the infrared projector 9 are reflected by the subject.

  Although only one optical lens 1 is shown in FIG. 1 for the sake of simplicity, the imaging apparatus actually includes a plurality of optical lenses.

  An optical filter 2 is provided between the optical lens 1 and the imaging unit 3. The optical filter 2 has two parts, an infrared cut filter 21 and a dummy glass 22. The optical filter 2 includes a state in which an infrared cut filter 21 is inserted between the optical lens 1 and the imaging unit 3 and a state in which a dummy glass 22 is inserted between the optical lens 1 and the imaging unit 3 by the driving unit 8. It is driven to either state.

  The imaging unit 3 includes an imaging element 31 in which a plurality of light receiving elements (pixels) are arranged in the horizontal direction and the vertical direction, and any one of red (R), green (G), and blue (B) corresponding to each light receiving element. And a color filter 32 in which filter elements of these colors are arranged. The image sensor 31 may be a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  As an example, as shown in FIG. 2, the color filter 32 has R, G, and B color filter elements arranged in an array called a Bayer array. The Bayer array is an example of a predetermined array of R, G, and B filter elements. In FIG. 2, the G filter element sandwiched between the R filter elements in each row is Gr, and the G filter element sandwiched between the B filter elements is Gb.

  In the Bayer array, horizontal rows in which R filter elements and Gr filter elements are alternately arranged and horizontal rows in which B filter elements and Gb filter elements are alternately arranged are vertically arranged. They are arranged alternately in the direction.

  FIG. 3 shows spectral sensitivity characteristics of the wavelengths of R light, G light, and B light and relative sensitivity in the imaging unit 3. The relative sensitivity is normalized to 1 at the maximum value. When the imaging apparatus is operated in the normal mode, it is necessary to cut infrared light having a wavelength of 700 nm or more in order to capture a good color image by visible light.

  Therefore, the drive unit 8 drives the optical filter 2 so that the infrared cut filter 21 is inserted between the optical lens 1 and the imaging unit 3 based on control by the control unit 7.

  As can be seen from FIG. 3, the imaging unit 3 has sensitivity even in an infrared light region having a wavelength of 700 nm or more. Therefore, when the imaging device is operated in the intermediate mode or the night vision mode, the driving unit 8 removes the infrared cut filter 21 between the optical lens 1 and the imaging unit 3 based on the control by the control unit 7 and the dummy glass. The optical filter 2 is driven so that 22 is inserted.

  In a state where the dummy glass 22 is inserted between the optical lens 1 and the imaging unit 3, infrared light having a wavelength of 700 nm or more is not cut. Therefore, the imaging apparatus can obtain R, G, and B color information by using the sensitivity of the portion surrounded by the dashed ellipse in FIG. The reason why the dummy glass 22 is inserted is to make the optical path length the same as the optical path length when the infrared cut filter 21 is inserted.

  The infrared projector 9 includes projectors 91, 92, and 93 that project infrared light having wavelengths IR1, IR2, and IR3, respectively. In the intermediate mode or the night vision mode, the light projecting control unit 71 in the control unit 7 performs control so as to selectively project infrared light with wavelengths IR1 to IR3 from the light projecting units 91 to 93 in a time division manner. .

  Incidentally, a silicon wafer is used for the image sensor 31. FIG. 4 shows the relationship between the wavelength and the relative detection rate when the reflectance at each wavelength is multiplied by the light receiving sensitivity of silicon when a material exhibiting each color of R, G, and B is irradiated with white light. Yes. Also in FIG. 4, the maximum value of the relative detection rate is normalized to 1.

  As shown in FIG. 4, in the infrared light region, for example, the reflected light at a wavelength of 780 nm is highly correlated with the reflected light of a material exhibiting an R color, and the reflected light at a wavelength of 870 nm is a reflection of a material exhibiting a B color. Correlation with light is high, and reflected light at a wavelength of 940 nm is highly correlated with reflected light of a material exhibiting G color.

  Therefore, in this embodiment, the wavelengths IR1, IR2, and IR3 of the infrared light projected by the light projecting units 91, 92, and 93 are set to 780 nm, 940 nm, and 870 nm. These wavelengths are examples of the wavelengths IR1 to IR3, and may be other than 780 nm, 940 nm, and 870 nm.

  The light projecting unit 91 irradiates the subject with infrared light having the wavelength IR1, and assigns an image signal obtained by imaging the light reflected from the subject to the R signal. The light projecting unit 93 irradiates the subject with infrared light having a wavelength IR2, and assigns a video signal obtained by imaging the light reflected from the subject to the G signal. The light projecting unit 92 irradiates the subject with infrared light having a wavelength IR3, and assigns a video signal obtained by imaging light reflected from the subject to the B signal.

  In this way, in principle, even in the intermediate mode or the night vision mode, it is possible to reproduce the same color as when the subject is imaged in an environment where visible light exists in the normal mode.

  Although the color image is different from the actual color of the subject, a wavelength IR1 of 780 nm may be assigned to the R light, a wavelength IR3 of 870 nm may be assigned to the G light, and a wavelength IR2 of 940 nm may be assigned to the B light. The wavelengths IR1, IR2, and IR3 can be arbitrarily assigned to the R light, G light, and B light.

  In the present embodiment, the wavelengths IR1, IR2, and IR3 that best reproduce the color of the subject are assigned to R light, G light, and B light, respectively.

  The control unit 7 controls imaging in the imaging unit 3, each unit in the video processing unit 5, and the video output unit 6. An imaging signal imaged by the imaging unit 3 is A / D converted by the A / D converter 4 and input to the video processing unit 5. The internal configuration and operation of the video processing unit 5 will be described later.

  The video output unit 6 includes a color gain setting unit 62 that multiplies each of three primary color data of R, G, and B described later by a predetermined color gain. The detailed operation of the color gain setting unit 62 will be described later. The color gain setting unit 62 may be provided in the video processing unit 5.

  The imaging unit 3 and the A / D converter 4 may be integrated. The video processing unit 5 and the control unit 7 may be integrated.

  The control unit 7 includes a mode switching unit 72 that switches between the normal mode, the intermediate mode, and the night vision mode. The mode switching unit 72 appropriately switches the operation in the video processing unit 5 as described later in correspondence with the normal mode, the intermediate mode, and the night vision mode.

  The control unit 7 uses a determination unit 78 that determines the relationship between the amount of visible light and the amount of infrared light in the surrounding environment, and a color that is controlled by the color gain setting unit 62 so that the color gains multiplied by the three primary color data are different. And a gain control unit 79. The term “infrared light” used here is mostly the light projected by the infrared projector 9 and reflected by the subject.

  For example, the determination unit 78 compares the amount of visible light with the amount of infrared light, the state where the visible light is mainly and the infrared light is subordinate, and the amount of infrared light is compared with the amount of visible light. What is necessary is just to determine which is the state in which infrared light is main and visible light is subordinate. The determination unit 78 determines the relationship between the amount of visible light and the amount of infrared light in the surrounding environment by calculating a specific light amount ratio between the amount of visible light and the amount of infrared light. May be.

  The ratio of the amount of light here does not have to be the ratio of the amount of visible light and the amount of infrared light itself, but the relationship (ratio, etc.) between the amount of visible light and the amount of infrared light in the surrounding environment. Any numerical value may be used as long as it changes.

  When the relationship between the amount of visible light and the amount of infrared light in the surrounding environment is determined by calculating such numerical values, it is not always necessary to determine which is the main and which is the subordinate. Hereinafter, for convenience, the relationship between the amount of visible light and the amount of infrared light in the surrounding environment may be simply referred to as “master-slave relationship”.

  The color gain control unit 79 varies the color gains multiplied by the three primary color data by the color gain setting unit 62 in accordance with the relationship between the amount of visible light and the amount of infrared light determined by the determination unit 78. To control.

  The video processing unit 5 includes switches 51 and 53, a previous signal processing unit 52, and a demosaic processing unit 54. The switches 51 and 53 may be physical switches, or may be conceptual switches for switching between operation and non-operation of the previous signal processing unit 52. A video signal is input from the video processing unit 5 to the control unit 7 in order to detect the brightness of the video being captured.

  The video data input to the previous signal processing unit 52 is also input to the determination unit 78. The determination unit 78 determines the master-slave relationship of the light amount based on the video data input to the previous signal processing unit 52.

  The function of the determination unit 78 may be provided in the previous signal processing unit 52. In this case, the previous signal processing unit 52 having the function of the determination unit 78 notifies the color gain control unit 79 of the determination result of the master-slave relationship of the light amount.

  As illustrated in FIG. 5, the previous signal processing unit 52 includes a surrounding pixel addition unit 521, a same-position pixel addition unit 522, and a synthesis unit 523.

  The video processing unit 5 generates R, G, and B primary color data and supplies them to the video output unit 6. The video output unit 6 outputs the three primary color data in a predetermined format to a display unit not shown.

  The video output unit 6 may output the R, G, and B signals as they are, or may convert the R, G, and B signals into luminance signals and color signals (or color difference signals) and output them. The video output unit 6 may output a composite video signal. The video output unit 6 may output a digital video signal or may output a video signal converted into an analog signal by a D / A converter.

  The color gain setting unit 62 multiplies each of the R, G, and B primary color data by a predetermined color gain in order to adjust the white balance when outputting a video signal of a predetermined format from the video processing unit 5. Here, the three primary color data are multiplied by the color gain in order to adjust the white balance, but the color gain may be multiplied in order to reproduce an image having a predetermined color temperature.

  The color gain setting unit 62 holds at least two sets of color gains to be multiplied by the three primary color data of R, G, and B. One set of color gains and another set of color gains are all different in color gain for R, G, and B data.

  The color gain setting unit 62 selects a set of color gains to be multiplied by the three primary color data under the control of the color gain control unit 79 when the imaging device is set to the intermediate mode by the mode switching unit 72. The color gain setting unit 62 multiplies the three primary color data by the selected set of color gains.

  The color gain setting unit 62 multiplies the three primary color data by a fixed set of color gains that are different between the normal mode and the night vision mode when the imaging device is set to the normal mode and the night vision mode by the mode switching unit 72. .

  Hereinafter, specific operations in the normal mode, the intermediate mode, and the night vision mode will be described.

<Normal mode>
In the normal mode, the control unit 7 causes the driving unit 8 to insert the infrared cut filter 21 between the optical lens 1 and the imaging unit 3. The light projection control unit 71 turns off infrared light projection by the infrared projector 9.

  The image signal picked up by the image pickup unit 3 is converted into video data which is a digital signal by the A / D converter 4 and input to the video processing unit 5. In the normal mode, the mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Tb.

  FIG. 6A shows exposures Ex1, Ex2, Ex3,... Actually, the exposure time varies depending on conditions such as the shutter speed. Here, the exposures Ex1, Ex2, and Ex3 indicate the maximum exposure time.

  FIG. 6B shows the timing at which each video signal frame is obtained. Based on an exposure (not shown) before the exposure Ex1, a frame F0 of the video signal is obtained after a predetermined time. Based on the exposure Ex1, a frame F1 of the video signal is obtained after a predetermined time. Based on the exposure Ex2, a frame F2 of the video signal is obtained after a predetermined time. The same applies to exposure Ex3 and thereafter. The frame frequency of the video signal is set to 30 frames / second, for example.

  The frame frequency of the video signal may be set as appropriate, such as 30 frames / second or 60 frames / second for the NTSC system and 25 frames / second or 50 frames / second for the PAL system. The frame frequency of the video signal may be 24 frames / second used in movies.

  The video data of each frame output from the A / D converter 4 is input to the demosaic processing unit 54 via the switches 51 and 53. The demosaic processing unit 54 performs demosaic processing on the input video data of each frame. The video processing unit 5 performs various other video processes other than the demosaic process and outputs R, G, and B primary color data.

  The demosaic processing in the demosaic processing unit 54 will be described with reference to FIG. 7A shows an arbitrary frame Fm of video data. The frame Fm is a frame composed of pixels in the effective video period. The number of pixels of the video data is, for example, horizontal 640 pixels and vertical 480 pixels in the VGA standard. Here, for simplification, the number of pixels of the frame Fm is significantly reduced, and the frame Fm is conceptually illustrated.

  The video data generated using the Bayer array imaging unit 3 is data in which R, G, and B pixel data are mixed in the frame Fm. The demosaic processing unit 54 generates R interpolation pixel data Ri obtained by calculating R pixel data at a pixel position where no R pixel data is present using surrounding R pixel data. The demosaic processing unit 54 generates an R frame FmR in which all the pixels in one frame shown in FIG. 7B are made up of R pixel data.

  The demosaic processing unit 54 generates G interpolation pixel data Gi obtained by calculating G pixel data at a pixel position where no G pixel data exists using surrounding G pixel data. The demosaic processing unit 54 generates a G frame FmG in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 generates B interpolated pixel data Bi obtained by calculating B pixel data at a pixel position where no B pixel data exists using surrounding B pixel data. The demosaic processing unit 54 generates a B frame FmB in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 may use at least R pixel data when interpolating R pixel data, may use at least G pixel data when interpolating G pixel data, and B pixel data. When interpolation is performed, at least B pixel data may be used. In order to improve interpolation accuracy, the demosaic processing unit 54 may use pixel data of another color different from the color of the interpolation pixel data to be generated when interpolating R, G, and B pixel data. Good.

  Since there are pixels outside the effective video period in the imaging unit 3, R, G, and B pixel data can be interpolated even in pixels located at the upper, lower, left, and right ends of the frame Fm.

  The R frame FmR, G frame FmG, and B frame FmB generated by the demosaic processing unit 54 are output as R, G, and B primary color data. In FIG. 7, the R, G, and B pixel data has been described in units of frames in order to facilitate understanding, but actually, the R, G, and B pixel data are sequentially output for each pixel.

<Intermediate mode: first intermediate mode>
In the intermediate mode (first intermediate mode and second intermediate mode described later), the control unit 7 causes the driving unit 8 to insert the dummy glass 22 between the optical lens 1 and the imaging unit 3. The light projection controller 71 turns on the infrared light projection by the infrared light projector 9. The mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Ta.

  FIG. 8A shows a state of infrared light projection by the infrared projector 9. The control unit 7 divides one frame period of the normal mode into 1/3, and controls so as to project infrared light in the order of the light projecting units 91, 92, 93, for example.

  In the example shown in FIG. 8A, infrared light having a wavelength IR1 (780 nm) is irradiated onto the subject in the first ３ period of one frame. In the next 1/3 period of one frame, the subject is irradiated with infrared light having a wavelength IR2 (940 nm). In the last 1/3 period of one frame, the subject is irradiated with infrared light having a wavelength IR3 (870 nm). The order of projecting infrared light with wavelengths IR1 to IR3 is arbitrary. However, for the following reason, the wavelength IR2 is the center, for example, the order of the wavelengths IR1, IR2, and IR3 is the best.

  In the intermediate mode and the night-vision mode, as will be described in detail later, one frame is based on three imaging signals generated by the imaging unit 3 in the state where infrared light of wavelengths IR1, IR2, and IR3 is projected. A video signal is generated. Therefore, when a moving subject is imaged, color misregistration or outline blur may occur.

  Generally, in the R, G, and B signals, the G signal has the greatest influence on the luminance signal. Therefore, it is preferable that the infrared light with the wavelength IR2 for generating the video signal (imaging signal) assigned to the G signal is set at the center, and the infrared light with the wavelengths IR1 and IR3 is projected before and after the infrared light.

  As shown in FIG. 8B, at the timing when infrared light having the wavelength IR1 is projected, the imaging unit 3 performs exposure Ex1R having a high correlation with the R light. At the timing when the infrared light having the wavelength IR2 is projected, the imaging unit 3 performs the exposure Ex1G having a high correlation with the G light. At the timing when the infrared light having the wavelength IR3 is projected, the imaging unit 3 performs exposure Ex1B having a high correlation with the B light.

  However, in the intermediate mode, since imaging is performed in an environment where there is a small amount of visible light, visible light and infrared light projected from the infrared projector 9 are mixed. Therefore, in the intermediate mode, the exposures Ex1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B... Are exposures that combine exposure with visible light and exposure with infrared light.

As shown in FIG. 8 (c), exposure Ex1R, Ex1G, based on EX1B, after a predetermined time, the exposure Ex1R frame corresponding to F1IR1, frame F1IR 2 corresponding to the exposure Ex1G, frame F1IR 3 corresponding to the exposure EX1B Is obtained.

  Further, based on the exposures Ex2R, Ex2G, and Ex2B, a frame F2IR1 corresponding to the exposure Ex2R, a frame F2IR3 corresponding to the exposure Ex2G, and a frame F2IR2 corresponding to the exposure Ex2B are obtained after a predetermined time. The same applies to exposures Ex3R, Ex3G, and Ex3B.

  The frame frequency of the imaging signal in FIG. 8C is 90 frames / second. In the intermediate mode, since one frame of the video signal in the normal mode is time-divided and infrared light of wavelengths IR1 to IR3 is projected, in order to output a video signal in the same format as in the normal mode, ( The frame frequency of the imaging signal in c) is three times the frame frequency in the normal mode.

  As will be described later, one frame of a video signal having a frame frequency of 30 frames / second shown in FIG. 8D is generated based on the imaging signal of 3 frames shown in FIG. For example, a frame F1IR is generated based on the frames F1IR1, F1IR2, and F1IR3, and a frame F2IR is generated based on the frames F2IR1, F2IR2, and F2IR3.

  The operation in the intermediate mode for generating the video signal of each frame in FIG. 8D based on the three frames of the imaging signal in FIG. 8C will be specifically described.

  Video data of each frame corresponding to the imaging signal shown in FIG. 8C output from the A / D converter 4 is input to the previous signal processing unit 52 via the switch 51.

  The previous signal processing in the previous signal processing unit 52 will be described with reference to FIG. FIG. 9A shows an arbitrary frame FmIR1 of the video data generated at the timing when the infrared light having the wavelength IR1 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR1 is attached with a subscript 1 indicating that it is generated in a state where infrared light having the wavelength IR1 is projected.

  FIG. 9B shows an arbitrary frame FmIR2 of video data generated at the timing when infrared light having a wavelength IR2 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR2 is attached with a subscript 2 indicating that it is generated in a state where infrared light having a wavelength IR2 is projected.

  FIG. 9C shows an arbitrary frame FmIR3 of the video data generated at the timing when the infrared light having the wavelength IR3 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR3 has a subscript 3 indicating that it is generated in a state where infrared light having the wavelength IR3 is projected.

  The frame FmIR1 shown in FIG. 9A is video data generated in a state where infrared light having a wavelength IR1 having a high correlation with the R light is projected. Therefore, the R pixel data is projected. The pixel data corresponding to the infrared light and the pixel data B and G are pixel data not corresponding to the projected infrared light. The hatching attached to the B, Gr, and Gb pixel data means that the pixel data does not correspond to the projected infrared light.

  The frame FmIR2 shown in FIG. 9B is video data generated in a state where infrared light having a wavelength IR2 having a high correlation with the G light is projected, so that the G pixel data is projected. The pixel data corresponding to the infrared light, and the R and B pixel data are pixel data not corresponding to the projected infrared light. The hatching attached to the R and B pixel data means that the pixel data does not correspond to the projected infrared light.

  The frame FmIR3 shown in FIG. 9C is video data generated in a state where infrared light having a wavelength IR3 having a high correlation with the B light is projected, so that the B pixel data is projected. The pixel data corresponding to the infrared light, and the R and G pixel data are pixel data not corresponding to the projected infrared light. The hatching attached to the R, Gr, and Gb pixel data means pixel data that does not correspond to the projected infrared light.

  The same-position pixel addition unit 522 in the previous signal processing unit 52 individually adds R, Gr, Gb, and B pixel data at the same pixel position according to the following formulas (1) to (3), and adds pixels Data R123, Gr123, Gb123, and B123 are generated. In the intermediate mode, the surrounding pixel addition unit 521 in the previous signal processing unit 52 does not operate.

R123 = ka × R1 + kb × R2 + kc × R3 (1)
G123 = kd × G1 + ke × G2 + kf × G3 (2)
B123 = kg × B1 + kh × B2 + ki × B3 (3)

  In the equations (1) to (3), R1, G1, and B1 are R, G, and B pixel data in the frame FmIR1, R2, G2, and B2 are R, G, and B pixel data in the frame FmIR2, R3, G3, B3 is R, G, B pixel data in the frame FmIR3. ka to ki are predetermined coefficients. G123 in Formula (2) is Gr123 or Gb123.

  At this time, the same-position pixel adding unit 522 adds R, Gr, Gb, and B pixel data at the same pixel position with hatching to the respective pixel data with R, Gr, Gb, and B that are not hatched. Is added.

  That is, the same-position pixel adding unit 522 adds the R pixel data at the same pixel position in the frames FmIR2 and FmIR3 to the R pixel data in the frame FmIR1 based on the equation (1), and adds the added pixel data R123. Generate. In other words, the addition pixel data R123 for red is generated using only the pixel data of the region corresponding to the red color filter in the light receiving element.

  The same-position pixel addition unit 522 adds the pixel data of Gr and Gb at the same pixel position in the frames FmIR1 and FmIR3 to the pixel data of Gr and Gb in the frame FmIR2 based on the equation (2), and adds pixel data Generate G123. That is, the green summed pixel data G123 is generated using only the pixel data of the region corresponding to the green color filter in the light receiving element.

  The same-position pixel addition unit 522 adds the B pixel data at the same pixel position in the frames FmIR1 and FmIR2 to the B pixel data in the frame FmIR3 based on the equation (3) to generate the added pixel data B123. . That is, the blue addition pixel data B123 is generated using only the pixel data of the region corresponding to the blue color filter in the light receiving element.

  The combining unit 523 in the previous signal processing unit 52 generates a frame FmIR123 of the combined video signal shown in (d) of FIG. 9 based on the added pixel data R123, Gr123, Gb123, and B123 generated at each pixel position. To do.

  Specifically, the synthesis unit 523 selects and synthesizes the addition pixel data R123 in the frame FmIR1, the addition pixel data Gr123 and Gb123 in the frame FmIR2, and the addition pixel data B123 in the frame FmIR3. Thereby, the synthesis unit 523 generates a frame FmIR123 of the synthesized video signal.

  In this way, the synthesis unit 523 generates a frame FmIR123 in which the addition pixel data R123, Gr123, Gb123, and B123 are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  In the first intermediate mode, video data of the frame FmIR123 is generated using pixel data that is not hatched and pixel data that is hatched.

  The reason why the pixel data at the same pixel position is added by the same position pixel addition unit 522 is as follows. Since the imaging is performed in an environment where visible light is present in the intermediate mode, the hatched pixel data includes respective color components based on exposure with visible light. Therefore, the sensitivity of each color can be increased by adding pixel data at the same pixel position.

  If the visible light and infrared light are mixed and there is a relatively large amount of visible light, exposure with visible light becomes dominant. In this case, the video data of the frame FmIR123 mainly includes components based on the video signal exposed with visible light. If visible light and infrared light are mixed and there is a relatively large amount of infrared light, exposure with infrared light becomes dominant. In this case, the video data of the frame FmIR123 mainly includes components based on the video signal exposed with infrared light.

  When visible light is relatively small, the relationship between the coefficients ka, kb, and kc in equation (1) is ka> kb, kc, and the relationship between the coefficients kd, ke, and kf in equation (2). , Kf> kd, ke, and in equation (3), the magnitude relationship between the coefficients kg, kh, ki is preferably kh> kg, ki. This is because the wavelength IR1 has a high correlation with the R light, the wavelength IR2 has a high correlation with the G light, and the wavelength IR3 has a high correlation with the B light.

  In this way, the R pixel data is mainly the R pixel data in the frame FmIR1, the G pixel data is the G pixel data in the frame FmIR2, and the B pixel data is mainly the B pixel data in the frame FmIR3. it can.

  The video data of the frame FmIR123 output from the previous signal processing unit 52 is input to the demosaic processing unit 54 via the switch 53. The demosaic processing unit 54 performs demosaic processing on the video data of the input frame FmIR123, as in the normal mode. The video processing unit 5 performs various other video processes other than the demosaic process and outputs R, G, and B primary color data.

  The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 10A shows a frame FmIR123. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R123i. The demosaic processing unit 54 generates an R frame FmIR123R in which all the pixels in one frame shown in FIG.

  The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding G pixel data, and generates the G interpolation pixel data G123i. The demosaic processing unit 54 generates a G frame FmIR123G in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B123i. The demosaic processing unit 54 generates a B frame FmIR123B in which all the pixels in one frame shown in FIG.

  As can be understood by comparing the operation of the demosaic processing unit 54 shown in FIG. 7 in the normal mode with the operation of the demosaic processing unit 54 shown in FIG. 10 in the intermediate mode, they are substantially the same. The operation of the demosaic processing unit 54 does not change regardless of whether it is the normal mode or the intermediate mode.

  In the normal mode, the previous signal processing unit 52 is disabled, and in the intermediate mode, the previous signal processing unit 52 may be operated except for the surrounding pixel adding unit 521. In the normal mode and the intermediate mode, the signal processing unit such as the demosaic processing unit 54 in the video processing unit 5 can be shared.

<Intermediate mode: second intermediate mode>
The operation in the second intermediate mode will be described with reference to FIGS. In the operation in the second intermediate mode, the description of the same part as the operation in the first intermediate mode is omitted. The frames FmIR1, FmIR2, and FmIR3 in (a) to (c) of FIG. 11 are the same as the frames FmIR1, FmIR2, and FmIR3 in (a) to (c) of FIG.

  The combining unit 523 selects and combines R1 which is R pixel data in the frame FmIR1, Gr2 and Gb2 which are G pixel data in the frame FmIR2, and B3 which is B pixel data in the frame FmIR3. As a result, the synthesis unit 523 generates a frame FmIR123 'of the synthesized video signal shown in FIG.

That is, the frame FmIR123 'is a frame FmIR1, FmIR 2, not hatched in FmIR 3 R, Gr, Gb, video data is gathered pixel data of B in one frame.

  That is, in the frame FmIR123 ′, the pixel data for red using only the pixel data in the region corresponding to the red color filter in the state where the infrared light having the wavelength IR1 is projected, and the infrared light having the wavelength IR2 are projected. Only the pixel data for the green color using only the pixel data of the region corresponding to the green color filter in the selected state, and the pixel data of the region corresponding to the blue color filter in the state of projecting the infrared light having the wavelength IR3 are used. The pixel data is for blue.

  As described above, the synthesis unit 523 generates a frame FmIR123 'in which the pixel data R1, Gr2, Gb2, and B3 are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  In the second intermediate mode, the same-position pixel adding unit 522 sets the coefficient ka in Equation (1) to 1, the coefficients kb and kc to 0, the coefficient ke in Equation (2) to 1, the coefficients kd and kf to 0, In equation (3), the coefficient ki is 1 and the coefficients kg and kh are 0.

  As a result, the R pixel data in the frame FmIR1, the Gr and Gb pixel data in the frame FmIR2, and the B pixel data in the frame FmIR3 have the same values.

  Therefore, similarly to the operation in the first intermediate mode, the combining unit 523 selects the R pixel data in the frame FmIR1, the Gr and Gb pixel data in the frame FmIR2, and the B pixel data in the frame FmIR3. FmIR123 'can be generated.

  In the second intermediate mode, the front signal processing unit 52 generates pixel data (hatched) with infrared light projected to generate pixel data having the same color as the pixel data. Video data of the frame FmIR123 ′ is generated using only (non-pixel data).

  According to the second intermediate mode, the sensitivity and color reproducibility are lower than those in the first intermediate mode, but the arithmetic processing can be simplified and the frame memory can be reduced.

  The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 12A shows the frame FmIR123 '. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R1i. The demosaic processing unit 54 generates an R frame FmIR123′R in which all the pixels in one frame shown in FIG. 12B are made up of R pixel data.

  The demosaic processing unit 54 calculates G pixel data at a pixel position where no G pixel data exists using the surrounding G pixel data, and generates G interpolation pixel data G2i. The demosaic processing unit 54 generates a G frame FmIR123′G in which all the pixels in one frame shown in FIG.

  The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B3i. The demosaic processing unit 54 generates a B frame FmIR123′B in which all the pixels of one frame shown in FIG.

  As described above, in the intermediate mode, the pixel data for red is generated from the pixel data obtained from the region corresponding to the red color filter in the light receiving element, and obtained from the region corresponding to the green color filter in the light receiving element. Green pixel data is generated from the pixel data, and blue pixel data is generated from the pixel data obtained from the region corresponding to the blue color filter in the light receiving element.

<Night Vision Mode: First Night Vision Mode>
In the night-vision mode (the first night-vision mode and the second night-vision mode described later), as in the intermediate mode, the control unit 7 causes the drive unit 8 to place the dummy glass 22 between the optical lens 1 and the imaging unit 3. Insert it. The light projection controller 71 turns on the infrared light projection by the infrared light projector 9. The mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Ta.

  The schematic operation in the night vision mode is the same as that in FIG. However, in the night vision mode, since the imaging is performed in an environment where there is almost no visible light, the exposures Ex1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B... In FIG. Is assumed.

  In an environment where there is almost no visible light and only infrared light, there is no difference in the characteristics of the filter elements in the color filter 32, so the imaging unit 3 can be regarded as a monochromatic imaging device. .

  Therefore, in the night vision mode, the surrounding pixel addition unit 521 in the previous signal processing unit 52 adds pixel data located in the periphery to each pixel data in order to improve the sensitivity of infrared light.

  Specifically, as illustrated in FIG. 13A, when the R pixel is the target pixel, the surrounding pixel adding unit 521 includes the G (Gr) positioned around the R pixel data of the target pixel. , Gb) and B pixel data of 8 pixels are added.

  In other words, in the intermediate mode, red pixel data was generated from the pixel data obtained from the region corresponding to the red color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. Pixel data for red is generated from the pixel data obtained from the above. In the example shown in FIGS. 13A to 13C, pixel data obtained from an area corresponding to nine pixels including the target pixel is used for each color.

  As shown in FIG. 13B, when the Gr pixel is the target pixel, the surrounding pixel adding unit 521 has eight R, Gb, and B pixels positioned around the pixel data of the target pixel Gr. Are added. As illustrated in FIG. 13C, when the Gb pixel is the target pixel, the surrounding pixel adding unit 521 has eight R, Gr, and B pixels positioned around the Gb pixel data of the target pixel. Are added.

  In other words, in the intermediate mode, the pixel data for green was generated from the pixel data obtained from the area corresponding to the green color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. The pixel data for green is generated from the pixel data obtained from the above.

  As illustrated in FIG. 13D, when the B pixel is the target pixel, the surrounding pixel adding unit 521 has eight R and G pixels located around the B pixel data of the target pixel. Add data.

  In other words, in the intermediate mode, blue pixel data was generated from the pixel data obtained from the region corresponding to the blue color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. The pixel data for blue is generated from the pixel data obtained from the above.

  The surrounding pixel adding unit 521 may simply add 9 pixels of the pixel data of the target pixel and the surrounding 8 pixel pixel data, or may apply a predetermined weight to the surrounding 8 pixel pixel data. May be added to the pixel data of the pixel of interest.

  Here, an example of weighting the pixel data of the surrounding eight pixels will be described. As can be seen from FIG. 3 or FIG. 32 described later, the sensitivities of R, G, and B are almost the same at the wavelength IR2 assigned to the G light and the wavelength IR3 assigned to the B light.

  Therefore, as shown in FIGS. 13B to 13D, when the G or B pixel is the target pixel, the surrounding pixel adding unit 521 includes the eight pixels located around the G or B pixel data. When pixel data is added without weighting, the sensitivity is almost nine times.

  However, as can be seen from FIG. 3 or FIG. 32 described later, the sensitivities of R, G, and B are greatly different at the wavelength IR1 assigned to R.

  Therefore, as shown in FIG. 13A, when the R pixel is the target pixel, the surrounding pixel adding unit 521 adds the pixel data of 8 pixels located in the periphery to the R pixel data without weighting. The sensitivity is not nearly 9 times, but a multiple smaller than 9 times.

  If the surrounding pixel adding unit 521 adds the pixel data of the surrounding 8 pixels to the pixel data of the target pixel without weighting, the R signal and the G and B signals have different sensitivities. Will shift.

  Therefore, it is preferable that the surrounding pixel addition unit 521 performs predetermined weighting on the pixel data of the surrounding eight pixels and adds the pixel data to the pixel data of the target pixel. The surrounding pixel adding unit 521 may make the weighting different when the R pixel is the target pixel and when the G and B pixels are the target pixel. There may be cases where the weighting is different between when the G pixel is the target pixel and when the B pixel is the target pixel.

  FIG. 14 shows a first example of weighting. The numbers attached below the R, Gb, Gr, and B pixels shown in FIG. 14 are weighting coefficients.

R that put the imaging unit 3, G, is the ratio of the spectral sensitivity of B, the wavelength IR1, R: G: B = 1: 0.8: 0.5, in the wavelength IR2 and IR3, R: G: B = Let it be 1: 1: 1. As an example, the basic weighting coefficient is set as pixel of interest: top / bottom / left / right pixels: diagonal pixels = 100: 70: 50.

  In this case, as shown in FIG. 14, when the Gr, Gb, and B pixels are the target pixel, the pixel data of the surrounding eight pixels may be multiplied by a basic weighting coefficient.

  When the R pixel is the pixel of interest, infrared light having a wavelength IR1 is projected, and therefore, the weighting coefficient is preferably set in consideration of the above-described ratio of R, G, and B spectral sensitivities. At the wavelength IR1, the spectral sensitivity of R / the spectral sensitivity of G is from 1 / 0.8 to 1.25, and the spectral sensitivity of R / B is from 1 / 0.5 to 2.0.

  Therefore, as shown in FIG. 14, when the R pixel is the target pixel, 87 is used as the weighting factor from 70 × 1.25 for the upper, lower, left, and right G pixel data, and the B pixel data in the diagonal direction is used. On the other hand, it is better to set 100 as a weighting coefficient from 50 × 2.0.

  The ratio of the spectral sensitivity of R, G, B in the imaging unit 3 varies depending on the type of imaging element 31 used and the manufacturer. It is preferable to set a weighting coefficient corresponding to each image sensor 31.

  FIG. 15 shows a second example of weighting. The ratio of the spectral sensitivities of R, G, and B in the imaging unit 3 is R: G: B = 1: 0.8: 0.5 at the wavelength IR1, and R: G: B = 1: 1: at the wavelength IR2. 1. It is assumed that R: G: B = 1: 0.95: 1 at the wavelength IR3. The basic weighting coefficient is the same as in the first example.

  In this case, as shown in FIG. 15, when the Gr and Gb pixels are the target pixel, the pixel data of the surrounding eight pixels may be multiplied by a basic weighting coefficient.

  When the R pixel is the pixel of interest, it is preferable to use a weighting factor that takes into account the ratio of R, G, and B spectral sensitivities, as in the first example. As shown in FIG. 15, the weighting coefficient when the R pixel is the target pixel is the same as that in the first example.

  When the B pixel is the pixel of interest, infrared light having a wavelength IR3 is projected, so it is preferable to set the weighting coefficient in consideration of the above-described ratio of R, G, and B spectral sensitivities. At the wavelength IR3, the B spectral sensitivity / G spectral sensitivity is 1.05 to 1.05.

  Therefore, as shown in FIG. 15, when the B pixel is the target pixel, 73 is preferably used as the weighting coefficient from 70 × 1.05 to the upper, lower, left, and right G pixel data. The weighting coefficient for the R pixel data may be 50, which is the basic weighting coefficient.

  The weighting for the pixel data of the surrounding 8 pixels may be partially 0. That is, the addition of the pixel data of the surrounding pixels to the pixel data of the target pixel in the surrounding pixel addition unit 521 is not limited to adding all the pixel data of 8 pixels.

  For example, the surrounding pixel addition unit 521 does not need to add pixel data of pixels positioned in the diagonal direction and add pixel data of pixels positioned in the vertical and horizontal directions to the pixel data of the target pixel.

  FIG. 16 is a third example of weighting, and shows an example in which the weighting coefficient for pixels in an oblique direction is set to zero. As an example, the basic weighting coefficient is set as pixel of interest: vertical and horizontal pixels: diagonal pixels = 100: 50: 0.

  The ratio of the spectral sensitivities of R, G, and B in the imaging unit 3 is the same as that in the first example. In this case, as shown in FIG. 16, when the R pixel is the target pixel, 62 is preferably used as the weighting coefficient from 50 × 1.25 for the upper, lower, left, and right G pixel data.

  The numerical values of the weighting coefficients shown in FIGS. 14 to 16 are merely examples for facilitating understanding of the weighting for surrounding pixels. The actual weighting factor may be set as appropriate in consideration of the dynamic range of the video signal output from the video output unit 6.

  By the way, there is an image pickup device that can read out a plurality of pixels called binning as a single pixel. When an image sensor having a binning function is used as the image sensor 31, an addition process by an image sensor having a binning function may be performed instead of the addition process by the surrounding pixel addition unit 521. Binning by the image sensor is substantially equivalent to addition processing by the surrounding pixel addition unit 521.

Frame FmIR1, FmIR 2, FmIR 3 of (a) ~ (c) Fig. 17 is the same as the frame FmIR1, FmIR 2, FmIR 3 of (a) ~ (c) of FIG. 9. In (d) to (f) of FIG. 17, R1ad, Gr1ad, Gb1ad, B1ad, R2ad, Gr2ad, Gb2ad, B2ad, R3ad, Gr3ad, Gb3ad, and B3ad are converted into R, Gr, Gb, and B pixel data, respectively. On the other hand, it is added pixel data obtained by adding pixel data of surrounding eight pixels.

Peripheral pixel adding unit 521, a frame FmIR1, FmIR 2, by performing the addition process shown in FIG. 13 for each of the pixel data of FmIR 3, frame FmIR1ad shown in (d) ~ (f) of FIG. 17, FmIR2ad , FmIR3ad is generated.

  The frames FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG. 18 are the same as the frames FmIR1ad, FmIR2ad, and FmIR3ad in (d) to (f) of FIG.

  Similarly to the first intermediate mode, the same-position pixel addition unit 522 adds the pixel data of R2ad and R3ad at the same pixel position in the frames FmIR2ad and FmIR3ad to the pixel data of R1ad in the frame FmIR1ad based on Expression (1). Then, the addition pixel data R123ad is generated.

  The same-position pixel addition unit 522 adds the pixel data of Gr1ad, Gb1ad, Gr3ad, and Gb3ad at the same pixel position in the frames FmIR1ad and FmIR3ad to the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad based on Expression (2). The addition pixel data Gr123ad and Gb123ad are generated.

  The same-position pixel addition unit 522 adds the pixel data of B1ad and B2ad at the same pixel position in the frames FmIR1ad and FmIR2ad to the pixel data of B3ad in the frame FmIR3ad based on the equation (3), and adds the added pixel data B123ad. Generate.

  Similar to the first intermediate mode, the combining unit 523 selects and combines the addition pixel data R123ad in the frame FmIR1ad, the addition pixel data Gr123ad and Gb123ad in the frame FmIR2ad, and the addition pixel data B123ad in the frame FmIR3ad. As a result, the synthesis unit 523 generates a frame FmIR123ad of the synthesized video signal shown in FIG.

  The synthesizing unit 523 generates a frame FmIR123ad in which the addition pixel data R123ad, Gr123ad, Gb123ad, and B123ad are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  FIG. 19A shows a frame FmIR123ad. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R123adi. The demosaic processing unit 54 generates an R frame FmIR123adR in which all pixels of one frame shown in FIG. 19B are made up of R pixel data.

  The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding G pixel data, and generates the G interpolation pixel data G123adi. The demosaic processing unit 54 generates a G frame FmIR123adG in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B123adi. The demosaic processing unit 54 generates a B frame FmIR123adB in which all the pixels of one frame shown in FIG.

  The first intermediate mode and the first night-vision mode are different in that the former does not operate the surrounding pixel adding unit 521 while the latter operates the surrounding pixel adding unit 521. The mode switching unit 72 may operate the surrounding pixel addition unit 521 in the night vision mode.

  The operation of the demosaic processing unit 54 in the night vision mode is substantially the same as the operation of the demosaic processing unit 54 in the normal mode and the intermediate mode. The signal processing unit such as the demosaic processing unit 54 in the video processing unit 5 can be shared in the normal mode, the intermediate mode, and the night vision mode.

<Night Vision Mode: Second Night Vision Mode>
The operation in the second night-vision mode will be described with reference to FIGS. In the operation in the second night vision mode, the description of the same part as the operation in the first night vision mode is omitted. The frames FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG. 20 are the same as the FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG.

  The combining unit 523 selects and combines R1ad that is R pixel data in the frame FmIR1ad, Gr2ad and Gb2ad that are G pixel data in the frame FmIR2, and B3ad that is B pixel data in the frame FmIR3. As a result, the synthesis unit 523 generates a frame FmIR123′ad of the synthesized video signal shown in FIG.

  The synthesizing unit 523 generates a frame FmIR123′ad in which the pixel data R1ad, Gr2ad, Gb2ad, and B3ad are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  As described with reference to FIG. 13, the pixel data R1ad for red in the frame FmIR123'ad is obtained from a wider area than the area used for generating the pixel data for red in the intermediate mode. It is generated from pixel data.

  In addition, the additional pixel data Gr2ad for green in the frame FmIR123'ad is generated from pixel data obtained from an area larger than the area used for generating the pixel data for green in the intermediate mode. ing.

  Furthermore, the additional pixel data B3ad for blue in the frame FmIR123'ad is generated from pixel data obtained from an area larger than the area used for generating the pixel data for blue in the intermediate mode. ing.

  In the second night-vision mode, as in the second intermediate mode, the same-position pixel addition unit 522 sets the coefficient ka in equation (1) to 1, the coefficients kb and kc to 0, and the coefficient ke in equation (2) to 1. The coefficients kd and kf are set to 0, the coefficient ki in the equation (3) is set to 1, and the coefficients kg and kh are set to 0.

  Thereby, the pixel data of R1ad in the frame FmIR1ad, the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad, and the pixel data of B3ad in the frame FmIR3ad have the same values.

  Therefore, as in the operation in the first night vision mode, the combining unit 523 selects the pixel data of R1ad in the frame FmIR1ad, the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad, and the pixel data of B3ad in the frame FmIR3ad. A frame FmIR123'ad can be generated.

  The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 21A shows a frame FmIR123′ad. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R1ad pixel data, and generates the R interpolation pixel data R1adi. The demosaic processing unit 54 generates an R frame FmIR123′adR in which all the pixels in one frame shown in FIG. 21B are made up of R pixel data.

  The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding Gr2ad and Gb2ad pixel data, and generates the G interpolation pixel data G2adi. The demosaic processing unit 54 performs interpolation to generate a G frame FmIR123′adG in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 generates B interpolation pixel data B3adi obtained by calculating B pixel data at a pixel position where no B pixel data exists using surrounding B3ad pixel data. The demosaic processing unit 54 generates a B frame FmIR123′adB in which all pixels in one frame shown in FIG.

  The second intermediate mode and the second night-vision mode differ in that the former does not operate the surrounding pixel adding unit 521 while the latter operates the surrounding pixel adding unit 521.

  Further, in the intermediate mode, pixel data for each color is generated from each pixel data obtained from the region corresponding to each color in the light receiving element, but in the night vision mode, the surrounding pixels are added. It can also be said that pixel data for each color is generated from pixel data obtained from a region wider than each region for generating pixel data for each color.

<Example of mode switching>
An example of mode switching by the mode switching unit 72 will be described with reference to FIG. FIG. 22A schematically shows, as an example, how the brightness of the surrounding environment changes as time passes from the daytime period to the nighttime period.

  As shown in FIG. 22 (a), the brightness decreases as time elapses from daytime to evening, and it becomes almost dark after time t3. The brightness shown in FIG. 22A substantially indicates the amount of visible light, and there is almost no visible light after time t3.

  The control unit 7 can determine the brightness of the surrounding environment based on the luminance level of the video signal (video data) input from the video processing unit 5. As shown in FIG. 22B, the mode switching unit 72 sets the normal mode when the brightness is equal to or higher than a predetermined threshold Th1 (first threshold), and sets the predetermined threshold Th2 (less than the threshold Th1). When it is equal to or greater than the second threshold), the intermediate mode is selected, and when it is less than the threshold Th2, the night vision mode is selected.

  The imaging apparatus according to the present embodiment automatically selects the normal mode from time t1 until time t1 when the brightness reaches the threshold value Th1, the intermediate mode from time t1 to time t2 when the brightness reaches the threshold value Th2, and the night vision mode after time t2. Switch automatically. In FIG. 22B, the intermediate mode may be either the first intermediate mode or the second intermediate mode, and the night vision mode may be either the first night vision mode or the second night vision mode.

  In FIG. 22A, the brightness immediately before time t3 when almost no visible light is present is set as the threshold value Th2, but the brightness at time t3 may be set as the threshold value Th2.

  As shown in FIG. 22 (c), the mode switching unit 72 has a period of time t1 in which the visible light is relatively high in the intermediate mode period, the time period on the time t2 side in the first intermediate mode. The period may be set to the second intermediate mode. In FIG. 22C, the night vision mode may be either the first night vision mode or the second night vision mode.

  In the imaging apparatus according to the present embodiment, the light projection control unit 71 controls on / off of the infrared projector 9, and the mode switching unit 72 switches the operation / non-operation of each unit in the video processing unit 5. Can be realized.

  As shown in FIG. 23, in the normal mode, the infrared projector 9 is off, the surrounding pixel adding unit 521, the same position pixel adding unit 522, and the synthesizing unit 523 are not operating, and the demosaic processing unit 54 is operating.

  In the first intermediate mode, the infrared projector 9 is on, the surrounding pixel adding unit 521 is not operating, and the same-position pixel adding unit 522, the combining unit 523, and the demosaic processing unit 54 are operating. In the second intermediate mode, the infrared projector 9 is on, the surrounding pixel adding unit 521 and the same-position pixel adding unit 522 are not operating, and the synthesizing unit 523 and the demosaic processing unit 54 are operating.

  The operation and non-operation in the same position pixel addition unit 522 can be easily switched by appropriately setting the values of the coefficients ka to ki in the expressions (1) to (3) as described above.

  In the first night-vision mode, the infrared projector 9 is on, and the surrounding pixel addition unit 521, the same-position pixel addition unit 522, the synthesis unit 523, and the demosaic processing unit 54 are all in an operating state. In the second night-vision mode, the infrared projector 9 is on, the same-position pixel adding unit 522 is not operating, and the surrounding pixel adding unit 521, the combining unit 523, and the demosaic processing unit 54 are operating.

  By the way, the surrounding pixel adding unit 521 sets the coefficient multiplied by the surrounding pixel data to a coefficient exceeding 0 (for example, 1) in the calculation formula for adding the surrounding pixel data to the pixel data of the target pixel. The surrounding pixel addition processing can be put into an operation state.

  Further, the surrounding pixel adding unit 521 can set the surrounding pixel addition processing to the non-operating state by setting the coefficient by which the surrounding pixel data is multiplied to 0 in the calculation formula.

  The operation and non-operation of the surrounding pixel addition unit 521 can be easily switched by appropriately setting the coefficient value.

<First Modification of Imaging Device>
The method by which the control unit 7 detects the brightness of the surrounding environment is not limited to the method based on the luminance level of the video signal.

  As shown in FIG. 24, the brightness of the surrounding environment may be detected by the brightness sensor 11. In FIG. 24, the brightness of the surrounding environment may be determined based on both the luminance level of the video signal and the brightness detected by the brightness sensor 11.

<Second Modification of Imaging Device>
The control unit 7 does not directly detect the brightness of the surrounding environment, but roughly assumes the brightness of the surrounding environment based on the time (date) and time (time zone) in one year, and the mode switching unit 72 may be switched to each mode.

  As shown in FIG. 25, in the mode setting table 12, any one of the normal mode, the intermediate mode, and the night vision mode is set corresponding to the combination of the date and the time zone. A clock 73 in the control unit 7 manages the date and time. The control unit 7 refers to the date and time indicated by the clock 73 and reads the mode set from the mode setting table 12.

  The light projection control unit 71 and the mode switching unit 72 control the imaging device so that the mode read from the mode setting table 12 is set.

<Third Modification of Imaging Device>
As shown in FIG. 26, the imaging device may be controlled so that the user manually selects the mode using the operation unit 13 and the light projection control unit 71 and the mode switching unit 72 are in the selected mode. The operation unit 13 may be an operation button provided on the housing of the imaging apparatus or a remote controller.

<Video signal processing method for mode switching>
A video signal processing method related to mode switching executed by the imaging apparatus shown in FIG. 1 will be described again with reference to FIG.

  In FIG. 27, when the imaging apparatus starts operation, the control unit 7 determines in step S1 whether or not the brightness of the surrounding environment is equal to or greater than a threshold value Th1. If it is equal to or greater than the threshold Th1 (YES), the control unit 7 causes the process in the normal mode to be executed in step S3. If it is not equal to or greater than the threshold value Th1 (NO), the control unit 7 determines whether or not the brightness of the surrounding environment is equal to or greater than the threshold value Th2 in step S2.

  If it is equal to or greater than the threshold value Th2 (YES), the control unit 7 causes the process in the intermediate mode to be executed in step S4. If it is not greater than or equal to the threshold Th2 (NO), the control unit 7 causes the process in the night-vision mode to be executed in step S5.

  Control part 7 returns processing to Step S1 after Steps S3-S5, and repeats after Step S1.

  FIG. 28 shows specific processing in the normal mode in step S3. In FIG. 28, the control unit 7 (light projection control unit 71) turns off the infrared light projector 9 in step S31. In step S32, the control unit 7 inserts the infrared cut filter 21. In step S33, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Tb. The order of steps S31 to S33 is arbitrary and may be simultaneous.

  In step S34, the control unit 7 causes the imaging unit 3 to image the subject. In step S <b> 35, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frames constituting the video signal generated by the imaging unit 3 imaging the subject.

  FIG. 29 shows specific processing in the intermediate mode in step S4. 29, in step S41, the control unit 7 (light projection control unit 71) turns on the infrared projector 9 so that infrared light with wavelengths IR1 to IR3 is projected in a time-sharing manner from the light projecting units 91 to 93. To.

  The control part 7 inserts the dummy glass 22 in step S42. In step S43, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Ta. The order of steps S41 to S43 is arbitrary and may be simultaneous.

  In step S44, the control unit 7 causes the imaging unit 3 to image the subject. The imaging unit 3 projects infrared light with a wavelength IR1 associated with R, infrared light with a wavelength IR2 associated with G, and infrared light with a wavelength IR3 associated with B, respectively. Take a picture of the subject in the

In step S45, the control unit 7 (mode switching unit 72) controls the pre-signal processing unit 52 so that the surrounding pixel addition unit 521 is deactivated and the synthesis unit 523 is operated to generate a synthesized video signal.

  The frames constituting the video signal generated when the imaging unit 3 images the subject in the state where infrared light of wavelengths IR1, IR2, and IR3 are respectively projected are the first frame, the second frame, 3 frames are assumed.

  The synthesizing unit 523 converts the three primary color pixel data based on the R pixel data in the first frame, the G pixel data in the second frame, and the B pixel data in the third frame into color The filters 32 are arranged so as to have the same arrangement as the filter elements. The synthesizer 523 thus generates a synthesized video signal by synthesizing the first to third frames into one frame.

  In step S46, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frame of the composite video signal.

  The demosaic processing unit 54 performs demosaic processing for generating an R frame, a G frame, and a B frame based on the frame of the composite video signal, and sequentially generates the demosaiced three primary color frames. .

  The demosaic processing unit 54 can generate an R frame by interpolating the R pixel data at a pixel position where the R pixel data does not exist. The demosaic processing unit 54 can generate the G frame by interpolating the G pixel data at the pixel position where the G pixel data does not exist. The demosaic processing unit 54 can generate the B frame by interpolating the B pixel data at the pixel position where the B pixel data does not exist.

  In the case of the first intermediate mode, the same position pixel addition unit 522 is operated in step S45, and in the case of the second intermediate mode, the same position pixel addition unit 522 is not operated in step S45. do it.

  FIG. 30 shows specific processing in the night-vision mode in step S5. In FIG. 30, the control unit 7 (light projection control unit 71) turns on the infrared projector 9 so that infrared light of wavelengths IR1 to IR3 is projected in a time division manner from the light projection units 91 to 93 in step S51. To.

  The control part 7 inserts the dummy glass 22 in step S52. In step S53, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Ta. The order of steps S51 to S53 is arbitrary and may be simultaneous.

  In step S54, the control unit 7 causes the imaging unit 3 to image the subject. In step S55, the control unit 7 (mode switching unit 72) controls the previous signal processing unit 52 to operate the surrounding pixel addition unit 521 and the synthesis unit 523 to generate a synthesized video signal.

  In step S56, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frame of the composite video signal.

  In the case of the first night vision mode, the same position pixel addition unit 522 is operated in step S55, and in the case of the second night vision mode, the same position pixel addition unit 522 is disabled in step S55. It may be an operation.

<Video signal processing program for mode switching>
In FIG. 1, the control unit 7 or an integrated part of the video processing unit 5 and the control unit 7 is configured by a computer (microcomputer), and the video signal processing program (computer program) is executed by the computer. It is also possible to realize the same operation as that of the image pickup apparatus. The image output unit 6 may be configured by a computer.

  An example of a procedure executed by the computer when the control in the intermediate mode, which is step S4 in FIG. 27, is configured by a video signal processing program will be described with reference to FIG. FIG. 31 shows processing executed by the computer by the video signal processing program relating to mode switching.

  In FIG. 31, the video signal processing program controls the infrared projector 9 to project infrared light of wavelengths IR1, IR2, and IR3 associated with R, G, and B to the computer in step S401. To execute the process.

  The step shown in step S401 may be executed outside the video signal processing program. In FIG. 31, the step of inserting the dummy glass 22 is omitted. The step of inserting the dummy glass 22 may also be executed outside the video signal processing program.

  In step S402, the video signal processing program captures the first frame of the video signal generated by the imaging unit 3 imaging the subject in a state where infrared light having the wavelength IR1 is projected on the computer. A process of acquiring pixel data to be configured is executed.

  In step S403, the video signal processing program captures the second frame of the video signal generated when the imaging unit 3 images the subject in a state where infrared light having the wavelength IR2 is projected on the computer. A process of acquiring pixel data to be configured is executed.

  In step S404, the video signal processing program captures a third frame of the video signal generated when the imaging unit 3 images the subject in a state where infrared light having the wavelength IR3 is projected on the computer. A process of acquiring pixel data to be configured is executed. The order of steps S402 to S404 is arbitrary.

  In step S405, the video signal processing program causes the computer to arrange the R, G, and B pixel data so as to be in the same arrangement as the arrangement of the filter elements in the color filter 32, and synthesize the synthesized video into one frame. A process for generating a signal is executed.

  In the intermediate mode, the video signal processing program does not cause the computer to execute the process of adding the surrounding pixels in step S405.

  In step S406, the video signal processing program causes the computer to perform a process of generating a R, G, B frame by performing a demosaic process on the frame of the composite video signal.

  Although illustration is omitted, when the control in the night vision mode, which is step S5 in FIG. 27, is configured by a video signal processing program, the computer executes the process of adding surrounding pixels in step S405 in FIG. You can do it.

  The video signal processing program may be a computer program recorded on a computer-readable recording medium. The video signal processing program may be provided in a state where it is recorded on a recording medium, or may be provided via a network such as the Internet so that the computer can download the video signal processing program. The computer-readable recording medium may be any non-temporary arbitrary recording medium such as a CD-ROM or a DVD-ROM.

  In the imaging apparatus configured as illustrated in FIG. 1, for example, a plurality of units may be provided as necessary, and the intermediate mode and the night vision mode may be executed simultaneously. In that case, the video output unit 6 may output both the video signal generated in the intermediate mode and the video signal generated in the night vision mode.

  The mode switching unit 72 switches between a state in which the video output unit 6 outputs the video signal generated in the intermediate mode and a state in which the video output unit 6 outputs the video signal generated in the night vision mode. May be. At this time, as described above, switching may be performed according to the brightness of the surrounding environment, the time, or the like. The video processing unit 5 (video processing device) may be separated from other units.

  Further, there may be a case where the normal mode is switched to the night vision mode or the night vision mode is switched to the normal mode without using the intermediate mode.

  When the intermediate mode is not used, the normal mode and the night vision mode may be selected and used in an environment where the use of the intermediate mode is suitable. In this case, the color image signal is not satisfactory as compared with the case where the intermediate mode is used, but imaging is possible.

  However, even with an imaging device equipped with only the normal mode and the night vision mode, the subject is photographed with one imaging device in a situation where the ambient brightness changes, for example, when the subject is photographed all day with a surveillance camera. There is an effect that can be done.

  Furthermore, the normal mode may be switched to the intermediate mode and the intermediate mode may be switched to the normal mode without using the night vision mode. When the night vision mode is not always used, the night vision mode may not be installed.

  The night vision mode may not be used in places with electric lights. An imaging apparatus equipped with only the normal mode and the intermediate mode can be used when the night vision mode does not have to be used.

  When the night vision mode is not used, the intermediate mode may be used in an environment where the above-described night vision mode is suitable. In this case, the color image signal is not good as compared with the case of using the night vision mode, but imaging is possible.

  Even in the case of an imaging device equipped only with the normal mode and the intermediate mode, similarly, there is an effect that a subject can be photographed with one imaging device in a situation where ambient brightness changes.

<Method for determining the relationship between the amount of visible light and the amount of infrared light>
As described above, the relationship between the amount of visible light and the amount of infrared light differs depending on the surrounding environment. If the white balance of the video signal is adjusted in a specific state where there is a relationship between the amount of visible light and the amount of infrared light, the relationship between the amount of visible light and the amount of infrared light is the specific state. In a different state, the white balance is lost. That is, if the relationship between the amounts of light is different, the color of the subject cannot be reproduced with high image quality in each state.

  Therefore, the determination unit 78 determines the master-slave relationship of the light amount according to a first example or a second example of a determination method described later. First, the characteristics of the spectral sensitivity characteristics in the imaging unit 3 described in FIG. 3 will be considered with reference to FIG.

  As described above, the wavelengths IR1, IR2, and IR3 are set to 780 nm, 940 nm, and 870 nm. As shown in FIG. 32, the sensitivity of R at the wavelength IR1 associated with R is Str, the sensitivity of G at the wavelength IR2 associated with G is Stg, and the sensitivity of B at the wavelength IR3 associated with B is Let it be Stb.

  As can be seen from FIG. 32, the values of sensitivities Str, Stg, and Stb are greatly different. Therefore, if the emission power of the infrared light with the wavelengths IR1 to IR3 is the same, the exposure amount of the imaging unit 3 differs depending on which infrared light with the wavelengths IR1, IR2, and IR3 is projected.

<First example of determination method>
A first example of the determination method will be described with reference to FIGS. 33 to 36 and FIG. It is assumed that the emission power of infrared light with wavelengths IR1 to IR3 is the same. As a first example of the determination method, the determination unit 78 includes at least an exposure amount of the imaging unit 3 in a state where infrared light with wavelengths IR1, IR2, and IR3 is selectively projected every predetermined period. The master-slave relationship of the light quantity is determined by comparing the exposure amounts of the two periods.

  FIG. 33 shows that the amount of visible light is sufficiently larger than the amount of infrared light, and the visible light is the main state. FIGS. 33A and 33B partially show FIGS. 8A and 8B. The infrared projector 9 selectively projects infrared light having wavelengths IR1 to IR3 as shown in FIG. 32A, and the imaging unit 3 exposes as shown in FIG. 32B.

  The exposure amount during the period when infrared light with wavelength IR1 is projected is AepR, the exposure amount during the period when infrared light with wavelength IR2 is projected is AepG, and infrared light with wavelength IR3 is projected. Let AepB be the amount of exposure during this period.

  If visible light is the main state, the exposure amount of the imaging unit 3 is hardly affected by the infrared light projected by the infrared projector 9. Therefore, the exposure amounts AepR, AepG, and AepB of the imaging unit 3 have substantially the same value regardless of which infrared light of the wavelengths IR1 to IR3 is projected, as shown in FIG. When the visible light is the main state, the exposure amounts AepR, AepG, and AepB of the imaging unit 3 are always substantially constant values.

  FIG. 34 shows that the amount of infrared light is sufficiently larger than the amount of visible light, and infrared light is the main state. 34A and 34B are the same as FIGS. 33A and 33B. If the infrared light is the main state, the exposure amount of the imaging unit 3 is affected by the infrared light projected by the infrared projector 9.

  Therefore, as shown in FIG. 34C, the exposure amount of the imaging unit 3 varies depending on which infrared light of the wavelengths IR1 to IR3 is projected. The exposure amounts AepR, AepG, and AepB have a relationship of AepR> AepB> AepG, reflecting the difference between the sensitivities Str, Stg, and Stb described in FIG.

  As can be seen by comparing (c) in FIG. 33 and (c) in FIG. 34, the determination unit 78 determines that the visible light is emitted if the exposure amount of the imaging unit 3 is substantially constant in a period obtained by dividing one frame into three. If the exposure amount of the main and imaging unit 3 varies in three divided periods, it is determined that infrared light is main.

  Specifically, at least two of the exposure amounts AepR, AepG, and AepB are compared, and if the difference is not more than a predetermined threshold, the visible light is mainly, and if there is a difference exceeding the predetermined threshold, infrared light is It can be determined to be the main. It is also possible to increase the determination accuracy by comparing the exposure amounts AepR, AepG, and AepB with each other.

  When comparing two of the exposure amounts AepR, AepG, and AepB, it is preferable to compare the exposure amount AepR and the exposure amount AepG having the largest difference.

  A more specific configuration and operation of the determination unit 78 that realizes the first example of the determination method will be described with reference to FIG. FIG. 35 is a conceptual explanatory diagram for determining the master-slave relationship of the light amount by comparing the exposure amount AepR and the exposure amount AepG.

  FIG. 35A shows an arbitrary frame FmIR1 of video data generated at the timing when infrared light having a wavelength IR1 is projected, which is the same as FIG. 9A. (B) in FIG. 35 is an arbitrary frame FmIR2 of the video data generated at the timing when the infrared light having the wavelength IR2 is projected, which is the same as (b) in FIG. Here, the pixel data of Gr and Gb is G, and the subscripts attached in FIG. 9 are omitted.

  FIG. 35C is a conceptual block diagram of the determination unit 78 in the first example. As shown in (c) of FIG. 35, the determination unit 78 includes an average value conversion unit 781, a difference calculation unit 782, and a comparison unit 783.

  The two averaging units 781 separately illustrate an averaging unit 781 at the timing when the pixel data of the frame FmIR1 is input and an averaging unit 781 at the timing when the pixel data of the frame FmIR2 is input. It is a thing.

  The averaging unit 781 averages pixel data in a predetermined area of the frame FmIR1 to generate an average value Fave (IR1). The averaging unit 781 averages pixel data in a predetermined area of the frame FmIR2 to generate an average value Fave (IR2). The average values Fave (IR1) and Fave (IR2) are input to the difference calculation unit 782.

  The predetermined area may be the whole of one frame, or may be a portion such as a central portion excluding an end of one frame. The predetermined area may be a plurality of frames. Hereinafter, the predetermined area is one frame.

  The difference calculation unit 782 calculates a difference between the average value Fave (IR1) and the average value Fave (IR2) to generate a difference value Vdf. Although not shown, the difference value Vdf is absoluteized by the absolute value circuit and input to the comparison unit 783.

  The comparison unit 783 compares the difference value Vdf with the threshold value Th10, and indicates that the visible light is mainly if the difference value Vdf is equal to or less than the threshold value Th10. If the difference value Vdf exceeds the threshold value Th10, infrared light is emitted. Determination data Ddet1 indicating the main is generated.

  The determination data Ddet1 may be, for example, binary data of 0 when visible light is dominant and 1 when infrared light is dominant. The determination data Ddet1 is input to the color gain control unit 79. The determination data Ddet1 may be a value that changes according to the difference value Vdf, and the color gain may be adaptively changed according to the value.

  The operation of the determination unit 78 will be further described with reference to the flowchart of FIG. In FIG. 36, the determination unit 78 determines whether or not it is the generation timing of the frame FmIR1 in step S101. If it is the generation timing of the frame FmIR1 (YES), the determination unit 78 averages the pixel data of the frame FmIR1 to generate an average value Fave (IR1) in step S102.

  If it is not the generation timing of the frame FmIR1 (NO), the determination unit 78 determines whether it is the generation timing of the frame FmIR2 in step S103. If it is the generation timing of the frame FmIR2 (YES), the determination unit 78 averages the pixel data of the frame FmIR2 to generate an average value Fave (IR2) in step S104. If it is not the generation timing of the frame FmIR2 (NO), the determination unit 78 returns the process to step S101.

  In step S105, the determination unit 78 calculates a difference between the average value Fave (IR1) and the average value Fave (IR2) to generate a difference value Vdf. In step S106, the determination unit 78 determines whether or not the difference value Vdf is equal to or less than the threshold value Th10.

  If the difference value Vdf is equal to or less than the threshold value Th10 (YES), the determination unit 78 determines that visible light is mainly in step S107, and returns the process to step S101. If the difference value Vdf is not equal to or less than the threshold value Th10 (NO), the determination unit 78 determines that infrared light is mainly used in step S108, and returns the process to step S101. Thereafter, the same processing is repeated.

  As described above, in the first example of the determination method, the signal level of the video signal (video data) input to the control unit 7, specifically, the average value of the pixel signal (pixel data) of one frame is compared. Thus, the exposure amount AepR and the exposure amount AepG are substantially compared.

  As shown in FIG. 37, in step S201, the color gain control unit 79 determines whether or not the visible light is mainly based on the input determination data Ddet1. If the visible light is mainly (YES), the color gain control unit 79 controls the color gain setting unit 62 to select the first set of color gains, and returns the process to step S201.

  If the visible light is not the main (NO), the color gain control unit 79 controls the color gain setting unit 62 to select the second set of color gains, and returns the process to step S201. Thereafter, the same processing is repeated.

  FIG. 34 and FIG. 35 show a case where the emission power of infrared light with wavelengths IR1 to IR3 is the same. It is also possible to determine the master-slave relationship between the amounts of light based on the exposure amounts AepR, AepG, and AepB after intentionally changing the emission power of infrared light with wavelengths IR1 to IR3.

  FIG. 38 shows the main state of infrared light, as in FIG. (A) and (c) of FIG. 38 are the same as (a) and (b) of FIG. As shown in (b) of FIG. 38, for example, it is assumed that the emission power of infrared light with a wavelength IR3 is made larger than the emission power of infrared light with a wavelength IR1 and IR2.

  The exposure amounts AepR, AepG, and AepB in this case are as shown in FIG. The determination unit 78 may determine the master-slave relationship of the light amount by comparing the exposure amount AepR and the exposure amount AepB, or may determine the master-slave relationship of the light amount by comparing the exposure amount AepG and the exposure amount AepB. Good. The determination unit 78 recognizes the presence or absence of a specific pattern related to the magnitudes of the exposure amounts AepR, AepG, and AepB, and determines that visible light is dominant if there is no specific pattern, and infrared light is main if there is a specific pattern. May be.

<Second example of determination method>
A second example of the determination method will be described with reference to FIGS. It is assumed that the emission power of infrared light with wavelengths IR1 to IR3 is the same. As a second example of the determination method, the determination unit 78 projects a pixel signal based on an imaging signal generated in a state where infrared light having a wavelength IR1 is projected, and other infrared light. The master-slave relationship of the light amount is determined by using the pixel signal based on the imaging signal generated in the state.

  The determination unit 78 uses an R pixel signal and a G or B pixel signal as pixel signals. The other infrared light may be either infrared light having a wavelength IR2 or infrared light having a wavelength IR3.

  As shown in FIG. 32, at the wavelength IR1, there is a large sensitivity difference between the R sensitivity Str and the G and B sensitivities. The sensitivity difference between the R sensitivity Str and the B sensitivity Stb1 is particularly large. At the wavelength IR2, the R and B sensitivities Str2 and Stb2 are almost the same as the G sensitivity Stg, and there is almost no difference in sensitivity between the two. Even at the wavelength IR3, the difference between the sensitivity Stb of B and the sensitivity of R and G is slight.

  The second example of the determination method determines the master-slave relationship of the light amount by utilizing the difference in sensitivity of R, G, and B at wavelengths IR1 to IR3.

  At the wavelength IR1, the difference in sensitivity between the R sensitivity Str and the B sensitivity Stb1 is the largest. Therefore, as the pixel signal to be compared with the R pixel signal, the B pixel signal is used rather than the G pixel signal. preferable. Therefore, the determination unit 78 uses the R pixel signal and the B pixel signal in a state where infrared light having the wavelength IR1 is projected.

  The R and B pixel signals in a state where infrared light with a wavelength IR2 is projected and the R and B pixel signals in a state where infrared light with a wavelength IR3 is projected may be any, but determination The former 78 uses the former.

  A specific configuration and operation of the determination unit 78 that realizes the second example of the determination method will be described with reference to FIG. FIG. 39 is also a conceptual explanatory diagram for determining the master-slave relationship of the light amount, as in FIG.

  39A is an arbitrary frame FmIR1 of video data generated at the timing when infrared light having the wavelength IR1 is projected, which is the same as FIG. 35A. FIG. 39B is an arbitrary frame FmIR2 of the video data generated at the timing when infrared light having the wavelength IR2 is projected, which is the same as FIG. 35B.

  FIG. 39C conceptually shows a state in which only R pixels in the frame FmIR1 are extracted. FIG. 39D conceptually shows a state in which only the B pixel in the frame FmIR1 is extracted.

  FIG. 39 (e) conceptually shows a state in which only R pixels in the frame FmIR2 are extracted. FIG. 39 (f) conceptually shows a state where only the B pixel in the frame FmIR2 is extracted.

  FIG. 39G is a conceptual block diagram of the determination unit 78 in the second example. As illustrated in (g) of FIG. 39, the determination unit 78 includes average value conversion units 784 and 785, difference calculation units 786 and 787, and a comparison unit 788.

  Each of the two averaging units 784 and 785 receives the R and B pixel data of the frame FmIR2 and the averaging units 784 and 785 at the timing when the R and B pixel data of the frame FmIR1 is input. The averaging units 784 and 785 at the timing are separately illustrated.

  The averaging unit 784 averages R pixel data in one frame of the frame FmIR1 to generate an average value Rave (IR1). The averaging unit 785 averages the B pixel data in one frame of the frame FmIR1 to generate an average value Bave (IR1).

  The averaging unit 784 averages R pixel data in one frame of the frame FmIR2 to generate an average value Rave (IR2). The averaging unit 785 averages the B pixel data in one frame of the frame FmIR2 to generate an average value Bave (IR2).

  The average value is obtained by shifting the timing for generating the average values Rave (IR1) and Rave (IR2) of the R pixel data and the timing for generating the average values Bave (IR1) and Bave (IR2) of the B pixel data. It is good also as only one conversion part. In this case, the previously generated average value is temporarily stored.

  The average values Rave (IR1) and Bave (IR1) and the average values Rave (IR2) and Bave (IR2) are input to the difference calculation unit 786. The two difference calculation units 786 are the difference calculation unit 786 at the timing when the average values Rave (IR1) and Bave (IR1) are input, and the timing at which the average values Rave (IR2) and Bave (IR2) are input. The difference calculation unit 786 is illustrated separately.

  The difference calculation unit 786 calculates a difference between the average value Rave (IR1) and the average value Bave (IR1) to generate a difference value Vdf1. The difference calculation unit 786 calculates a difference between the average value Rave (IR2) and the average value Bave (IR2) to generate a difference value Vdf2. Although not shown, the difference values Vdf1 and Vdf2 are absoluteized by the absolute value circuit and input to the difference calculation unit 787.

  The difference calculation unit 787 calculates a difference between the difference value Vdf1 and the difference value Vdf2, and generates a difference value Vdf12. Although not shown, the difference value Vdf12 is absoluteized by the absolute value circuit and input to the comparison unit 788.

  The comparison unit 788 compares the difference value Vdf12 with the threshold value Th20, and indicates that the visible light is mainly if the difference value Vdf12 is equal to or less than the threshold value Th20, and if the difference value Vdf12 exceeds the threshold value Th20, infrared light is emitted. Determination data Ddet2 indicating that it is main is generated. The determination data Ddet2 may also be binary data. The determination data Ddet2 is input to the color gain control unit 79.

  FIG. 40 shows a state in which visible light is mainly used. 40A and 40B are the same as FIGS. 33A and 33B. FIG. 40C shows a frame generated based on the exposure shown in FIG. As described with reference to FIG. 8, the timing of the exposure shown in FIG. 40B and the frame shown in FIG. 40C are off, but here, the timing difference is ignored and the position in the time direction (left-right direction) is ignored. The state where these are matched is illustrated.

  In the state where the visible light is the main, as described with reference to FIG. 33, the exposure amounts AepR, AepG, and AepB of the imaging unit 3 are almost constant values, and the influence of the sensitivity differences of R, G, and B at the wavelengths IR1 to IR3 is almost the same. I do not receive it.

  Therefore, as shown in (d) of FIG. 40, the difference value Vdf1 generated at the timing of the frame FmIR1 and the difference value Vdf2 generated at the timing of the frame FmIR2 are substantially the same value. Therefore, the difference value Vdf12 is a small value.

  In FIG. 40D, the difference values Vdf1 and Vdf2 are described so as to extend over the frame periods of the frames FmIR1 and FmIR2 for convenience. Actually, the difference values Vdf1 and Vdf2 are generated at a predetermined timing after the pixel data of the frames FmIR1 and FmIR2 are input to the determination unit 78.

  FIG. 41 shows a state in which infrared light is mainly used. 41A to 41C are the same as FIGS. 40A to 40C. When infrared light is the main state, it is affected by the difference in sensitivity between R, G, and B at wavelengths IR1 to IR3.

  Therefore, as shown in FIG. 41 (d), the difference value Vdf1 generated at the timing of the frame FmIR1 and the difference value Vdf2 generated at the timing of the frame FmIR2 are greatly different. Therefore, the difference value Vdf12 is a large value.

  The operation of the determination unit 78 will be further described with reference to the flowchart of FIG. 42, in step S301, the determination unit 78 determines whether it is the generation timing of the frame FmIR1.

  If it is the generation timing of the frame FmIR1 (YES), the determination unit 78 averages the R pixel data of the frame FmIR1 to generate an average value Rave (IR1) in step S302. In step S303, the determination unit 78 averages the B pixel data of the frame FmIR1 to generate an average value Bave (IR1).

  In step S304, the determination unit 78 calculates a difference between the average value Rave (IR1) and the average value Bave (IR1) to generate a difference value Vdf1.

  On the other hand, if it is not the generation timing of the frame FmIR1 in step S301 (NO), the determination unit 78 determines whether it is the generation timing of the frame FmIR2 in step S305.

  If it is the generation timing of the frame FmIR2 (YES), the determination unit 78 averages the R pixel data of the frame FmIR2 to generate an average value Rave (IR2) in step S306. In step S307, the determination unit 78 averages the B pixel data of the frame FmIR2 to generate an average value Bave (IR2).

  In step S308, the determination unit 78 calculates a difference between the average value Rave (IR2) and the average value Bave (IR2) to generate a difference value Vdf2.

  If it is not the generation timing of the frame FmIR2 in step S305 (NO), the determination unit 78 returns the process to step S301.

  In step S309, the determination unit 78 calculates a difference between the difference value Vdf1 and the difference value Vdf2, and generates a difference value Vdf12. In step S310, the determination unit 78 determines whether or not the difference value Vdf12 is equal to or less than the threshold value Th20.

  If the difference value Vdf12 is equal to or less than the threshold value Th20 (YES), the determination unit 78 determines in step S311 that visible light is mainly, and returns the process to step S301. If the difference value Vdf12 is not equal to or less than the threshold value Th20 (NO), the determination unit 78 determines that infrared light is the main in step S312, and returns the process to step S101. Thereafter, the same processing is repeated.

  In step S201 in FIG. 37, the color gain control unit 79 determines whether or not visible light is mainly based on the input determination data Ddet2.

  If the visible light is mainly (YES), the color gain control unit 79 controls the color gain setting unit 62 to select the first set of color gains, and returns the process to step S201.

  If the visible light is not the main (NO), the color gain control unit 79 controls the color gain setting unit 62 to select the second set of color gains, and returns the process to step S201. Thereafter, the same processing is repeated.

<Video signal processing method according to determination of master-slave relationship of light quantity>
A video signal processing method according to the determination of the master-slave relationship of the amount of light executed by the imaging apparatus shown in FIG. 1 will be described again with reference to FIG.

  In FIG. 43, in step S501, the imaging unit 3 images a subject in a state where infrared light of wavelengths IR1, IR2, and IR3 is selectively projected by the infrared projector 9, and R, G, B An imaging signal corresponding to each of the above is generated.

  In step S502, the video processing unit 5 generates R, G, and B primary color data based on the imaging signal. In step S503, the determination unit 78 determines the master-slave relationship between the amount of visible light in the surrounding environment and the amount of infrared light projected by the infrared projector 9.

  In step S504, the color gain control unit 79 controls the color gain setting unit 62 to select the first set of color gains if visible light is the main (YES). Therefore, the video output unit 6 outputs a video signal obtained by multiplying the three primary color data by the first set of color gains in step S505.

  In step S504, the color gain control unit 79 controls the color gain setting unit 62 to select the second set of color gains if the visible light is not the main (NO). Therefore, the video output unit 6 outputs a video signal obtained by multiplying the three primary color data by the second set of color gains in step S506.

  Through the processing in steps S504 to S506, the imaging apparatus selects one of a plurality of sets of color gains to be multiplied by the three primary color data according to the determined master-slave relationship, and sets the color gain of the selected set to the above-described color gain. A video signal multiplied by the three primary color data is output.

  When the imaging device is set to the intermediate mode, the processes in steps S501 to S506 are repeated.

<Video signal processing program for controlling the operation of the imaging apparatus including the determination of the master-slave relationship of light quantity>
With reference to FIG. 44, a video signal processing program in the case of controlling the operation of the imaging device including determination of the master-slave relationship of the light amount by a computer program will be described.

  In FIG. 44, the video signal processing program is generated in step S601 by imaging a subject in a state where infrared light of wavelengths IR1, IR2, IR3 is selectively projected by the infrared projector 9 on the computer. A process of acquiring video data based on the imaging signals corresponding to each of the R, G, and B performed is executed.

  In step S602, the video signal processing program executes a process of determining, on the computer, the relationship between the amount of visible light in the surrounding environment and the amount of infrared light projected by the infrared projector 9 based on the video data. Let Although not shown, the determination result is temporarily held in the computer.

  In step S603, the video signal processing program causes the computer to execute processing for confirming the stored determination result.

  If visible light is the main (YES), in step S604, the video signal processing program causes the computer to determine a set of color gains to be multiplied by the three primary color data generated by the color gain setting unit 62 based on the video data. A process of controlling the color gain setting unit 62 is executed so that one set is obtained.

  If the visible light is not the main (NO), in step S605, the video signal processing program causes the computer to determine a set of color gains to be multiplied by the three primary color data generated by the color gain setting unit 62 based on the video data. A process for controlling the color gain setting unit 62 is executed so as to obtain a set of two.

  Through steps S603 to S605, the video signal processing program causes the computer to select one of a plurality of sets of color gains according to the determined master-slave relationship, as a set of color gains to which the color gain setting unit 62 multiplies the three primary color data. The process of controlling the color gain setting unit 62 is executed so as to select.

  When the imaging apparatus is controlled to the intermediate mode, the video signal processing program causes the computer to repeatedly execute the processes in steps S601 to S605.

  Similarly, the video signal processing program for controlling the operation of the imaging device including the determination of the master-slave relationship of the light quantity may be a computer program recorded on a computer-readable non-transitory recording medium. This video signal processing program may also be provided in a state of being recorded on a recording medium, or may be provided via a network.

<Example of color gain group switching>
With reference to FIG. 45, how the color gain set used by the color gain setting unit 62 is switched will be described. 45 (a) and 45 (b) are the same as FIGS. 22 (a) and 22 (b).

  Since the visible light is mainly used in the normal mode, the color gain set used by the color gain setting unit 62 is fixed as the first set. Since the infrared light is mainly used in the night vision mode, the color gain set used by the color gain setting unit 62 is fixed in the second set. The color gain setting unit 62 may select the first group and the second group according to the mode switching state by the mode switching unit 72.

  In the intermediate mode, according to the determination result according to the first or second example of the determination method described above, the first group is selected when the visible light is relatively large, and the second group is selected when the infrared light is relatively large. Is selected.

  Therefore, when the color gain setting unit 62 shifts from the normal mode to the intermediate mode and from the intermediate mode to the night-vision mode, the first set is used until the middle mode, and after the middle mode, the second group is used. Will be used.

  In FIG. 45C, for the sake of convenience, the period of the intermediate mode is divided into two, and the normal mode side is the first set and the night vision mode is the second set. Actually, the timing of switching from the state selected by the first group to the state of selecting the second group depends on how to set the threshold values Th10 and Th20 and various other conditions.

  The color gain setting unit 62 may hold at least three color gain groups, and the color gain group used in the night vision mode may be a third group different from the second group used in the intermediate mode.

  It is also possible to configure to switch between the intermediate mode and the night-vision mode using the determination result of the master-slave relationship of the light amount described above. In the state of FIGS. 33 and 40 where the visible light is main, the intermediate mode may be used, and in the state of FIGS. 34 and 40 where the infrared light is main, the night-vision mode may be used.

  By the way, the state where the visible light is the main and the infrared light is the slave is not necessarily the state where the visible light is more than the infrared light. The state where the infrared light is the main and the visible light is the subordinate is not necessarily the state where the infrared light is a lot of visible light. The relationship between visible light and infrared light need not be determined based on a specific ratio of visible light and infrared light. For example, the relationship between visible light and infrared light may be determined so that the generated video signal has higher image quality.

  The present invention is not limited to the embodiment described above, and various modifications can be made without departing from the scope of the present invention. For example, in the imaging apparatus shown in FIG. 1, the infrared projector 9 may be detachable from the casing of the imaging apparatus. The infrared projector 9 may be configured outside the imaging apparatus. The imaging device may have a configuration for controlling the infrared projector 9 when the infrared projector 9 is attached.

  You may measure each light quantity for the relationship between visible light and infrared light. The determination of the master-slave relationship by the determination unit 78 includes a case based on a measurement result obtained by measuring the light amount. In the determination of the master-slave relationship based on the video signal described above, it is easy to reflect the relationship between visible light and infrared light near the subject. Therefore, the determination of the master-slave relationship based on the video signal is better than the determination of the master-slave relationship based on the light quantity measurement result.

  The control unit 7 may change various parameters according to the relationship between the amount of visible light and the amount of infrared light in the surrounding environment. The present embodiment is not limited to the color gain, and can be applied to a technique for changing various parameters for image processing of an imaging signal imaged by the imaging unit 3.

  The control unit 7 may switch between various shooting modes according to the relationship between the amount of visible light and the amount of infrared light in the surrounding environment. The present embodiment is applicable not only to the intermediate mode and the night vision mode but also to switching control between various known shooting modes. This embodiment can also be applied to other imaging techniques using infrared light.

  Furthermore, the control unit 7 and the video processing unit 5 can be realized by using one or a plurality of hardware processors (CPUs). Use of hardware and software is optional. The imaging device may be configured only by hardware, or a part of the configuration may be realized by software.

DESCRIPTION OF SYMBOLS 3 Image pick-up part 5 Image processing part 7 Control part 9 Infrared projector 32 Color filter 54 Demosaic processing part 62 Color gain setting part 71 Light projection control part 72 Mode switching part 78 Judgment part 79 Color gain control part 521 Surrounding pixel addition part 522 Same position Pixel addition unit 523 synthesis unit 781, 784, 785 averaging unit 782, 786, 787 difference calculation unit 783, 788 comparison unit

Claims (7)

A filter element of three primary colors of red, green, and blue is provided with a color filter arranged in a predetermined arrangement in the surface, and first infrared light having a first wavelength associated with red is projected. The subject is imaged to generate a first frame of the video signal, and the subject is imaged with the second infrared light having the second wavelength associated with green. The second frame of the video signal is generated, the subject is imaged in the state where the third infrared light having the third wavelength corresponding to the blue color is projected, and the third frame of the video signal is captured. An imaging unit for generating
The red pixel data at the same pixel position in the first to third frames, the green pixel data at the same pixel position, and the blue pixel data at the same pixel position are individually added to obtain a red first data A same-position pixel addition unit that generates one addition pixel data, green second addition pixel data, and blue third addition pixel data;
The pixel data of the three primary colors of the first addition pixel data, the second addition pixel data, and the third addition pixel data generated by the same position pixel addition unit are used as the filter element in the color filter. A synthesizing unit that generates a synthesized video signal synthesized into one frame by arranging so as to be the same arrangement as the arrangement;
Based on the frame of the synthesized video signal generated by the synthesis unit, a red frame obtained by interpolating red pixel data at a pixel position where no red pixel data exists, and a green frame at a pixel position where no green pixel data exists. A demosaic processing unit that generates a frame of three primary colors by performing a demosaic process that generates a green frame obtained by interpolating pixel data and a blue frame obtained by interpolating blue pixel data at a pixel position where blue pixel data does not exist When,
An imaging apparatus comprising: The fourth is obtained by adding the pixel data located in the vicinity to the red pixel data by adding the pixel data located in the periphery to the pixel data in each pixel position in the first to third frames. Pixel data, a fifth addition pixel data obtained by adding pixel data located around the green pixel data, and a sixth addition obtained by adding pixel data located around the blue pixel data A surrounding pixel adding unit for generating pixel data;
The same-position pixel addition unit adds the fourth to sixth addition pixel data at the same pixel position in the first to third frames individually, thereby adding the first to third addition pixels. The imaging apparatus according to claim 1, wherein data is generated.   The image processing apparatus further includes a mode switching unit that switches between a mode in which the surrounding pixel addition unit is inoperative and a mode in which the surrounding pixel addition unit is operated according to the brightness of the surrounding environment when the subject is imaged. The imaging device according to claim 2. When the brightness of the surrounding environment when imaging the subject is greater than or equal to the first threshold, the imaging unit images the subject in a state where the first to third infrared lights are not projected, and the surroundings When the brightness of the environment is less than the first threshold value, infrared imaging is performed so that the imaging unit images the subject in a state where the first to third infrared lights are projected. A light projection control unit for controlling on / off;
When the brightness of the surrounding environment is equal to or greater than the first threshold, the synthesis unit is disabled, and the demosaic processing unit is in a state where the first to third infrared lights are not projected. A mode for performing demosaic processing for generating a red frame, a green frame, and a blue frame based on a frame of a video signal generated by the imaging unit imaging a subject, and the brightness of the surrounding environment When it is less than the first threshold, the synthesizing unit is operated, and the demosaic processing unit performs a red frame, a green frame, and a blue frame based on the frame of the synthesized video signal generated by the synthesizing unit. A mode switching unit that switches between modes for performing demosaic processing for generating
The imaging apparatus according to claim 2, further comprising:   The mode switching unit further switches to a mode in which the surrounding pixel addition unit is inoperative when the brightness of the surrounding environment is less than the first threshold and is equal to or more than the second threshold, and the brightness of the surrounding environment is set. The imaging apparatus according to claim 4, wherein when the value is less than the second threshold value, the mode is switched to a mode in which the surrounding pixel addition unit is operated. The first infrared light having the first wavelength associated with red is obtained by an imaging unit including a color filter in which filter elements of three primary colors of red, green, and blue are arranged in a predetermined arrangement in a plane. Image a subject in a projected state to generate a first frame of a video signal;
Imaging the subject in a state in which the second infrared light having the second wavelength associated with green is projected by the imaging unit to generate a second frame of the video signal;
Imaging the subject in a state where the third infrared light having the third wavelength corresponding to blue is projected by the imaging unit to generate a third frame of the video signal;
The red pixel data at the same pixel position in the first to third frames, the green pixel data at the same pixel position, and the blue pixel data at the same pixel position are individually added to obtain a red first data 1 addition pixel data, green second addition pixel data, and blue third addition pixel data are generated,
The pixel data of the three primary colors of the first addition pixel data, the second addition pixel data, and the third addition pixel data are arranged to be the same as the arrangement of the filter elements in the color filter. To generate a composite video signal combined into one frame,
Based on the frame before Kigo adult video signal, and red frame interpolating red pixel data to the pixel position where the red pixel data is not present, the green pixel data in the pixel position where the green pixel data does not exist interpolation A three-primary-color frame is generated by performing demosaic processing for generating a green frame and a blue frame obtained by interpolating blue pixel data at pixel positions where no blue pixel data exists Processing method. On the computer,
The filter elements of the three primary colors of red, green, and blue are arranged in a predetermined arrangement in the plane in a state where the first infrared light having the first wavelength associated with red is projected Acquiring pixel data constituting a first frame of a video signal generated by an imaging unit including a color filter imaging a subject;
A second frame of a video signal generated when the imaging unit images the subject in a state where the second infrared light having the second wavelength associated with green is projected. Obtaining pixel data to perform,
A third frame of a video signal generated by the imaging unit imaging the subject in a state where the third infrared light having the third wavelength associated with blue is projected is configured. Obtaining pixel data;
The red pixel data at the same pixel position in the first to third frames, the green pixel data at the same pixel position, and the blue pixel data at the same pixel position are individually added to obtain a red first data Generating one addition pixel data, green second addition pixel data, and blue third addition pixel data;
The pixel data of the three primary colors of the first addition pixel data, the second addition pixel data, and the third addition pixel data are arranged to be the same as the arrangement of the filter elements in the color filter. A step of generating a composite video signal combined into one frame,
Based on the frame of the composite video signal, a red frame obtained by interpolating red pixel data at a pixel position where no red pixel data exists, and a green color obtained by interpolating green pixel data at a pixel position where no green pixel data exists. Generating a frame of three primary colors by performing a demosaic process for generating a frame of (2) and a blue frame obtained by interpolating blue pixel data at a pixel position where blue pixel data does not exist;
A video signal processing program characterized in that

Images