WO2020017638A1 - 画像生成装置及び撮像装置 - Google Patents

画像生成装置及び撮像装置 Download PDF

Info

Publication number
WO2020017638A1
WO2020017638A1 PCT/JP2019/028470 JP2019028470W WO2020017638A1 WO 2020017638 A1 WO2020017638 A1 WO 2020017638A1 JP 2019028470 W JP2019028470 W JP 2019028470W WO 2020017638 A1 WO2020017638 A1 WO 2020017638A1
Authority
WO
WIPO (PCT)
Prior art keywords
signal
color
infrared light
visible light
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/028470
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
光一朗 石神
角　博文
基史 祖父江
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanolux Co Ltd
Original Assignee
Nanolux Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanolux Co Ltd filed Critical Nanolux Co Ltd
Priority to CN201980040150.6A priority Critical patent/CN112335233B/zh
Priority to US17/251,215 priority patent/US11284044B2/en
Priority to JP2020531386A priority patent/JP7203441B2/ja
Publication of WO2020017638A1 publication Critical patent/WO2020017638A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/135—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on four or more different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image

Definitions

  • the present invention relates to an image generation device and an imaging device that generate a color image from visible light and near-infrared light.
  • Patent Document 1 There has been proposed an image capturing apparatus that detects infrared light reflected by a subject or infrared light emitted by the subject to form a color image of the subject.
  • a color image is generated from near-infrared light by utilizing the fact that the same subject spectral reflectance characteristic as that in the visible wavelength region is observed in the near-infrared wavelength region. I have. Specifically, light in the near-infrared region having a high correlation with the color when the same subject is visually observed under visible light is detected, and a display color is pseudo-generated from the detected information. If this technology is used, it is possible to capture a color image even in an extremely low illuminance environment or darkness.
  • the imaging devices described in Patent Documents 2 and 3 described above have a problem that a color image cannot be captured in an environment where there is almost no visible light because color information is obtained from a visible light signal.
  • the imaging device described in Patent Literature 1 can generate a color image using only near-infrared light, and thus can shoot even in darkness (0 lux).
  • the color is set based on the correlation with visible light. Therefore, there is a problem in color reproducibility.
  • an object of the present invention is to provide an image generation device and an imaging device capable of generating a color image with clear and excellent color reproducibility even when photographing in an environment where there is not enough visible light. I do.
  • the image generating apparatus the three or more types of visible light signals based on visible light having different wavelengths or wavelength ranges output from the solid-state imaging device, the mutual wavelength or wavelength output from the solid-state imaging device
  • a first color signal generation unit configured to combine at least two types of near-infrared light signals based on near-infrared light having different regions to generate three or more types of first color signals;
  • the above-described near-infrared light signal is obtained simultaneously with the three or more kinds of visible light signals, and at least the first near-infrared light based on the near-infrared light having a peak wavelength in the range of 700 to 870 nm.
  • the first color reproduction signal generator may perform signal processing on the three or more types of visible light signals and the two or more types of near-infrared light signals, and then combine them.
  • the image generation device of the present invention has a signal separation unit that separates the optical signal output from the solid-state imaging device into the visible light signal and the near-infrared light signal and outputs the separated signal.
  • the separated visible light signal and near infrared light signal may be input to the first color signal generation unit.
  • the image generation device of the present invention may further include the three or more types of visible light signals output from the solid-state imaging device, the two or more types of near-infrared light signals or components based on the visible light, and the near-infrared light component.
  • a second color signal generator that generates three or more second color signals from two or more mixed light signals containing both light-based components, and combines the first color signal and the second color signal.
  • An image generation unit that generates a color image.
  • the second color signal generation unit adds three or more types of near-infrared light signals or the mixed light signal to the three or more types of visible light signals, and generates three or more types of second color signals. Generate.
  • the second signal generation unit may generate the second color signal by selecting any one of the visible light signal, the near infrared light signal, and the mixed light signal.
  • the image generation unit may add a difference between the first color signal and the second color signal to the second color signal. At this time, after performing a noise reduction process on a difference between the first color signal and the second color signal, the difference can be added to the second signal. Further, the second color signal may be added to the first signal after being subjected to a sharpening process.
  • An imaging device is a solid-state imaging device that detects three or more visible lights having different wavelengths or wavelength ranges and two or more near-infrared lights having mutually different wavelengths or wavelength ranges, with the image generation device described above. And an imaging unit that converts visible light and near-infrared light received from a subject into respective electric signals.
  • the imaging device of the present invention may include an illumination unit that irradiates the subject with the two or more near-infrared lights.
  • a color signal is generated by combining three or more types of visible light signals and two or more types of near-infrared light signals under specific conditions.
  • FIG. 1 is a block diagram illustrating a configuration of an image generation device according to a first embodiment of the present invention.
  • FIG. 2 is a diagram illustrating an optical signal input to the image generation device 10 illustrated in FIG. 1.
  • FIG. 2 is a block diagram illustrating a configuration example of signal processing units 3a and 3b illustrated in FIG. 1.
  • FIG. 4 is a block diagram illustrating another configuration example of the signal processing units 3a and 3b illustrated in FIG. 1. It is a block diagram showing the composition of the image generation device of the 1st modification of a 1st embodiment of the present invention. It is a block diagram showing the composition of the image generation device of a 2nd embodiment of the present invention.
  • FIG. 7 is a block diagram illustrating a configuration example of a signal processing unit 6 illustrated in FIG. 6.
  • FIG. 7 is a block diagram illustrating a configuration example of an image generation unit 22 illustrated in FIG. 6.
  • FIG. 7 is a block diagram illustrating another configuration example of the image generation unit 22 illustrated in FIG. 6.
  • FIG. 14 is a block diagram illustrating a configuration of an image generation device according to a first modification of the second embodiment of the present invention.
  • FIG. 11 is a diagram illustrating an example of separation calculation in a signal separation unit 4 illustrated in FIG. 10. It is a block diagram showing the composition of the imaging device of a 3rd embodiment of the present invention.
  • FIGS. 3A to 3D are diagrams illustrating examples of pixel arrangement of a solid-state imaging device.
  • FIG. 9 is a diagram illustrating another example of a pixel arrangement of a solid-state imaging device.
  • FIG. 14A and 14B are diagrams showing a method of generating a color image using the image pickup device having the pixel arrangement shown in FIG. 14, wherein A is for capturing an image based on visible light, and B is for capturing an image based on near-infrared light. The case where an image is taken is shown.
  • FIG. 15 is a diagram illustrating a method of generating a black-and-white image using the imaging device having the pixel arrangement illustrated in FIG. 14.
  • FIG. 1 is a block diagram illustrating the configuration of the image generating apparatus according to the present embodiment.
  • an image generation device 10 of the present embodiment generates a color image from three or more types of visible light signals and two or more types of near-infrared light signals, and includes a first color signal generation unit. 1 and an image generation unit 2, and signal processing units 3a and 3b are further provided as necessary.
  • FIG. 2 is a diagram showing an optical signal input to the image generation device 10 shown in FIG.
  • the visible light signals S R , S G , and S B input to the image generation device 10 of the present embodiment are output from the solid-state imaging device, respectively, This is based on three types of visible light having different wavelengths or wavelength ranges detected by the pixels R, G, and B.
  • the near-infrared light signals S IR1 and S IR2 are obtained at the same time as the visible light signals S R , S G and S B , and are mutually detected by the near-infrared pixels IR 1 and IR 2 of the solid-state imaging device. And two types of near-infrared light having different wavelengths or wavelength ranges.
  • At least two types of near-infrared light signals include at least a first near-infrared light signal SIR1 based on near-infrared light having a peak wavelength in the range of 700 to 870 nm and a near-infrared light signal SIR1 having a peak wavelength of 870 to 2500 nm.
  • a second near-infrared light signal SIR2 based on near-infrared light in the range, preferably in the range of 870 to 1100 nm.
  • the present invention shows a case where three kinds of visible light signals and two kinds of near-infrared light signals are used, but the present invention is not limited to this, and the visible light signal is Three or more types and near-infrared light signals may be two or more types.
  • the present invention also includes a case where one or two or more of the visible light and the near-infrared light to be detected are not detected in the solid-state imaging device. In that case, the corresponding visible light signal or near-infrared light signal is used. Is treated as having a signal strength of 0.
  • the first color signal generation unit 1 combines the near-infrared light signals S IR1 and S IR2 with the visible light signals S R , S G and S B at an arbitrary ratio to generate three or more types of color images. This is to generate one-color signals Rl, Gl, Bl.
  • the synthesis ratio of the visible light signals S R , S G , S B and the near-infrared light signals S IR1 , S IR2 depends on the light signals such as luminance and chroma level (chroma) from the viewpoint of emphasizing color reproducibility. And / or the intensity (signal level) of the optical signal.
  • the combination ratio of the visible light signal and the near infrared light signal may be determined in consideration of the exposure amount, the irradiation amount of the illumination light, and the like in addition to the quality and intensity of each optical signal.
  • the composition ratio in one image may be the same for the entire image, but may be locally changed.
  • the composition ratio can be determined using, for example, an integral value in a certain pixel range.
  • the pixel range at that time is desirably within an effective area that is emphasized during observation, but if it is difficult to provide a detection unit for each response level for the effective area, as a simple alternative method, A small detection-dedicated area may be provided at the end outside the effective pixel, and the determination may be made based on the detection result of only that area.
  • the synthesis ratio can be determined by using a result of performing a filtering process on a response of an adjacent pixel or a response of a peripheral pixel locally.
  • the two types of pairs to be compared are not limited to one set of information, but may be a plurality of sets, or may be converted into one representative component ratio information by an arbitrary weighted average or the like.
  • composition ratio is locally modulated, for example, by giving a different composition ratio independently for each pixel, it is possible to optimize the image quality of the entire image finally obtained.
  • a different combination ratio is given independently for each pixel, there are disadvantages in that the load for calculation and the cost increase, but the near-infrared light signal can compensate for the shortage of the amount of visible light. Since the amount of visible light irradiation is not always uniform within the angle of view, changing the composition ratio for each position or area can compensate for the visible light response and reduce excess / shortage on the entire screen. Appropriate image quality can be obtained.
  • the image generation unit 2 generates a color image from the first color signals Rl, Gl, Bl generated by the first color signal generation unit 1. At that time, if necessary, white balance (White Balance: WB), image interpolation (Interpolation: ITP), color correction (Color Correction), gradation correction (Tone Correction), noise reduction (Noise Reduction: NR), etc. Perform development processing.
  • white Balance White Balance: WB
  • image interpolation Interpolation: ITP
  • color correction Color Correction
  • gradation correction gradation correction
  • noise reduction Noise Reduction: NR
  • the image generation device 10 of the present embodiment has the visible light signals S R , S G , S B and the near-infrared light signals S IR1 , S IR2 input to the first color signal generation unit 1. May be provided with signal processing sections 3a and 3b for performing various signal processing.
  • FIG. 3 is a block diagram illustrating a configuration example of the signal processing units 3a and 3b. As shown in FIG. 3, each of the signal processing units 3a and 3b includes a white balance adjustment unit WB, an image interpolation processing unit ITP, a color correction processing unit CC, a tone correction unit Tone, and the like.
  • visible light signal S R as achromatic surface of the object is represented as achromatic, S G, is S B and the near infrared light signal S IR1, the level of S IR2 are trimmed.
  • the processing may be performed as red R, green G, and blue B, respectively.
  • the near-infrared light signal SIR1 is regarded as red (R)
  • the near-infrared light signal SIR2 is regarded as green (G)
  • B there is no blue
  • the image interpolation processing unit ITP performs an interpolation operation on a signal having a different pixel position for each type of color filter (a signal having a different type of color filter for each pixel position) according to the color filter array.
  • Three types of signals of red (R), green (G) and blue (B) are obtained.
  • the near-infrared light signal SIR1 is regarded as red (R)
  • the near-infrared light signal SIR2 is regarded as green (G), similarly to the above-described white balance.
  • processing may be performed assuming that there is no blue (B).
  • the color correction processing unit CC performs color correction by, for example, a 3 ⁇ 3 linear matrix operation. At that time, by adjusting the linear matrix coefficient, it is possible to appropriately represent various colors of the subject. More specifically, coefficients of the visible light signals S R , S G , S B and the near-infrared light signals S IR1 , S IR2 are determined in advance so that desired color reproducibility can be obtained. Is calculated by using, for example, the following Expressions 1 and 2.
  • the tone correction unit Tone corrects each optical signal having a linear response characteristic with respect to the amount of light so that the tone becomes appropriate as an image. Specifically, a non-linear conversion is performed on an input signal that has a linear response to the amount of light in accordance with a coding standard that assumes display display or the like.
  • the conversion characteristic (conversion function) in the tone correction unit Tone may add an intentional non-linear characteristic in order to correct or express the appearance of the image. Interpolation can be performed by providing a one-dimensional numerical table, and various operation methods can be applied.
  • the processes performed by the signal processing units 3a and 3b are not limited to the above-described white balance, image interpolation, color correction, and gradation correction, and may be performed in addition to or instead of these processes. Processing such as black level correction (BK), color space conversion (CSC), and noise reduction (NR) can also be performed. In addition, the order of each process is not limited to the order shown in FIG. 3, and the execution position can be set as appropriate.
  • different processing may be performed on the visible light signals S R , S G , and S B and the near-infrared light signals S IR1 , S IR2 , and a part or all of these processing may be performed by the first color signal generation.
  • the first color signals Rl, Gl, output from the first color signal generation unit 1, Bl may be performed.
  • FIG. 4 is a block diagram showing another example of the configuration of the signal processing units 3a and 3b.
  • the near-infrared light signal output from the solid-state imaging device does not have sufficient color separation as compared with the visible light signal, and is concentrated around an achromatic color.
  • the difference in state is large depending on the position in the signal space (color space), and the distribution is complicated. Therefore, in the image generating apparatus of the present embodiment, by making the signal processing units 3a and 3b have a configuration as shown in FIG. 4, a reliable correction effect can be obtained for such an optical signal.
  • the signal processing unit 3a that processes the visible light signals S R , S G , and S B includes a white balance adjustment unit WB, an image interpolation processing unit ITP, a color correction processing unit CC1, a tone correction unit Tone,
  • the color space conversion unit CSC is provided in this order
  • the signal processing unit 3b that processes the near infrared light signals S IR1 and S IR2 includes a white balance adjustment unit WB, an image interpolation processing unit ITP, a gradation correction unit Tone, a color space
  • the conversion unit CSC and the color correction processing unit CC2 are provided in this order.
  • the color space conversion unit CSC converts a visible light signal and a near-infrared signal into a luminance chromaticity signal by a matrix calculation represented by the following Expression 3.
  • the conversion destination space is a virtual separation space for distinguishing coefficients in the next correction stage, and is a temporary temporary signal space (Y ′, Cb ′, Cr ′).
  • Y ′ conforms to the ITU-RBT.709 standard
  • a RY 0.2126
  • a GY 0.7152
  • a BY 0.0722
  • a Rb -0.1146
  • a Gb -0.3854
  • a Bb 0.5
  • a Rr 0.5
  • a Gr -0.4542
  • a Br -0.0458
  • the processing in the color space conversion unit CS does not necessarily have to conform to the standard of the output signal, and can be adjusted to a convenient space in order to give a correction effect to each optical signal.
  • the color correction processing unit CC2 performs color correction on the luminance and chromaticity signals S Y , S Cb , and S Cr that have been subjected to the color space correction after the gradation correction. This is performed using
  • each coefficient has the following values and the correction is invalid, but by changing and adjusting each coefficient from this state, the colors of various subjects can be expressed to some extent independently with few parameter operations. can do.
  • the image generating apparatus uses three or more types of visible light signals and two or more near-infrared light signals acquired simultaneously by the solid-state imaging device, and responds to the signal quality and intensity. Because color images are generated from color signals obtained by adjusting the combination ratios, clear and excellent color reproducibility can be obtained even when shooting in an environment where there is not enough visible light. .
  • the image generation apparatus of the present embodiment instead of changing the shooting conditions (visible light shooting or near-infrared light shooting) in the imaging system, changing the synthesis conditions in the developing system enables shooting. Since it is possible to cope with changes in the brightness of the environment, it is possible to realize an imaging device capable of obtaining a high-quality color image regardless of the amount of visible light.
  • the image generation device of the present embodiment is particularly suitable for an imaging device that captures a moving image for a long time, such as a monitoring camera.
  • FIG. 5 is a block diagram showing a configuration of an image generation device according to a first modification of the first embodiment of the present invention.
  • the same components as those of the image generation device 10 shown in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.
  • the image generation device 11 of the present modification includes a signal separation unit 4 in addition to the first color signal generation unit 1, the image generation unit 2, and the signal processing units 3a and 3b.
  • the signal separation unit 4 separates an optical signal output from the solid-state imaging device into a visible light signal and a near-infrared light signal and outputs the separated signals.
  • mixed light signals SR + IR1 , SG + IR2 , and SB + IR3 including both the visible light component and the near infrared light component output from the near infrared light pixel of the solid-state imaging device are converted into near infrared light.
  • the signals are separated into signals S IR1 , S IR2 , S IR3 and visible light signals S R , S G , S B.
  • the separated visible light signals S R , S G , S B are output together with the visible light signals S R , S G , S B output from the visible light pixels, and are output to the signal processing unit 3 a or the first signal generation unit. 1 and the separated near-infrared light signals S IR1 , S IR2 , and S IR3 are input to the signal processing unit 3b or the first signal generation unit 1.
  • the near-infrared light pixel of the solid-state imaging device is configured to detect only near-infrared light, that is, when a signal including both visible light and near-infrared light is not output from the solid-state imaging device, signal separation is performed.
  • the unit 4 is unnecessary.
  • the signal separating unit 4 outputs two types of near-infrared light signals S IR1 and S IR2 .
  • the image generation device of the present modified example includes the signal separation unit 4, even if the pixel of the solid-state imaging device is configured to detect both visible light and near-infrared light, the above-described first image generation device can be used. As in the case of the embodiment, it is possible to generate a clear and excellent color reproducibility color image even when photographing in an environment where there is not enough visible light.
  • the configurations and effects of the present modification other than those described above are the same as those of the above-described first embodiment.
  • FIG. 6 is a block diagram illustrating the imaging apparatus according to the present embodiment. 6, the same components as those of the image generating apparatus 10 shown in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.
  • the image generation device 20 according to the present embodiment includes a second color signal generation unit 5 and a signal in addition to the first color signal generation unit 1, the image generation unit 22, and the signal processing units 3a and 3b.
  • a processing unit 6 is provided.
  • the second color signal generation unit 5 converts the visible light signals S R , S G , S B and the near-infrared light signals S IR1 , S IR2 , S IR3 output from the solid-state imaging device into second color signals Rh, Gh. , Bh.
  • the visible light signal and the near-infrared light signal are combined with emphasis on color reproducibility, but in the second color signal generation unit 5, the signal-to-noise ratio (signal- A visible light signal and a near-infrared light signal are combined with emphasis on to-noise ratio (SN ratio).
  • Second color signal Rh, Gh as a method of generating Bh, for example, a method of adding the visible light signals S R, S G, S B and the near-infrared light signal S IR1, S IR2, S IR3 , or, A method of selecting any of the visible light signals S R , S G , S B and the near-infrared light signals S IR1 , S IR2 , S IR3 can be used.
  • FIG. 7 is a block diagram illustrating a configuration example of the signal processing unit 6.
  • the signal processing unit 6 performs various kinds of signal processing on the second color signals Rh, Gh, and Bh generated by the above-described second signal generation unit 5, and includes a white balance adjustment unit. It includes a WB, an image interpolation processing unit ITP, a tone correction unit Tone, and the like.
  • the processes of the white balance adjustment unit WB, the image interpolation processing unit ITP, and the tone correction unit Tone are the same as those of the signal processing units 3a and 3b described above.
  • the image generation unit 22 includes first color signals Rl, Gl, and Bl generated by the first color signal generation unit 1 and emphasizing color reproducibility, and a second color signal generated by the second color signal generation unit 5 and emphasized by the SN ratio.
  • a color image is generated from Rh, Gh, and Bh. At this time, it is possible to reduce noise included in the first color signals Rl, Gl, Bl emphasizing color reproducibility, and adjust the edge expression and sharpness of the second color signals Rh, Gh, Bh emphasizing the SN ratio. preferable.
  • FIGS. 8 and 9 are block diagrams showing a configuration example of the image generation unit 22.
  • the image generation unit 22 calculates a difference between the first color signals Rl, Gl, Bl and the second color signals Rh, Gh, Bh. Perform noise reduction processing. Then, the noise-reduced difference is added again to the second color signals Rh, Gh, Bh to obtain a color image.
  • a sharpening processing unit SH may be provided in addition to the noise reduction processing unit NR, and the second color signals Rh, Gh, and Bh may be subjected to sharpening processing such as edge enhancement and sharpness.
  • the noise-reduced difference is added to the second color signals Rh, Gh, and Bh that have been subjected to the sharpening processing.
  • the first color signals Rl, Gl, Bl emphasizing color reproducibility contain a relatively large amount of noise.
  • the noise contained in the first color signals Rl, Gl, Bl may be directly reduced from the input signal.
  • the second color signals Rh, Gh, Bh with an emphasis on the SN ratio By performing the noise reduction processing on the difference from the above, noise can be effectively reduced.
  • the image generating apparatus of the present embodiment uses the visible light signal and the near-infrared signal simultaneously acquired by the solid-state imaging device to use the first color signals Rl, Gl, Bl emphasizing color reproducibility. And the second color signals Rh, Gh, and Bh with an emphasis on the SN ratio, and a color image is generated from these two types of signals. Therefore, regardless of the amount of visible light, clear and excellent color reproducibility are obtained. A color image can be obtained.
  • the configuration and effects of the present embodiment other than those described above are the same as those of the above-described first embodiment and its modifications.
  • FIG. 10 is a block diagram showing a configuration of an image generating apparatus according to the present modification.
  • the same components as those of the image generating apparatus 20 shown in FIG. 6 are denoted by the same reference numerals, and detailed description thereof will be omitted.
  • the image generation device 21 of the second embodiment has the same configuration as that of the image generation device 20 of the above-described second embodiment except that the signal separation unit 4 is provided before the signal processing units 3a and 3b. Is the same as
  • FIG. 11 is a diagram illustrating an example of separation calculation in the signal separation unit 4.
  • the image generation device 21 sends visible light signals S R , S G , and S B to the image generation device 21.
  • Mixed light signals S R + IR1 and SG + IR2 including both visible light components and near-infrared light components.
  • the signal separating unit 4 a visible light signal S R, S G, is separated into S B and the near-infrared light signal S IR1, S IR2.
  • a method for separating the visible light signal and the near-infrared light signal is not particularly limited, for example, as shown in FIG. 11, a visible light signal S R corresponding to the red light R, The visible light signal SG corresponding to the green light G and the visible light signal SB corresponding to the blue light B are output as they are. Furthermore, by the mixed light signal S R + IR1 containing a component corresponding to the component and the near infrared light IR1 corresponding to the red light R subtract visible light signal S R, the near-infrared light signal S IR1 obtained.
  • These five types of optical signals are input to the first color signal generator 1 after being subjected to signal processing as needed.
  • a visible light signal S R which is outputted from the solid-state imaging device, S G, the mixed light signal SS R + IR1, S G + IR2 and S B, the second color signal Rh, Gh, the Bh Generated.
  • Second color signal Rh, Gh, the generation of Bh like the second embodiment described above, Ya method adding the visible light signals S R, S G, the mixed light signal S B S R + IR1, S G + IR2 , A method of selecting one of the visible light signals S R , S G , S B and the mixed light signal S R + IR1 , S G + IR2 .
  • FIG. 12 is a block diagram illustrating a configuration of the imaging device of the present embodiment.
  • the imaging device 30 of the present embodiment includes an imaging unit 31 and a developing unit 32, and the developing unit 32 includes the above-described image generation devices 10, 11, 20, and 21. Further, the imaging device 30 of the present embodiment may be provided with an illumination unit 33 that irradiates the subject 40 with two or more near-infrared lights.
  • the imaging unit 31 includes a solid-state imaging device that detects three or more visible lights having different wavelengths or wavelength ranges from each other and two or more near-infrared lights having different wavelengths or wavelength ranges from each other. It converts near-infrared light into respective electric signals and outputs them.
  • FIGS. 13A to 13D and FIG. 14 are diagrams showing examples of the pixel arrangement of the solid-state imaging device.
  • the pixel arrangement of the solid-state imaging device is not particularly limited. For example, when visible light and near-infrared light are detected by different pixels, the pixel arrangement shown in FIG. 13A can be adopted. When detecting visible light and near-infrared light with the same pixel, the pixel arrangement shown in FIG. 13B can be adopted.
  • a spectroscopic element for dispersing the reflected light from the subject 40 may be provided in the imaging unit 31, and visible light and near-infrared light may be detected by two or more solid-state imaging elements.
  • a configuration in which one solid-state imaging device detects visible light and another solid-state imaging device detects visible light and near-infrared light may be adopted.
  • a configuration in which one solid-state imaging device detects only visible light and another solid-state imaging device detects only near-infrared light may be employed in which one solid-state imaging device detects only visible light and another solid-state imaging device detects only near-infrared light.
  • the imaging element of the imaging unit 31 includes a red pixel R for detecting red light, a blue pixel B for detecting blue light, and two green pixels G for detecting green light. Visible light detection region, a first near-infrared pixel IR1 for detecting the first near-infrared light, a second near-infrared pixel IR2 for detecting the second near-infrared light, and a third near-infrared pixel IR2. It is also possible to adopt a configuration in which the near-infrared-light detection areas configured by two third near-infrared pixels IR3 that detect infrared light are alternately arranged.
  • FIG. 15A and 15B are diagrams illustrating a method of generating a color image using the image pickup device having the pixel arrangement illustrated in FIG. 14.
  • FIG. 15A illustrates a case where an image based on visible light is captured, and FIG. The case where an image based on the image is captured will be described.
  • a visible light color image is generated using the image pickup device having the pixel arrangement shown in FIG. 14, a visible light signal S R based on light detected by each of the pixels R, G, and B in the visible light detection area shown in FIG. 15A. , S G, using S B.
  • the near-infrared light signals S IR1 , S IR2 , and S IR3 detected in step (1) are synthesized to generate a color image.
  • the image is based on the light detected by the pixels IR1, IR2, and IR3 in the near-infrared light detection area illustrated in FIG. 15B.
  • the near infrared light signals SIR1 , SIR2 and SIR3 are used. Specifically, each pixel IR1, IR2, IR3 near infrared light signal S outputted from IR1, S IR2, S IR3 performs signal processing, if necessary visible light signals S R, S G, S B Are combined to generate a color image.
  • FIG. 16 is a diagram illustrating a method of generating a black-and-white image using the imaging device having the pixel arrangement illustrated in FIG.
  • the imaging apparatus can generate not only a color image but also a two-dimensional black-and-white image based on light of a specific wavelength.
  • the image pickup device having the pixel arrangement shown in FIG. 14 it is possible to generate six types of images of red light, blue light, green light, and three types of near-infrared light having different wavelengths.
  • the visible light signals S R , S G , and S B and the near-infrared light signals S IR1 , S IR2 , and S IR3 are individually processed to generate a black-and-white image. I do.
  • the solid-state imaging device used in the imaging apparatus according to the present invention is not limited to those shown in FIGS. 13A to 13D or FIG. 14, but may include three or more visible lights having different wavelengths or wavelength ranges. What is necessary is just to be able to detect two or more near-infrared lights having different wavelengths or wavelength ranges.
  • the imaging device of the present embodiment uses the above-described image generation device, it is possible to generate a clear and excellent color reproducibility color image regardless of the amount of visible light. Further, the imaging device according to the present embodiment does not need to switch between imaging with visible light and imaging with near-infrared light according to the brightness of the imaging environment unlike the conventional imaging device. No sensor or the like is required for detecting the error.
  • the configurations and effects of the imaging device according to the embodiment other than those described above are the same as those of the above-described first and second embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Color Television Image Signal Generators (AREA)
  • Studio Devices (AREA)
PCT/JP2019/028470 2018-07-20 2019-07-19 画像生成装置及び撮像装置 Ceased WO2020017638A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201980040150.6A CN112335233B (zh) 2018-07-20 2019-07-19 图像生成装置以及摄像装置
US17/251,215 US11284044B2 (en) 2018-07-20 2019-07-19 Image generation device and imaging device
JP2020531386A JP7203441B2 (ja) 2018-07-20 2019-07-19 画像生成装置及び撮像装置

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018136621 2018-07-20
JP2018-136621 2018-07-20

Publications (1)

Publication Number Publication Date
WO2020017638A1 true WO2020017638A1 (ja) 2020-01-23

Family

ID=69164644

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/028470 Ceased WO2020017638A1 (ja) 2018-07-20 2019-07-19 画像生成装置及び撮像装置

Country Status (4)

Country Link
US (1) US11284044B2 (https=)
JP (1) JP7203441B2 (https=)
CN (1) CN112335233B (https=)
WO (1) WO2020017638A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116097298A (zh) * 2020-07-27 2023-05-09 元平台技术有限公司 用于使用结构光进行深度图构建的红外和非红外通道混合器

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013255144A (ja) * 2012-06-08 2013-12-19 Hitachi Consumer Electronics Co Ltd 撮像装置
JP2014135627A (ja) * 2013-01-10 2014-07-24 Hitachi Ltd 撮像装置
JP2015173451A (ja) * 2009-07-30 2015-10-01 国立研究開発法人産業技術総合研究所 画像撮影装置および画像撮影方法
JP2017005484A (ja) * 2015-06-10 2017-01-05 株式会社 日立産業制御ソリューションズ 撮像装置
WO2017159041A1 (ja) * 2016-03-14 2017-09-21 ソニー株式会社 信号処理装置、撮像装置および信号処理方法

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000278701A (ja) * 1999-03-24 2000-10-06 Sony Corp 赤外線撮像装置
JP4630781B2 (ja) 2005-09-26 2011-02-09 キヤノン株式会社 波長成分の割合検出装置及びそれを用いた撮像装置
JP2010161453A (ja) 2009-01-06 2010-07-22 Olympus Corp 赤外線照射式撮像装置
JP2011015087A (ja) * 2009-06-30 2011-01-20 Panasonic Corp 撮像装置および撮像方法
JP5874116B2 (ja) 2009-07-30 2016-03-02 国立研究開発法人産業技術総合研究所 画像撮影装置および画像撮影方法
JP5485004B2 (ja) 2010-04-23 2014-05-07 パナソニック株式会社 撮像装置
CN105432076B (zh) 2013-07-25 2017-10-10 日立麦克赛尔株式会社 摄像装置
JP6136948B2 (ja) * 2014-01-24 2017-05-31 株式会社Ｊｖｃケンウッド 撮像装置、映像信号処理方法及び映像信号処理プログラム
JP6240813B2 (ja) * 2015-04-23 2017-11-29 富士フイルム株式会社 画像処理装置、撮像装置、画像処理方法及び画像処理プログラム
US10699395B2 (en) * 2016-10-14 2020-06-30 Mitsubishi Electric Corporation Image processing device, image processing method, and image capturing device
TWI666935B (zh) * 2017-07-12 2019-07-21 謝基生 增強近紅外擷取影像的微型熱像儀
US11343434B2 (en) * 2018-12-20 2022-05-24 Canon Kabushiki Kaisha Image processing apparatus and control method for same

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015173451A (ja) * 2009-07-30 2015-10-01 国立研究開発法人産業技術総合研究所 画像撮影装置および画像撮影方法
JP2013255144A (ja) * 2012-06-08 2013-12-19 Hitachi Consumer Electronics Co Ltd 撮像装置
JP2014135627A (ja) * 2013-01-10 2014-07-24 Hitachi Ltd 撮像装置
JP2017005484A (ja) * 2015-06-10 2017-01-05 株式会社 日立産業制御ソリューションズ 撮像装置
WO2017159041A1 (ja) * 2016-03-14 2017-09-21 ソニー株式会社 信号処理装置、撮像装置および信号処理方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116097298A (zh) * 2020-07-27 2023-05-09 元平台技术有限公司 用于使用结构光进行深度图构建的红外和非红外通道混合器
JP2023535538A (ja) * 2020-07-27 2023-08-18 メタ プラットフォームズ テクノロジーズ， リミテッド ライアビリティ カンパニー 構造光を使用した深度マッピングのための赤外線・非赤外線チャネルブレンダ

Also Published As

Publication number Publication date
US11284044B2 (en) 2022-03-22
US20210250556A1 (en) 2021-08-12
CN112335233B (zh) 2022-07-12
JP7203441B2 (ja) 2023-01-13
CN112335233A (zh) 2021-02-05
JPWO2020017638A1 (ja) 2021-08-02

Similar Documents

Publication Publication Date Title
JP6725613B2 (ja) 撮像装置および撮像処理方法
EP2426928B1 (en) Image processing apparatus, image processing method and program
US6995791B2 (en) Automatic white balance for digital imaging
CN103430551B (zh) 使用具有纵色像差的透镜单元的成像系统及其操作方法
CN102204258B (zh) 图像输入装置
CN107534727B (zh) 图像处理装置、摄像装置、图像处理方法及记录有图像处理程序的计算机可读非暂时性有形记录介质
TWI416940B (zh) 影像處理裝置及影像處理程式
EP1594321A2 (en) Extended dynamic range in color imagers
US20060146064A1 (en) Hexagonal color pixel structure with white pixels
CN102170530B (zh) 信号处理设备及方法、固态图像捕获设备和电子信息器件
CN104756488B (zh) 信号处理设备以及信号处理方法
KR102102740B1 (ko) 이미지 프로세싱 장치 및 이미지 프로세싱 방법
JP2002232906A (ja) ホワイトバランス制御装置
JP6484504B2 (ja) 撮像装置
US9105105B2 (en) Imaging device, imaging system, and imaging method utilizing white balance correction
CN115802183B (zh) 图像处理方法及其相关设备
JP7203441B2 (ja) 画像生成装置及び撮像装置
JP7657629B2 (ja) 撮像装置
JP2020182163A (ja) 画像処理装置、画像処理方法およびプログラム
JP6384595B2 (ja) 撮像装置、データ生成装置および画像処理装置
JP2005323141A (ja) 撮像装置、カメラ、及び信号処理方法
CN113615148A (zh) 相机色彩图像处理
JPWO2020017638A5 (https=)
JP2001057680A (ja) ホワイトバランス調整方法および装置並びに記録媒体
CN120321517A (zh) 一种图像处理方法、装置、设备以及计算机存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19837695

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020531386

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19837695

Country of ref document: EP

Kind code of ref document: A1