JP5034665B2 - Image input processing device, pixel signal processing circuit, and edge detection method - Google Patents

Description

  The present invention relates to an image input processing device, a pixel signal processing circuit, and an edge detection method for capturing a subject and executing noise reduction or edge enhancement of an image by filter processing based on a generated pixel signal.

In a camera device that captures moving images and still images, for example, when the ISO sensitivity is increased and the image is taken, the automatic exposure function works, the exposure time of the sensor (imaging device) is shortened, and the signal output from the sensor is greatly amplified. Since it is amplified at a degree (gain), it is greatly affected by noise generated by the sensor itself and peripheral IC, resulting in a noisy image with a low signal-to-noise ratio (S / N ratio).
Noise is generated due to influences from various surroundings such as the sensor's own factors and the operating frequency of the peripheral IC. There are also variations in the characteristics of the signal source (sensor) itself and variations in time, and various noise patterns from low frequency to high frequency affect the image.
Signal processing performed internally by the camera device to reduce or suppress noise is known to use a noise reduction circuit that uses a filter with a large number of taps. The S / N ratio of the image is improved.

  However, if processing is performed with a normal low-pass filter (for example, a Gaussian filter) having a large number of taps, the edge information is diffused accordingly, so that the sharpness of the edge is lowered and the resolution is lowered. Will be output. In order to solve this problem, in the camera signal processing, a noise reduction (NR) technique for improving the S / N ratio by performing filter processing on the input image while maintaining the resolution is required.

  As one of the approaches to noise reduction (NR), invisible light, especially light from an object that has passed an IR (Infrared Radiation) cut filter that cuts the near-infrared region, is converted into red (R), green (G), blue (B) Other than an image having a high color reproducibility that is received by an imaging device including a predetermined color filter and output from the imaging device, the same subject is not passed through the IR cut filter as an image having a larger amount of information. A technique is known in which edge information is detected from an image obtained by acquiring a captured image and maintaining a large amount of information by not passing through the IR cut filter (see, for example, Patent Document 1).

  In Patent Document 1, an image for acquiring edge information is called an invisible light image (infrared light image in the embodiment), and a wide range of frequency components from a low frequency to a high frequency before passing through an IR cut filter are obtained. Contains image information.

  In Patent Document 1, high-frequency components are extracted by passing an invisible light image (infrared light image) through a high-pass filter, while gain adjustment is performed on a visible light image, that is, an image with high color reproducibility captured through an IR filter. Remove noise through a low-pass filter. However, when the low-pass filter is passed, the resolution information is diffused and the sharpness of the edge is lowered. Therefore, in the said patent document 1, the visible light image after this low-pass filtering and the infrared light image after the said high-pass filter containing edge information are synthesize | combined. This achieves both noise reduction and prevention of edge information diffusion (edge sharpness reduction).

With respect to the IR filter, the color filter layer of a single imaging device can have a function of selecting whether to transmit or block infrared light components.
For example, Patent Document 2 includes a transmission filter of three primary colors of red (R), green (G), and blue (B), and an infrared pass filter that has sensitivity in the infrared region and transmits infrared light. A color filter in which a repetitive minimum unit (pixel unit) is configured is disclosed. Patent Document 2 describes that the infrared pass filter may be a white (W) filter.

By the way, if there is an infrared light component, the color reproducibility deteriorates and the image is generally whitish. Usually, an IR cut filter is provided as an optical component separately from the imaging device, or the above-mentioned patent document. As shown in FIG. 2, it is necessary to provide a color transmission filter having an IR cut filter function on the pixel itself that generates the primary color signal.
JP 2006-180269 A JP-A-2005-006066

  For example, a pixel having red (R), green (G), blue (B), and white (W) color filters is arranged in a 2 × 2 array as in the imaging device shown in Patent Document 2, and this array is two-dimensional. It is possible to provide an infrared pass filter that passes infrared light (IR) only in the white (W) pixels. Accordingly, an R / G / B visible light image and a (W + IR) infrared light image can be output from one imaging device.

FIG. 13 shows the configuration of the image processing unit described in Patent Document 1.
The illustrated image processing unit 100 includes a gain adjustment unit 101, a low-pass filter 102 as an NR processing unit, a high-pass filter 103, and an image synthesis unit 104.

  The gain adjustment unit 101 adjusts the gain of the input visible light image (Visible). Thereby, the gradation value (pixel value) for each pixel of the visible light image (Visible) imaged darkly due to insufficient exposure is increased to a pixel value close to the image imaged with appropriate exposure. As a gain adjustment method, there are a method of multiplying a pixel value of a visible light image (Visible) by a constant, a gamma correction based on an exponential function, an arbitrary gain adjustment method based on a polynomial function, and the like.

The low-pass filter 102 includes an edge detection unit, and the edge detection unit detects an edge from an infrared light image (infr) having a larger amount of information. The low-pass filter 102 serving as a noise reduction unit removes noise from the visible light image (Visible) by performing low-pass filter processing while storing information at the detected edge, and the basic image (Base) is transferred to the image composition unit 104. Output.
On the other hand, the high-pass filter 103 extracts edges and details from the input infrared light image (infr), and outputs the obtained details image (Edge) to the image composition unit 104.
The image synthesis unit 104 synthesizes the basic image (Base) from the low-pass filter 102 and the detailed image (Edge) from the high-pass filter 103 to obtain an output image (OUT).

  Thus, the technique described in Patent Document 1 uses, for example, a visible light image (Visible) and an infrared light image (infr) obtained from a pixel signal from the imaging device described in Patent Document 2, It is possible to perform NR processing on a visible light image (Visible).

  However, the infrared light image (infr) has a large amount of information, so it is generally easy to determine the edge. On the other hand, depending on the color temperature of the light source and subject conditions (infrared reflectance, etc.) Exists. This is because the combination of colors of the visible light component included in the infrared light image (infr) with the edge as a boundary blurs the edge.

FIG. 14A shows an A (W + IR) image as an infrared light image (infr), and FIG. 14B shows an R component (R image) of an input image as a visible light image (Visible). This is shown in the example.
In the R image shown in FIG. 14B, the edge between the brown palette and the dark brown lattice frame exists relatively clearly at the boundary B1 displayed as “with edge”.
On the other hand, in the A image shown in FIG. 14A, the G component, B component, and IR component are superimposed in addition to the R component at the location B1 corresponding to “with edge” in FIG. Since the color is configured, the R component difference is hidden behind other components that do not have a difference, and the edge becomes extremely unclear. As a result, “no edge” is determined.
Edge blurring in the same A pixel also occurs at other boundaries B2 (blue and lattice frame boundary), boundary B3 (boiled brown and lattice frame boundary), boundary B4 (light brown and lattice frame boundary), etc. ing.

  Note that this edge blurring may occur even in a monochrome image. This is because when the IR component is superimposed and the ratio thereof is large, the original edge information is diluted.

  Further, in the edge enhancement filter described in, for example, Japanese Patent Application Laid-Open No. 2000-13642, when edge blurring occurs in the color image or the monochrome image described above, the edge enhancement is not performed although the edge is present. The inconvenience that is weak is caused.

  As described above, the present invention avoids the inconvenience due to the edge determination of only the pixel signal that the information amount is large because it includes the near infrared light component.

An image input processing apparatus according to the present invention includes an imaging device, an infrared light reduction unit, an edge information acquisition unit, an edge determination unit, and a filter processing unit.
The imaging device receives a light that has passed through an infrared cut filter to reduce a near-infrared light component and outputs a first pixel signal including a visible light component, and an infrared cut filter The pixel array is formed by regularly arranging second pixels that output second pixel signals including both visible light components and near-infrared light components by receiving light that does not pass therethrough .
The infrared light reducing unit processes the first pixel signal to reduce a near infrared light component remaining after passing through the infrared cut filter.
The edge information acquisition unit outputs a first difference that is a difference between two first pixel signals output from two first pixels and two second pixels corresponding to the two first pixels. And a second difference that is a difference between two second pixel signals, and a third difference that is a difference between two adjacent signals on the same color data array of the signal from the infrared light reduction unit. By doing so, edge information is acquired.
The edge determination unit compares each of the first difference, the second difference, and the third difference with a predetermined reference value, and determines the presence or absence of an edge based on the comparison result .
The filter processing unit changes a predetermined basic filter coefficient used when there is no edge when the edge determination unit determines that there is an edge , and the first pixel signal processed by the infrared light reduction unit Is filtered.

Good Mashiku, the infrared light reducing portion, Ru color correction unit der to perform color correction by reducing the near-infrared light component above remains in the second pixel signal.

The pixel signal processing circuit according to the present invention includes an infrared light reduction unit, an edge information acquisition unit, an edge determination unit, and a filter processing unit.
The infrared light reduction unit receives the light that has passed through the infrared cut filter, reduces the near-infrared light component, and processes the first pixel signal including the visible light component, thereby processing the infrared light. The near infrared light component remaining after passing through the cut filter is reduced.
The edge information acquisition unit receives a second pixel signal including both a visible light component and a near infrared light component by receiving light that does not pass through an infrared cut filter, and the first pixel signal , respectively. Two pixels of the second pixel signal corresponding to the two pixel data obtained by obtaining a first difference from two adjacent pixel data on the same color data array of the first pixel signal The second difference is obtained from the data , and the edge information is obtained by obtaining a third difference that is a difference between two adjacent signals on the same color data array of the signal from the infrared light reducing unit .
The edge determination unit compares each of the first difference, the second difference, and the third difference with a predetermined reference value, and determines the presence or absence of an edge based on the comparison result.
The filter processing unit changes a predetermined basic filter coefficient used when there is no edge when the edge determination unit determines that there is an edge , and the first pixel after processing by the infrared light reduction unit Filter the signal.

The edge detection method according to the present invention reduces the near-infrared light component by receiving the light that has passed through the infrared cut filter, and passes the first pixel signal including the visible light component and the infrared cut filter. Obtaining a second pixel signal including both a visible light component and a near-infrared light component by receiving no light, and processing the first pixel signal to pass through the infrared cut filter. steps and, the first difference is a difference between the two first pixel signals output from the two first pixel, 2 corresponding to the two first pixel to reduce the near-infrared light component remaining after obtains a second difference is a difference between the two second pixel signal output from one of the second pixel, on the same color data array of the first pixel signal after the infrared light component is reduced The difference that is the difference between two adjacent signals A step of acquiring three differences, a step of comparing each of the first difference, the second difference, and the third difference with a predetermined reference value, and determining the presence or absence of an edge based on a result of the comparison. Including.

  In the present invention having the above-described configuration, the edge information acquisition unit is configured to perform the first difference and the second difference related to the same edge, ie, the second difference, from the first pixel signal including the visible light component and the second pixel signal including the near infrared light component. Get two differences. Then, the edge determination unit determines the presence / absence of an edge using the larger of the two differences.

  Since the first pixel signal includes a visible light component, the first pixel signal is filtered. Examples of the filter processing include noise reduction and edge enhancement. At that time, for example, “with edge” for specifying a portion for storing information or an image portion (edge portion) corresponding to a portion to be enhanced, A determination result of “no edge” is used. Specifically, in the filter process, the filter process is performed using the basic filter coefficient set except for the edge part. However, in the edge part, the filter process is executed after the basic filter coefficient set is changed. For this reason, for example, the noise reduction and the edge enhancement are performed on the edge portion.

  In the present invention, the larger one of the first difference and the second difference, that is, the difference indicating that the edge is clearer is used to determine whether or not there is an edge.

  According to the present invention, the edge determination of only the pixel signal that includes a near-infrared light component and that the amount of information is large avoids the inconvenience that the edge determination cannot be performed even though the edge actually exists, and is always appropriate. The advantage that the filtering process can be performed on the edge portion is obtained.

Embodiments of the present invention will be described below with reference to the drawings.
<< First Embodiment >>
FIG. 1 is a block diagram of a camera apparatus incorporating a pixel signal processing circuit according to an embodiment of the present invention.
The camera device corresponds to one aspect of the “image input processing device”. The camera device may be either a video camera mainly for moving image shooting or a digital still camera mainly for still image shooting.

  The illustrated camera apparatus includes an optical component 1 including a lens and an optical filter, an imaging device 2, an analog front end (AFE) circuit 3 for processing analog pixel signals, and A / D conversion of analog pixel signals into digital signals. An AD converter (ADC) 5 that outputs a digital video signal to various signal processing units, and a D / A converter that D / A converts the video signal subjected to various signal processing into an analog signal and outputs it as a video signal 14 ( DAC) 12.

  The optical component 1 is a component in which a lens, a diaphragm, and the like are housed in a lens housing. The optical component 1 can perform focus control and aperture control for exposure amount control, and an aperture drive unit for exposure control, a drive unit for automatic focus, and control circuits thereof are also included in the optical component 1. It is.

  The imaging device 2 is a CCD sensor or a CMOS image sensor. The imaging device 2 is fixed in the camera apparatus main body so that light (image) from a subject incident through the optical component 1 can be imaged on its own imaging surface. In addition, the imaging device 2 has a pixel array in which a large number of photosensors are arranged in a matrix and a set (pixel unit) of several photosensors adjacent to the light incident side of the imaging surface of the pixel array. And an on-chip multilayer filter formed to be

When the imaging device 2 is a CCD sensor, a timing generator (TG) 13 is connected to the imaging device 2. The imaging device 2 can receive a clock signal, a vertical synchronization signal, and a horizontal synchronization signal from the TG 13. When the imaging device 2 is a CCD, a transfer pulse synchronized with a vertical synchronization signal or a horizontal synchronization signal is also supplied from the TG 13.
The TG 13 is a circuit that generates these timing control signals from a system clock signal (not shown) under the control of the microcomputer 10. The imaging device 2 controls various operations such as a transfer operation and a shutter speed change by these timing control signals.
Note that when the imaging device 2 is a CMOS sensor, the function of the TG 13 can be provided in the imaging device 2.

  The optical filter included in the optical component 1 is, for example, for blocking high-frequency components higher than the Nyquist frequency in order to prevent aliasing distortion, and does not have an infrared cut filter function. The on-chip multilayer filter of the imaging device 2 has the function of the infrared cut filter.

FIG. 2 shows output spectral characteristics of an imaging device having an on-chip multilayer filter. In this graph, the horizontal axis indicates the wavelength of incident light, and the vertical axis indicates the output level when the pixel signal of each color is, for example, 8 bits, that is, the output gradation value of 0 to 1024.
This imaging device has red (R), green (G), and blue (B) even in a region where the frequency is higher than the lower limit of the near-infrared light region (700 to 800 [nm], indicated by 750 [nm] in the figure). It can be seen from FIG. 2 that both white and white (W) have sensitivity.

  In the present embodiment, although details are omitted, the imaging device 2 may be a spectral image sensor using a diffraction grating, in which case the separation of visible light and infrared light is incomplete, and as a result The accuracy of edge detection is reduced. In the following description, the description will be continued with an example in which the imaging device 2 includes an on-chip multi-layer filter.

FIG. 3A and FIG. 3B show two examples of one pixel unit having the color arrangement of the on-chip multi-layer filter of the imaging device 2.
The color arrangement is not limited to these two examples, but a typical primary color filter will be described here. Note that the color filter may be an array other than the illustrated example in the primary color filter, or may be a complementary color filter in which a plurality of arbitrarily selected complementary colors are regularly arranged, as variously proposed.

The color (selection) layer of the on-chip / multi-layer filter has a color arrangement shown in FIG. 3A or FIG. 3B, for example.
The color arrangement shown in FIG. 3A is referred to as “W checkered”. The color arrangement shown in FIG. 3B is referred to as “W checkered zigzag”. “W checkered” and “W checkered zigzag” include white (W) pixels having sensitivity in the wavelength region covering all detection wavelength regions of the G pixel, the R pixel, and the B pixel, and the W pixels are arranged in the checkered pattern. ing.
On the other hand, the IR cut layer of the on-chip / multi-layer filter is configured such that the G pixel, the R pixel, and the B pixel are IR cut and the W pixel is IR transmissive.

  Hereinafter, the description will be continued on the assumption that the on-chip multilayer filter has the function of a primary color filter.

  In general, near infrared rays have a wavelength of 0.7 to 0.8 [μm] (limit on the long wavelength side of visible red light) to 2.5 [μm] (or 1.5 to 3 [μm]). The near-infrared light component has a harmful effect such as making the color whitish. Therefore, in the first pixel signal (pixel signal for each pixel of red (R), green (G), and blue (B) (hereinafter referred to as R0, G0, B0 signals)), the infrared light component IR is Need to be removed. However, it is difficult to completely block the near-infrared light component by the IR cut filter. Usually, the output spectral characteristics of the imaging device are red (R) and green in the near-infrared light region as shown in FIG. Each of (G), blue (B), and white (W) has sensitivity. From the wavelength region where the wavelength is slightly lower than 700 [nm] to the long wavelength side, there is almost no human visual sensitivity. Therefore, in the imaging device 2 having the output spectral characteristics shown in the figure, the IR cut layer (or IR cut filter) is designed to suppress the long wavelength side from, for example, the vicinity of 650 [nm] indicated by a thick dashed line. .

FIG. 4 shows an example of the structure of an on-chip multi-layer filter formed integrally with the imaging device 2 for realizing, for example, the transmission spectral characteristics shown in FIG.
Each filter shown in FIGS. 4A and 4B is a layer having a predetermined thickness and a different refractive index between adjacent layers on the incident surface side where light (image) from a subject is incident on the imaging device 2. In this example, the dielectric films 21_1 to 21_5 (or 22_1 to 22_5) are stacked. The dielectric films 21_1 to 21_5 (or 22_1 to 22_5) have a five-layer structure, but the number of layers is arbitrarily set according to the function and performance to be obtained. Further, although the thickness is the same in the figure, the thickness and further the material can be arbitrarily determined according to the required function and performance.

FIG. 4A shows a part of an on-chip multi-layer filter that transmits transmitted light L3 in a specific wavelength region corresponding to a specific color from incident light L1 (here, white light) to the light receiving unit side, and the like. The structure of the color filter that reflects the color component (wavelength region other than the specific wavelength region in the visible light region) as reflected light L2 is shown. This color filter is formed in the R pixel, G pixel, and B pixel shown in FIG. 3 with the material, film thickness, number of layers, and the like of each layer of the dielectric films 21_1 to 21_5 determined according to the transmitted color. . On the other hand, since the color filter shown in FIG. 4A is not provided for the W pixel, light of all wavelengths is guided to the light receiving unit side.
The color filter shown in FIG. 4A is optimally designed so that there is almost no loss of transmitted light amount. Therefore, when the same incident light (white light) is incident, the total of L1R0, G0, and B0 signals (R0 + G0 + B0) The white (W) pixel signal (hereinafter referred to as the W signal) has substantially the same value in the visible light region, that is, is equivalent.

FIG. 4B shows another part of the on-chip multi-layer filter, which reflects most of the infrared light component IR from the incident light L1 (here, white light) and receives the visible light VL. The structure of the infrared cut filter which permeate | transmits to a part side is shown. The infrared reflection performance of the infrared cut filter is determined by the material, film thickness, number of layers, etc., of each layer of the dielectric films 22_1 to 22_5. The four types of pixels (R pixel, G pixel, B pixel and (W pixel) is formed such that the infrared light component IR can be removed.
The infrared cut wavelength, that is, the reflection center wavelength of the infrared light component IR can be controlled by the material, film thickness, number of layers, and other means of each layer.
Further, in the R, G, and B pixels in FIG. 3, the infrared light component IR is reduced by providing the infrared cut filter in FIG. 4B, but in the W pixel, the infrared cut in FIG. 4B is performed. Since no filter is provided, the infrared light component IR is also incident on the light receiving portion and photoelectric conversion is performed.

  Returning to FIG. 1, the AFE circuit 3 to which the pixel signal (analog signal) from the imaging device 2 is input is a process to be performed on the analog signal, for example, removal of reset noise by correlated double sampling (in the case of CCD). This is a circuit that performs other noise removal, amplification, and the like.

  The ADC 5 converts the processed analog signal into a digital signal having a predetermined bit, for example, 8 or 10 bits. This digital signal includes a pixel signal string having a gradation value of the predetermined bit for each pixel. That is, the digital signal output from the ADC 5 includes an alternating pixel signal sequence of A pixels and pixels of other colors.

The signal processing unit 4 is an aspect of a “pixel signal processing circuit”, and includes an edge information acquisition unit (EDGE) 4C, an edge determination unit (E_DET) 544, and filter processing as a configuration for processing a digital signal from the ADC 5. Part (FIL) 4E.
For example, the signal processing unit 4 may be integrated as an IC as a semiconductor chip, or may be provided as a module or a mounting substrate on which a plurality of components are mounted, and may be incorporated in the camera device. The signal processing unit 4 is connected between the ADC 5 and the DAC 12, and includes the signal amplification, noise reduction (NR) processing, separation of luminance signal and color signal, luminance signal processing, and color signal processing as preprocessing as the various signal processing described above. Etc. Between the signal processing unit 4 and the DAC 12, a processed luminance signal and color signal mixing circuit (YC_MIX) 9 is connected. The signal processing unit 4 including the signal processing unit 4 and the mixing circuit 9 may be used as the signal processing unit 4, and may be integrated into an IC or a module.

FIG. 5 illustrates a part (NR processing unit) 54A of the signal processing unit 4 including the edge information acquisition unit 4C, the edge determination unit 544, and the filter processing unit 4E illustrated in FIG. A configuration example is shown.
The illustrated NR processing unit 54A separately inputs a first pixel signal including a visible light component and a second pixel signal including a near infrared light component.
Here, a primary color pixel signal is taken as an example, and the first pixel signal is an R0, G0, B0 signal. The second pixel signal includes a W signal as a visible light component equivalent to the visible light component (R0 signal + G0 signal + B0 signal) of the first pixel signal, and a pixel signal of the near infrared light component IR (hereinafter referred to as IR). Signal). Here, the second pixel signal is represented by a signal obtained by adding the W signal and the IR signal (referred to as A signal or A pixel signal).
Note that the IR signal includes most of the near-infrared component of the light from the subject. That is, since the optical component 1 and the imaging device 2 shown in FIG. 2 do not have at least an IR cut filter in the light receiving path, almost all of the near-infrared light component from the subject is removed except for some loss. It is included in the IR signal.

  As illustrated in FIG. 5, the NR processing unit 54A includes a noise reduction (NR) unit 4B, an edge information acquisition unit 4C, and a synthesis processing unit 4D.

  The edge information acquisition unit 4C is a circuit that acquires edge information (for example, a pixel value difference between adjacent pixels) ED from the A pixel signal and outputs it to the NR unit 4B. In this example, the first edge acquisition unit (EDGE1) 4C1 And a second edge acquisition unit (EDGE2) 4C2.

The first edge acquisition unit 4C1 inputs R0, G0, B0 signals (first pixel signals) including visible light components, calculates the difference between the pixel data (gradation values) of the R0, G0, B0 signals, A first difference ED1 is obtained. Similarly, the second edge acquisition unit 4C2 inputs an A signal (second pixel signal) including a near-infrared light component, calculates a difference between pixel data (tone values) of the A signal, and calculates the first difference. Find ED1.
The first and second edge acquisition units 4C1 and 4C2 operate in synchronization with each other by a clock signal, and obtain a difference in pixel data between two pixels belonging to the same location on the imaging screen, that is, the same pixel unit pair. The pixel unit pair to which the pixel taking the difference belongs is preferably two pixel units adjacent in the horizontal direction or the vertical direction, but may be separated by several units.
The obtained first and second differences ED1, ED2 are output from the first and second edge acquisition units 4C1, 4C2 to the NR unit 4B.

  The NR unit 4B performs noise reduction processing on the input R0, G0, B0 signals. As a configuration for noise reduction processing, the NR unit 4B includes an edge determination unit (E_DET) 544 and a filter processing unit (FIL) 4E.

The edge determination unit 544 receives the first difference ED1 from the first edge acquisition unit 4C1 and the second difference ED2 from the second edge acquisition unit 4C2, and compares each difference with a predetermined reference value. As the reference value, it is preferable to use at least one reference that determines the boundary between the presence and absence of an edge.
The edge determination unit 544 can determine that “there is an edge” when the difference exceeds (or exceeds) the reference, and “there is no edge” when the difference is less than (or less than) the reference.

  The filter processing unit 4E performs NR processing at a portion other than the edge portion while preserving the edge gradation difference at the image location (edge portion) recognized based on the acquired first and second differences ED1 and ED2. In order to preserve the edge gradation difference, the NR unit 4B performs almost no noise reduction processing at the edge unit, or applies NR processing more weakly.

More preferably, the second edge acquisition unit 4C2 has a function of extracting local detailed information (edge / texture information ET) of the edge part from the A pixel signal. Further, in order to reflect the extracted edge / texture information ET in the R, G, B image, the synthesis processing unit 4D that combines the edge / texture information ET with the R0, G0, B0 signals after the NR processing includes the NR processing It is provided in the part 54A.
For this reason, the image indicated by the output signal from the synthesis processing unit 4D is a high-quality image in which edge gradation is preserved, noise is reduced, and deterioration in image quality at the edge portion is prevented.

In the present embodiment, all configurations in the previous stage in the signal processing order correspond to one aspect of the “pixel signal generation unit” as compared with the NR processing unit 54A illustrated in FIG.
That is, in FIG. 1, the ADC 5, the AFE circuit 3, the imaging device 2, and the optical component 1 correspond to the “pixel signal generation unit”.

  The imaging device 2 discharges red (R), green (G), blue (B), and All (A = W + IR) pixel signals as time-series serial signals in the scanning order during display. A configuration for separating the first pixel signal (for example, R0, G0, B0 signal) and the second pixel signal (for example, A pixel signal) from this signal is required before the NR processing unit 54A in the signal processing unit 4. is there.

The NR unit 4B inputs the separated first pixel signal (R0, G0, B0 signal) for each color, and outputs each pixel signal (R0 signal, G0 signal, B0 signal) constituting the R0, G0, B0 signal. When executing the process, the target pixel of the process is sequentially changed and the process is repeated. The NR unit 4B performs noise reduction when it is determined based on the edge information acquired by the edge information acquisition unit 4C that there is no edge in each target pixel and its surroundings, and each target pixel and its surroundings are determined. When it is determined that there is an edge, noise reduction is not effectively performed.
Such processing can be executed by a low-pass filter. At this time, in particular, an edge preserving filter such as a cross bilateral filter is used to preserve the edge gradation difference. The cross bilateral filter will be described in detail in the second embodiment below.

The R0, G0, and B0 signals after NR processing are separated into a luminance signal and a color signal, and after the predetermined processing is performed on each of the separated signals, the signals are output from the signal processing unit 4.
The mixing circuit 9 is a circuit that mixes (synthesizes) the luminance signal and the color signal processed by the signal processing unit 4 to generate a video signal.
The DAC 12 is a circuit that converts a video signal into an analog video signal 14 and outputs the analog video signal 14.

The microcomputer 10 is a circuit that controls the imaging device 2, the AFE circuit 3, the signal processing unit 4TG13, and all other components.
The microcomputer 10 is connected to a rewritable memory that holds control parameters and the like, for example, a nonvolatile memory (NVM) 11.
Note that a monitor display unit, a circuit that further encodes the video signal 14 and outputs it to the monitor display unit, a circuit for processing and outputting audio, and the like are not shown in FIG.

According to the present embodiment, the edge determination is performed using either the A (W + IR) signal and the R0, G0, B0 signals. As a result, the edge determination based on the A image can correctly determine the edge even at the color boundary erroneously determined as “no edge”.
As described above, more efficient and accurate filter processing can be executed.

  In FIG. 5, a configuration for performing other processing such as gain adjustment and color correction may be provided between the input terminals of the R0, G0, and B0 signals and the filter processing unit 4E.

<< Second Embodiment >>
In the second embodiment, a configuration desirable when including a color correction unit and an operation of a cross bilateral filter as a desirable filter processing unit 4E will be described in detail. Although the adoption of the color correction unit and the cross bilateral filter is arbitrary, in a more realistic embodiment, it is desirable to have the color correction unit and the cross bilateral filter.

FIGS. 6A and 6B show two examples of the configuration of the NR processing unit of this embodiment. These drawings correspond to FIG. 5 of the first embodiment, and the same components are denoted by the same reference numerals.
The NR processing unit 54B illustrated in FIG. 6A includes a gain adjustment unit 4A and a color correction unit CC) 4F before the filter processing unit 4E.

Here, the R1, G1, and B1 signals input to the gain adjusting unit 4A are different from the R0, G0, and B0 signals input to the NR processing unit 54A in FIG. 5 according to the first embodiment.
Since the R0, G0, and B0 signals are generated by receiving light from the infrared cut filter shown in FIG. 4B at the light receiving portion in the pixel and performing photoelectric conversion, the infrared light component IR is generated. Does not contain much. Even if it is included, an infrared light component IR corresponding to a relatively small amount of light transmitted without being reflected by the infrared cut filter is included in the R0, G0, and B0 signals.

On the other hand, the imaging device 2 that outputs R1, G1, and B1 signals does not include the infrared cut filter shown in FIG. Therefore, the R1 signal from the R pixel, the G1 signal from the G pixel, and the B1 signal from the B pixel include the infrared light component IR in substantially the same manner as the light from the subject.
Further, in the imaging device 2 in the present embodiment, in addition to the R pixel, the G pixel, the B pixel, and the A pixel, for example, a black (BL) pixel is provided in the pixel unit. In the BL pixel, the material, thickness, number of layers, and the like of each of the dielectric films 21_1 to 21_5 are determined so that the color filter illustrated in FIG. 4A blocks light in the entire frequency region of visible light. ing. However, the color filter of the BL pixel transmits almost the entire amount of the infrared light component IR. As shown in FIG. 6A, an IR signal as a BL pixel signal is input and applied to the color correction unit 4F.

  The gain adjustment unit 4A has at least a gain amplifier (variable gain amplifier), and amplifies (or attenuates) the R1, G1, and B1 signals in accordance with a given gain value (gain adjustment). The gain value of the gain amplifier is controlled from the microcomputer 10 in FIG.

  For example, the color correction unit 4F independently inputs the A signal, the R1, G1, and B1 signals after gain adjustment, and the IR signal, and performs color correction from these signals. In this color correction, the R1, G1, and B1 signals that are input are subjected to processing represented by the following equations (1-1) to (1-3) using the IR signal, and R is used as the corrected primary color signal. , G and B signals are output.

[Equation 1]
R = R1-αr × IR (1-1)
G = G1-αg × IR (1-2)
B = G1-αb × IR (1-3)

Here, the symbols “αr”, “αg”, and “αb” are coefficients for red correction, green correction, and blue correction, respectively.
As can be seen from these equations, by removing the IR component in the color correction, it is possible to generate R, G, B signals with clear colors without being affected by infrared light. The generated R, G, B signals are sent from the color correction unit 4F to the filter processing unit 4E.

  On the other hand, in the NR processing unit 54C illustrated in FIG. 6B, the BL signal is not provided and the A signal is input to the color correction unit 4F. In this case, first, the pixel data of the R1, G1, and B1 signals are summed in the same pixel unit, and the obtained data (R1 + G1 + B1) is subtracted from the A pixel data (= W + IR) to obtain the infrared light component IR. To extract. Thereafter, R, G, B signals are generated using the above equations (1-1) to (1-3).

  However, since this method does not strictly separate the visible light component and the infrared light component IR in terms of spectral characteristics, it is particularly inaccurate for red correction near near infrared light. However, although the calculation burden increases somewhat, there is a great advantage that it is not necessary to provide BL pixels. Conversely, the method of FIG. 6A is advantageous in that the correction accuracy is higher.

6A and 6B employ different methods for detecting edge information.
As in the first embodiment, the NR processing unit 54B illustrated in FIG. 6A provides two first and second edge acquisition units 4C1 and 4C2 to acquire edge information.
On the other hand, the NR processing unit 54C illustrated in FIG. 6B further includes another third edge acquisition unit 4C3 as a configuration included in the edge information acquisition unit 4C. The third edge acquisition unit 4C3 calculates a difference (third difference ED3) from the output of the color correction unit 4F by the same method as the other edge acquisition units 4CA and 4CB, and uses the third difference ED3 as the edge determination unit 544. Output to.

As in the first embodiment, the edge determination unit 544 determines whether each of the three differences (ED1 to ED3) is greater than or equal to a predetermined reference, and performs the first implementation based on the difference that meets this condition. Edge determination is performed in the same manner as the form.
The edge determination result is sent to the filter processing unit 4E and used for the filter processing.

  Here, the reason why the edge information is acquired by the first edge acquisition unit 4C1 before color correction and before gain adjustment as shown in FIG. 6A will be described.

The gain adjustment unit 4A is provided to increase the gradation value of the entire screen when the screen becomes dark, for example, because the screen brightness, particularly the color temperature of the light source is different. For this reason, normally, gain values are uniformly set for the R1 signal, the G1 signal, and the B1 signal, and signal amplification is performed. Since the noise component increases at this time, such a gain adjustment unit that may cause a significant gain increase is arranged before the NR unit 4B. In addition, the color correction unit 4F performs color correction on the signal after gain adjustment, and the signal after color correction is as shown in the above-described equations (1-1) to (1-3). Since the IR component is subtracted from the color signal level, the signal level is lowered and the S / N ratio is lowered even if the noise level is the same.
Therefore, for example, as shown in FIG. 7B, the S / N ratio may be significantly lowered in the image after color correction as compared to the image before gain adjustment shown in FIG.

  For the above reasons, it is desirable to acquire edge information in front of the gain adjusting unit 4A as shown in FIG. However, in the case of a camera device or the like of a model that is assumed that there is no significant gain increase and the IR component is relatively small due to reasons such as specifying the light source to be used, as shown in FIG. The third edge acquisition unit 4C3 may be configured to acquire edge information after color correction.

In FIG. 6A, the A signal is used instead of the IR signal as shown in FIG. 6B, and conversely, in FIG. 6B, the A signal is used as shown in FIG. 6A. The use of IR signals is a matter that can be arbitrarily changed.
Further, the gain adjusting unit 4A does not need to be at the illustrated position, and can be omitted from the NR processing units 54B and 54C. In that case, if the noise level does not change significantly before and after the color correction unit 4F, the edge information acquisition unit 4C is configured by the first and second edge acquisition units 4C1 and 4C2 as shown in FIG. 5 and FIG. Instead, the edge information acquisition unit 4C can be configured by the second and edge acquisition units 4C2 and 4C3 of FIG. In this case, the first edge acquisition unit 4C1 is omitted.

  Hereinafter, a more detailed embodiment including the NR processing unit 54A which is a characteristic unit will be described.

[Configuration of signal processor]
FIG. 8 is a block diagram illustrating a configuration example of the signal processing unit 4.
The signal processing unit 4 illustrated is roughly divided into a PRE block 41 that performs preprocessing, a Y block 42 that extracts and processes a luminance signal (Y), a C block 43 that extracts and processes a color signal (C), and a screen An OPD (Optical Detector) block 44 for detecting brightness is connected to each microcomputer 10 (referred to as a CPU block in FIG. 8) via a parallel interface (PIO) 45. Each block shown in the figure is controlled under the control of the microcomputer 10, whereby processing such as PRE block processing, automatic exposure (AE), automatic white balance (AWB), and the like are executed.

  The PRE block 41 includes a digital automatic gain control (AGC) circuit 51, a shading / defect correction circuit 52, a delay line portion (indicated by “D” in the figure) 53, an NR (Noise Reduction) block 54, a pattern generation circuit 56, and And a black integration circuit 57.

The AGC circuit 51 is a circuit that adjusts the gain of an input digital pixel signal. The AGC circuit 51 is usually provided in order to obtain a signal amplitude suitable for processing in the subsequent stages, and amplifies the entire single pixel signal, that is, the visible light component and the near infrared light component uniformly. Is called.
In this example, since the NR block 54 includes a gain adjustment unit, the signal amplitude necessary for the processing of the Y block 42 and the C block 43 can be obtained. Therefore, the AGC circuit 51 can be omitted if there is no special circumstance such as, for example, the input amplitude is too small in the preceding stage of the NR block 54 and the required processing accuracy cannot be obtained.

  The shading / defect correction circuit 52 is a shading correction that corrects the difference in brightness caused by the difference in the light receiving position between the center and the side of the light receiving surface of the sensor (imaging device 2), and the pixel signal from the sensor is missing. This circuit corrects data.

The delay line unit 53 performs delay for several lines in units of video signal standard lines (horizontal pixel signal strings) having a predetermined number of pixels in the horizontal and vertical directions for the processing of the NR block 54. Circuit. For example, when processing of the NR block 54 requires a delay of 5 lines, four 1-line delay units are connected in series, each output of the 1-line delay unit, a line that is not delayed (a line that outputs the input as it is), Can be input to the NR block 54 in parallel.
Note that an image memory may be provided as an alternative to the delay line unit 53 to read data for the required number of lines. The specific configuration and operation of the NR block 54 will be described later.

In the present embodiment, since the imaging device 2 shown in FIG. 1 is a single plate type and has the color arrangement shown in FIG. 3, the pixel signal output from the imaging device 2 is red (R ), Green (G), and blue (B) pixel signals having color information and white (W) pixel signals are alternately mixed on the time axis. Therefore, the image represented by this pixel signal has a mosaic arrangement for each color. An image having such a mosaic color arrangement is referred to as a “mosaic image”.
If a mosaic image is used as it is for NR processing, there is a missing part of information, and accurate processing cannot be performed. Therefore, the NR block 54 has a function of demosaicing the mosaic image. “Demosaic” is a method of generating color information of a portion having no specific color information in a mosaic image for each color (a specific color mosaic image) from surrounding pixels having specific color information by interpolation processing or the like. This is a process of converting to a “demosaic image” having color information in the pixel corresponding portion. Although a detailed configuration for demosaicing is not shown, a circuit configuration for high-precision interpolation processing by repeating simple linear interpolation processing or color estimation and synthesis is generally employed.

The pattern generation circuit 56 is a circuit that generates a test pattern when the imaging device 2 is not connected.
The black integration circuit 57 is a circuit that detects the black level of the digital pixel signal.

The Y block 42 is a circuit that inputs and processes the demosaic pixel signal of the A pixel having the largest amount of information among the demosaic pixel signals output from the PRE block 41.
The Y block 42 includes a Y generating unit 61 that generates a luminance signal (Y) from the demosaic pixel signal of the A pixel, an aperture control generating unit 62 that generates an aperture control (abbreviated as aperture control) signal from the luminance signal (Y), and an aperture controller. A gamma (γ) correction unit 63 for the signal.
The luminance signal (Y) generated by the Y generator 61 is supplied to the OPD block 44. In the aperture controller 62, the luminance signal (Y) generated by the Y generator 61 is corrected to the luminance signal (Y) that emphasizes only the outline portion of the image, and the corrected luminance signal (Y) is γ corrected. To the unit 63. The gamma correction unit 63 outputs the luminance signal (Y) after the gamma correction to the mixing circuit 9 in FIG.

The C block 43 is a circuit that inputs and processes demosaic pixel signals of R, G, and B pixels.
The C block 43 includes an RGB matrix circuit 71, a white balance (WB) and gamma (γ) correction circuit (referred to as “WB / γ correction”) 72, and a color difference signal (RG) and (BG) conversion circuit. 73 (referred to as a color difference conversion circuit) and a generation circuit (referred to as a chroma generation circuit) 74 for chroma signals Cr and Cb.

The RGB matrix circuit 71 is a circuit that inputs a demosaic pixel signal of R, G, and B pixels and outputs a color signal (R / G / B signal) that is synchronized (synchronized) for each same pixel unit. . The R / G / B signal is output to the OPD block 44 and the WB / γ correction circuit 72.
The WB / γ correction circuit 72 performs gain balance for each color from the input R / G / B signal and performs white balance (WB) adjustment. At this time, the brightness information from the OPD block 44 is referred to. Further, color gamma (γ) correction is applied to each pixel intensity of the R / G / B signal after white balance. At this time, a numerical value “gamma (γ)” is used to represent the response characteristics of the gradation of the image. This numerical value is held in, for example, the nonvolatile memory 11 shown in FIG. 1 or a storage area in the microcomputer 10 and supplied to the WB / γ correction circuit 72 via the PIO 45 in FIG. Gamma correction is correction processing for correctly displaying the brightness and color saturation of a displayed image.

The color difference conversion circuit 73 is a circuit that converts the R / G / B signal after the gamma correction into color difference signals (RG) and (BG).
The chroma generation circuit 74 is a circuit that further generates chroma signals Cr and Cb from the output of the color difference conversion circuit 73. The generated chroma signals Cr and Cb are sent to the mixing circuit 9 shown in FIG.

The OPD block 44 includes a luminance integration circuit 44A that generates a luminance integration value used for, for example, automatic exposure control (AE), and an R / G / B that generates an R / G / B integration value for each color used for white balance, for example. B integration circuit 44B.
The luminance integration circuit 44A generates a luminance integration value by integrating the luminance signal (Y), for example, for one screen. The integrated luminance value is supplied to the aperture control circuit provided in the optical component 1 in FIG. 1 and the analog gain circuit built in the imaging device 2 via the microcomputer 10.
The R / G / B integration circuit 44B generates an R / G / B integration value by integrating the R / G / B signal for each color from the RGB matrix circuit 71, for example, for each screen. The R / G / B integral value is supplied to the microcomputer 10 via the PIO 45, and the result of calculating the WB gain is supplied to the WB / γ correction circuit 72.

[Details of NR block]
FIG. 9 is a more detailed block diagram of the NR block 54.
The illustrated NR block 54 includes a separation unit 541, a synchronization (synchronization) processing unit 542, an A demosaic processing unit 543, an RGB demosaic processing unit 545, a gain adjustment unit 4A, an NR unit 4B, an edge information acquisition unit (EDGE) 4C, A synthesis processing unit 4D and a color correction unit (CC) 4F are included. Among them, the A demosaic processing unit 543, the RGB demosaic processing unit 545, the gain adjustment unit 4A, the NR unit 4B, the edge information acquisition unit 4C, the synthesis processing unit 4D, and the color correction unit 4F are the NR processing unit 54B shown in FIG. include.

The separation unit 541 is a circuit that separates an A pixel signal that forms a mosaic image of A (W + IR) pixels and pixel signals (R1, G1, B1 pixel signals) of other colors.
The synchronization processing unit 542 is a circuit that inputs the separated A pixel signal and the R1, G1, and B1 pixel signals, performs synchronization (synchronization) processing, and outputs them. The A pixel signal is input to the A demosaic processing unit 543, and the R1, G1, and B1 pixel signals are input to the RGB demosaic processing unit 545.

The A demosaic processing unit 543 is a circuit that performs demosaic processing on the input A pixel signal and generates a demosaic A image. Similarly, the RGB demosaic processing unit 545 is a circuit that performs demosaic processing on the input R1, G1, and B1 pixel signals to generate demosaic R, G, and B images.
These demosaic processes may be simple demosaic processes such as a linear interpolation method, for example. More preferably, the demosaic processes are executed by a circuit for high-precision interpolation processing capable of repeating color estimation and synthesis, for example. Specifically, in addition to the interpolation processing dedicated circuit, the demosaic processing may be realized by a computer-based control unit such as a DSP and a program function for operating the control unit.

The edge information acquisition unit 4C acquires the first and second differences ED1 and ED2 from the demosaic A image by a predetermined method. At this time, the edge information acquisition unit 4C has a peripheral pixel composed of an arbitrary number of pixels (usually the same odd number in the horizontal and vertical directions) such as 3 × 3, 5 × 5,. Get edge information in a range. The target pixel is a processing target pixel that performs noise reduction by filter processing in the NR unit 4B. The target pixel sequentially changes in one direction in the order of input, for example, along the horizontal direction of the video signal. Each time the target pixel is sequentially changed to an adjacent pixel, the edge information acquisition unit 4C re-recognizes the peripheral pixel range so that the changed target pixel is centered and repeats edge information acquisition. The acquired first and second differences ED1, ED2 are given from the edge information acquisition unit 4C to the NR unit 4B.
As a specific method of acquiring edge information, a method of obtaining a difference in pixel value between a target pixel and other pixels in a peripheral pixel range centering on the target pixel can be employed.

  The color correction unit 4F performs color correction on the demosaic R, G, B images RD, GD, BD after gain adjustment, for example, by the method using the subtraction described above.

In this example, the NR unit 4B includes a cross bilateral filter. The basic filter configuration is a well-known two-dimensional LPF. For example, the LPF configuration disclosed in Patent Document 1 can be adopted.
At this time, edge determination is performed from the acquired first and second differences ED1 and ED2. The edge determination method has already been described. In addition, the edge determination unit 544 changes the filter coefficient based on the edge determination result so as to save more edge gradation differences. At this time, the original edge gradation difference can be emphasized. Then, the changed filter coefficient is applied to the two-dimensional filter processing of the R, G, B image output from the color correction unit 4F.
As described above, when the edge determination is performed from the first and second differences ED1 and ED2 of the demosaiced image including the A pixels and the two-dimensional filter processing of the R, G, and B images is performed using the edge determination result, the pixel value An edge gradation difference corresponding to the difference can be stored, which is more desirable. Such a two-dimensional filter that refers to image information different from the filter processing target is particularly referred to as a cross bilateral filter. Although the outline of the cross bilateral filter processing will be described later, since edge detection (edge information acquisition and edge determination) is performed using the demodulated A image or R, G, B image, the edge detection accuracy is high. Edge information is effectively preserved at the output of the cross bilateral filter. The processed R, G, B pixel signals are output to the synthesis processing unit 4D.

  The edge information acquisition unit 4C has a function of extracting edge / texture information ET from the A pixel signal (strictly speaking, the signal of the A image after demosaicing). The extraction range may be, for example, an area obtained by combining (merging) all peripheral pixel areas that are determined to be edges and include pixel value differences into one by AND processing. The edge / texture information ET is output from the edge information acquisition unit 4C to the synthesis processing unit 4D.

The synthesis processing unit 4D replaces the edge information of the R, G, B pixel signal input from the NR unit 4B with the edge / texture information ET input from the edge information acquisition unit 4C, thereby synthesizing the image (signal mixing). )I do.
The R, G, B images from the synthesis processing unit 4D are sent to the C block 43 for color signal processing, and the demosaic A image AD from the A demosaic processing unit 543 is sent to the Y block 42 for luminance signal processing. .

The gain adjustment unit 4A includes a gain amplifier GA.
The gain amplifier GA receives the demosaic R, G, and B images from the RGB demosaic processing unit 545, and each pixel signal (R pixel signal, G pixel signal, demosaic B image, demosaic R image, demosaic G image, demosaic B image) B pixel signal) is multiplied by a gain value G to uniformly change the signal amplitude. At this time, the gain value may be changed for each color. However, in this example, since a circuit for color balance correction is provided in the subsequent stage, the same gain value G is used here.

The gain adjusting unit 4A shown in FIG. 9 includes a part of the function of the microcomputer 10, calculates a gain value G based on predetermined brightness information (or reads out from the nonvolatile memory 11 or the like), and gain Set to changeable to amplifier GA.
The value of the gain value G is used, for example, for screen brightness adjustment. For example, automatic exposure control (AE) is performed based on the brightness information from the OPD block 44 of FIG. At this time, the microcomputer 10 executes backlight correction for brightening a dark subject and brightness correction according to the color temperature for brightening the entire screen when the visible light component is small due to the color temperature of the light source.

  Next, an outline of NR processing by the cross bilateral filter (including an example of the noise reduction method of the present invention) will be described first.

[NR processing (including edge detection method)]
FIG. 10 is a process flow diagram after imaging that schematically illustrates edge determination and noise reduction.
In step ST0 shown in FIG. 10, the imaging of the subject is performed by the imaging device 2 including A, R, G, and B color pixels in the same pixel unit.
Thereafter, A / D conversion is performed on the obtained pixel signal by the ADC 5 shown in FIG. 1 (step ST1), and then the demosaic processing is performed by the A demosaic processing unit 543 and the RGB demosaic processing unit 545 of FIG. Performed (step ST2). Note that the AFE processing by the AFE circuit 3, the separation processing by the separation unit 541, the synchronization processing by the synchronization processing unit 542, and other processing are not shown for the sake of drawing.

In step ST3, edge information acquisition is performed by the edge information acquisition unit (EDGE) 4C shown in FIG. 9, and subsequently, filter processing is performed in step ST4. The filtering process in step ST4 includes edge determination (step ST41), LPF coefficient setting (step ST42), and filtering (step ST43).
In parallel with the start of step ST3, the gain adjustment and color correction processes in step ST5 are started. First, steps ST3 and ST4 will be described.

The edge information acquisition (ST3) is executed, for example, by a difference calculation circuit (not shown) such as a subtractor included in the edge information acquisition unit 4C under the control of the microcomputer 10.
This difference calculation may be performed by the microcomputer 10 itself, but in the case of hardware, a difference calculation circuit is required. As shown in the demosaic A image AD of FIG. 10, the difference calculation circuit includes the target A pixel At and includes, for example, a 3 × 3 peripheral pixel range, the pixel value of the target A pixel At, and the periphery of the target A pixel At. The first difference ED1 is calculated with the pixel value of the peripheral A pixel Ap located at.

FIG. 10 shows a demosaic R image RD, demosaic G image GD, demosaic corresponding to the demosaic A image AD of 3 × 3, in addition to the demosaic A image AD corresponding to the 3 × 3 peripheral pixel range after demosaic. A B image BD is also shown.
The center pixel of the 3 × 3 demosaic R image RD is the target R pixel Rt. Similarly, the center pixel of the 3 × 3 demosaic G image GD is the target G pixel Gt, and the center pixel of the 3 × 3 demosaic B image BD is the target B pixel Bt.
These four target pixels were always obtained in the same pixel unit shown in FIG. 3A or FIG. 3B in the pixel array of the imaging device 2 at the same time specified by the clock signal. Is. When the processing target changes sequentially, the four target pixels are sequentially shifted, for example, by one pixel in the horizontal direction, and accordingly, the 3 × 3 peripheral pixel range for each color is also shifted by one pixel in the same direction. Is done.

The difference value calculation circuit calculates a difference diff (px, py) in eight directions illustrated in the demosaic A image AD. Here, the coordinates (px, py) represent local relative coordinates in the (x, y) absolute position coordinate system (corresponding to the pixel address of the display screen) taken as shown in the figure, and the target in the demosaic A image AD A relative position with respect to the A pixel At (a distance of one pixel is represented by “1”).
When the central pixel (target A pixel At) of the demosaic A image AD is set to A0 (x, y) in absolute position coordinates, the peripheral pixel can be expressed as Ap (x-xp, y-yp), and the difference To get the difference diff (px, py).
An array (numerical matrix) of the first differences ED1 as shown in the figure is obtained by performing the calculation of the differences eight times.

  The above is the calculation method of the first difference ED1 obtained from the demosaic A image AD, but the array (numerical value) of the second difference ED2 from the demosaic R image RD, the demosaic G image GD, and the demosaic B image BD by a similar method. Matrix) is calculated.

  The obtained first and second differences ED1 and ED2 (difference array) are sent from the edge information acquisition unit 4C to the NR unit 4B and are subjected to filter processing.

FIG. 11 is a diagram for explaining the concept of filter processing edge determination (ST41) and filtering (ST43). FIG. 12 is a diagram for explaining the concept of filtering LPF coefficient setting (ST42) and filtering.
As shown in FIGS. 11 and 12, the NR unit 4B includes a cross bilateral filter edge determination unit 544, a filter unit 546, and a filter setting unit 547.
The edge determination unit 544 may be configured by hardware (dedicated circuit), or the processing procedure may be realized by software processing by the microcomputer 10. The filter unit 546 is configured by hardware including an X-direction filter unit and a y-direction filter unit, each including a shift register, an adder, and a multiplier. The filter setting unit 547 reads a basic filter coefficient set from the non-volatile memory 11, for example, and modifies it. The processing procedure is realized by software processing by the microcomputer 10.

  The edge determination unit 544 inputs, from the edge information acquisition unit 4C, the first difference ED1 that is the difference (diff (px, py)) between the target A pixel At and the pixel value of the peripheral A pixel Ap located in the vicinity thereof. . Further, the edge determination unit 544 determines the difference (diff) between the target color pixel (any one of Rt, Gt, and Bt) and the surrounding color pixel (any one of Rp, Gp, and Bp (not shown)) located in the vicinity thereof. (px, py)) is input from the edge information acquisition unit 4C. Then, it is determined whether each of the first difference ED1 and the second difference ED2 exceeds a predetermined standard (or more). If both are large, the determination of “with edge” is made. In the case where an edge actually exists, in the present embodiment, since the two first and second differences ED1 and ED2 are used, there is a high possibility that one of them is determined to be “with edge”.

It should be noted that the first and second differences ED1 and ED2 are evaluated for variation, and if the variation is large to some extent, it is regarded as a noise component and may not be used for edge determination even when the difference exceeds the reference. .
In addition, even when there is an edge, the size of the edge is ranked according to a plurality of different criteria, and when it is judged as “with edge”, the degree of change of the filter coefficient is changed according to the rank. Is also possible.

In the filter coefficient setting (ST42), first, the filter setting unit 547 such as the microcomputer 10 reads the basic LPF coefficient W0 from the nonvolatile memory 11, for example. The filter setting unit 547 corrects the basic LPF coefficient W0 that has been read out, at the place where the edge determination unit 544 determines that “there is an edge”.
Specifically, as shown in FIG. 12, the basic LPF coefficient W0 is lowered at the determination point of “with edge” to calculate the corrected LPF coefficient W (x, px) in the x direction, and the calculated LPF The corresponding portion of the basic LPF coefficient W0 is replaced with the coefficient W (x, px). Similarly, the LPF coefficient W (y, py) in the y direction is calculated, and the corresponding portion of the basic LPF coefficient W0 is replaced with the calculated LPF coefficient W (y, py).

The rate at which the LPF coefficient is lowered may be determined in advance by the filter setting unit 547, or may be dynamically changed and controlled according to the magnitude of the difference value.
An example of how to obtain the LPF coefficient is expressed by the following equation (2) by taking the x direction as an example. The arithmetic expression in the y direction is obtained by replacing “x” in the following expression (2) with “y”. This arithmetic expression uniquely determines how to reduce the coefficient according to the variance (σ ² ).

  Here, each gradation value in the x direction of the A pixel data is expressed as “edge (x)” (see FIGS. 11 and 12).

  The filter unit 546 performs the filtering (ST43) of FIG. 10. For example, as shown in FIG. 11, in the X direction filtering, R, G, B pixel data (input in (x)) is input and output. Eject out (x). Here, since the input in (x) of the filter unit 546 is any of R pixel data, G pixel data, and B pixel data after gain adjustment and color correction by the gain amplifier GA, the noise level is large. As shown, the edge tone difference is obscured by noise.

  On the other hand, since the A pixel data includes W pixel data (equivalent to the sum of R, G, and B pixel data) and the near infrared light component IR, the amount of original information is large and there is no need to adjust the gain. Therefore, as shown in FIG. 11, the S / N ratio is high and the edge gradation difference is clear. The fact that the edge is unclear with the filter input and clear with the A image is the same for the input in (y) of y-direction filtering and the y-direction gradation value edge (y) of the A pixel data. This is the reason why the A pixel signal data is used for setting the LPF coefficient described above. It can be seen from the above-described equation (2) that the difference information of the A pixel signal data is reflected in the LPF coefficient W in the portion indicated by the broken line.

  The filter unit 546 uses the filter coefficient (weighting coefficient) set by the filter setting unit 547 to perform bilateral, that is, input in (x) (or input in (y) for each color in each of the x direction and the y direction. ) And the surrounding 8 pixel values are weighted, and the gradation value of the target pixel value is changed and output. The processing performed by the filter unit 546 is expressed by the following equation (3) by taking the x direction as an example. The arithmetic expression in the y direction is obtained by replacing “x” in the following expression (3) with “y”.

  At this time, the filter unit 546 repeatedly calculates and changes the LPF coefficient and filtering each time the target pixel is changed while sequentially forwarding the target pixel to the adjacent pixels. The change of the target pixel is completely synchronized with the edge information acquisition unit 4C. Perform the filtering represented by the above formula (3) in the peripheral pixel range centered on a certain target pixel, and when the filtering is completed, change to the next target pixel and perform the same processing on the changed target pixel And the surrounding 8 pixels.

FIG. 10 conceptually shows numerical values of the basic LPF coefficient W0. In FIG. 10, the coefficients are represented by relatively large integers for convenience of drawing, but are different from the actual coefficient levels.
When coefficients “1”, “3”, “1” and coefficients are aligned in the column direction (vertical direction of display pixels), if it is determined that the vicinity between the adjacent columns is the edge center, “1”, “3”, “1 "Is reduced. Here, for convenience of explanation, for example, the coefficient is multiplied by “0.2”, and the coefficient is changed to make the edge clearer. Then, the filter process is executed using the LPF coefficient set after correction in this way.

In a normal two-dimensional filter (bilateral filter), edge information diffuses to surrounding pixels when filtering is performed.
On the other hand, in the cross bilateral literal filter of this embodiment, in order to preserve the sharpness of the edge, the A pixel signal data is used and a part of the basic LPF coefficient for noise removal is modified. If the LPF coefficient optimized for noise removal is changed, the noise removal capability may be reduced. However, since the edge is detected locally when viewed from the entire screen, the edge determination point is used in this way. The entire noise can be reduced sufficiently without applying strong noise reduction.

When the above processing is repeated for a certain color while changing the target pixel in the pixel signal input order, that is, the display scanning order, the image of that color is retained in the x direction in FIGS. 11 and 12 while retaining its color information. As shown by the output out (x), it is close to the A pixel distribution. That is, a gradual change is retained as color information, but an abrupt or random change is smoothed, and noise components are reduced (removed or suppressed).
The same applies to the y direction and the other two colors.

The R, G, B signals after noise reduction (the signals of demosaic R, G, B images RD, GD, BD) are sent from the NR unit 4B to the synthesis processing unit 4D.
On the other hand, the edge / texture information ET is output from the edge information acquisition unit 4C to the synthesis processing unit 4D, and a part of the demosaic R, G, B images RD, GD, BD corresponding to the edge / texture information ET The image composition processing is executed by replacing the edge / texture information ET.

The R, G, and B images after image synthesis are output to the C block 43, and the demosaic A image AD is output to the Y block 42.
FIG. 10 shows the processing after image synthesis. As shown in the figure, white balance and gamma (γ) correction are first performed on the image after image synthesis, and the R, G, B images after the γ correction are processed. 9 may be configured to output to a Y block 42 for Y processing and a C block 43 for C processing.

<< Third Embodiment >>
The present embodiment shows that the filter processing unit may be a filter (band filter) for edge enhancement.
As the edge enhancement filter, for example, a configuration described in Japanese Patent Application Laid-Open No. 2000-13642 by the applicant of the present application can be employed. According to the publication, the level of the luminance signal is detected, and the filter coefficient corresponding to the luminance difference is obtained by calculation. The filter coefficient is input to a correction unit including a band filter, a delay unit, an amplifier, an adder in the output stage, and the like. The bandpass filter delays an input luminance signal by a shift register, and (N + 1) delayed outputs are paralleled by (N + 1) delay outputs from the N intermediate taps thereof and the input and output of the shift register. Enter one at a time. Each of these multipliers receives a predetermined correction coefficient and multiplies it by the delay output. This correction coefficient is appropriately changed according to the difference between input signals (in this case, luminance signals). The multiplication result is synthesized by the adder in the bandpass filter, and is combined with the delayed luminance signal by the adder in the output stage through an amplifier, and the edge-enhanced luminance signal is output from the adder in the output stage. Is done.

  Since this edge emphasis filter itself is known, a detailed circuit diagram is not shown, but in this embodiment, for example, FIGS. 1 and 5 used in the first embodiment, and FIG. 6A used in the second embodiment. The filter processing unit 4E shown in FIG. 6B can be replaced with, for example, the edge enhancement filter having the above configuration. Further, in the case of edge enhancement, the image replacement of the edge portion by the edge / texture information ET is unnecessary, and the above-described FIG. 5 and the synthesis processing unit 4D of FIGS. 6A and 6B are unnecessary, The output of the edge enhancement filter unit provided instead of the filter processing unit 4E is sent to the subsequent stage as it is. Further, the second edge acquisition unit 4C2 does not need to have the function of extracting the edge / texture information ET.

The process of the above Japanese Patent Laid-Open No. 2000-13642 is for the luminance signal, but here the second edge acquisition unit 4C2 calculates the difference, that is, the second difference ED2 with respect to the A signal (second pixel signal). To detect.
On the other hand, as in the first and second embodiments, for the R0, G0, and B0 signals as the first pixel signal, for example, the difference, that is, the first difference ED1 is used as the first edge acquisition unit 4C1 (and The third edge acquisition unit 4C3) detects. Edge determination based on the first and second differences ED1, ED2 is performed in the same manner as in the first and second embodiments.
In the process of the above Japanese Patent Laid-Open No. 2000-13642, the coefficient is obtained according to the difference, but here, the coefficient is changed to the determined coefficient at the place where “edge presence” is determined in the edge determination. However, the coefficient can be changed according to the difference.

  As described above, since the present invention has a feature in the edge determination method, it can be applied to both a noise reduction filter and an edge enhancement filter which are performed while preserving edge information.

In the camera devices (image input processing devices) of the first to third embodiments described above, the edge information acquisition unit 4C includes the first pixel signal (for example, R0, G0, B0 signals) including a visible light component, and the near infrared. A plurality of differences of a first difference ED1 and a second difference ED2 (possibly a third difference ED3) are acquired for the same edge from a second pixel signal (for example, A pixel) including a light component. Then, the edge determination unit 544 compares the two (or three) differences with a predetermined reference, and determines “with edge” or “without edge” according to the comparison result.
In the subsequent filter processing, the filter processing is performed using the basic filter coefficient set except for the edge portion. However, in the edge portion, the filter processing is executed after the basic filter coefficient set is changed. Therefore, for example, the noise reduction and the edge enhancement are performed on the edge portion.

For this reason, when imaging a subject, the amount of signal information is large, so the S / N ratio is high, the edge is clear (the image of the second pixel signal), and the edge is clear by combining the color components of the edge boundary. An image having a good edge of any other image (image of the first pixel signal) is used for edge determination.
In other words, depending on the color combination of the edge boundary, a noise reduction process that satisfactorily stores the edge information while avoiding the disadvantage of the background art that only the second pixel signal including infrared light cannot be determined, or Since high-precision edge enhancement processing is performed, a high-quality image is always output.

In particular, in the second embodiment, the signal amplification is preferably performed before input to the color correction processing in which the S / N ratio is decreased because the infrared light component IR is subtracted to reduce the signal level, and more preferably before the color correction. Since the first difference ED1 is detected from the first pixel signal before the gain adjustment to be performed, there is an advantage that the influence of noise in the difference calculation can be avoided as much as possible, and highly accurate edge determination is possible.
In the first and second embodiments, since the detailed image of the edge portion is extracted and synthesized as the edge / texture information ET, the noise reduction effect cannot be obtained at the edge portion, or the color is different due to filtering. However, the original noise level and color information can be saved.
In the edge determination for the edge enhancement filter according to the third embodiment, when edge smearing occurs, there is an inconvenience that the edge is not enhanced although the edge exists, and the degree of edge enhancement is weak. Can be avoided.

  Although the color image processing has been described above, the present invention can be similarly applied to a camera device that captures a monochrome image. In this case, the first pixel signal is a monochrome pixel signal from which the IR component is removed, and the second pixel signal is a signal in which the IR component is included in the monochrome pixel signal. In this case, since the second pixel signal has a higher S / N ratio because it includes an IR component, if the ratio of the IR component becomes extremely large depending on the imaging environment, it is difficult to detect an edge based on the second pixel signal. May be. In that case, in the first to third embodiments, it is possible to determine “there is an edge” by using the first difference ED1 detected for edge detection by the first pixel signal. Therefore, as with the color image described above, noise reduction processing that preserves edge information satisfactorily while avoiding the inconvenience of the background technology that only the second pixel signal including infrared light cannot be determined, or high accuracy Since the edge enhancement processing is performed, a high quality image is always output from the filter processing unit.

1 is a block diagram of a camera device incorporating a pixel signal processing circuit according to an embodiment of the present invention. It is a graph which shows the output spectral characteristic of the imaging device used for description of this embodiment. (A) and (B) are diagrams showing the color arrangement of an on-chip multilayer filter for one pixel unit. (A) is a schematic cross-sectional view showing a structural example of a color filter, and (B) is a schematic sectional view showing a structural example of an infrared cut filter. It is a block diagram of the NR process part in connection with 1st Embodiment. (A) And (B) is a block diagram of the NR process part in connection with 2nd Embodiment. (A) is an image before gain adjustment, and (B) is a diagram showing an image after color correction. It is a block diagram of a signal processing part. It is a block diagram which shows the detail of NR block. It is a processing flow figure after imaging which showed edge details and noise reduction typically in detail. It is a figure which shows notionally the relationship between edge determination and filtering. It is a figure which shows notionally the relationship between filter coefficient setting and filtering. FIG. 10 is a block diagram illustrating a configuration of an image processing unit described in Patent Document 1. (A) is an A image, (B) is a figure which shows R component (R image) of the input image of visible light by a color palette image, respectively.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Optical component, 2 ... Imaging device, 3 ... AFE circuit, 4 ... Signal processing part, 4A ... Gain adjustment part, 4B ... NR part, 4C ... Edge information acquisition part, 4D ... Composition processing part, 4E ... Filter processing part 4F ... color correction unit, 5 ... ADC, 9 ... mixing circuit, 10 ... microcomputer, 11 ... nonvolatile memory, 12 ... DAC, 13 ... TG, 14 ... video signal, 41 ... PRE block, 42 ... Y block, 43 ... C block, 53 ... delay line section, 54 ... NR block, 54A ... NR processing section, 541 ... separation section, 542 ... simulation processing section, 543 ... A demosaic processing section, 544 ... edge determination section, 545 ... RGB Demosaic processing unit, 546 ... filter unit, G ... gain value, ED1 ... first difference, ED2 ... second difference, ET ... edge / texture information

Claims (5)

By receiving the light that has passed through the infrared cut filter, the near infrared light component is reduced, and the first pixel that outputs the first pixel signal including the visible light component and the light that does not pass through the infrared cut filter An imaging device having a pixel array formed by regularly arranging second pixels that output a second pixel signal including both a visible light component and a near-infrared light component by receiving light ; and
An infrared light reduction unit that reduces the near-infrared light component remaining after passing through the infrared cut filter by processing the first pixel signal;
A first difference, which is a difference between two first pixel signals output from the two first pixels, and two second pixel signals output from the two second pixels corresponding to the two first pixels. Edge information by acquiring a third difference that is a difference between two adjacent signals on the same color data array of signals from the infrared light reduction unit. An edge information acquisition unit to acquire;
An edge determination unit that compares each of the first difference, the second difference, and the third difference with a predetermined reference value, and determines the presence or absence of an edge based on a comparison result;
When the edge determination unit determines that there is an edge, the predetermined basic filter coefficient used when there is no edge is changed, and the first pixel signal processed by the infrared light reduction unit is filtered. A filter processing unit to be applied;
An image input processing apparatus. The first pixel is a plurality of pixels of a primary color system or a complementary color system provided in a pixel unit which is a basic structural unit of a pixel array of the imaging device,
The image input processing device according to claim 1 , wherein the second pixel is a pixel that outputs a pixel signal including a visible light component equivalent to a sum of pixel values from the plurality of pixels. The infrared light reducing portion includes an image input apparatus according to claim 1 or 2 which is a color correction unit for performing color correction by reducing the near-infrared light component above remains in the second pixel signal. The near-infrared light component is reduced by receiving the light that has passed through the infrared cut filter, and the first pixel signal including the visible light component is processed to remain after passing through the infrared cut filter. An infrared light reduction unit for reducing near infrared light components;
By receiving light that does not pass through an infrared cut filter, a second pixel signal containing both a visible light component and a near-infrared light component and the first pixel signal are input, and the same first pixel signal A first difference is obtained from two adjacent pixel data on the color data array, and a second difference is obtained from the two pixel data of the second pixel signal corresponding to the two pixel data obtained from the first difference. An edge information acquisition unit that acquires edge information by acquiring a third difference that is a difference between two adjacent signals on the same color data array of signals from the infrared light reduction unit ;
An edge determination unit that compares each of the first difference, the second difference, and the third difference with a predetermined reference value, and determines the presence or absence of an edge based on a comparison result;
When the edge determination unit determines that there is an edge, a predetermined basic filter coefficient used when there is no edge is changed, and a filter process is performed on the first pixel signal processed by the infrared light reduction unit A filter processing unit for applying
A pixel signal processing circuit. By receiving the light that has passed through the infrared cut filter, the near infrared light component is reduced, and the first pixel signal including the visible light component and the light that does not pass through the infrared cut filter are received by the visible light. Obtaining a second pixel signal including both a component and a near-infrared light component;
Reducing the near-infrared light component remaining after passing through the infrared cut filter by processing the first pixel signal;
A first difference is a difference between the two first pixel signals output from the two first pixel, wherein two of the two second pixel signals output from the two second pixels corresponding to the first pixel And a third difference that is a difference between two adjacent signals on the same color data array of the first pixel signal after the infrared light component is reduced. Steps,
Comparing each of the first difference, the second difference, and the third difference with a predetermined reference value, and determining the presence or absence of an edge based on a result of the comparison;
Edge detection method including: