JP4309237B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging device, and more specifically, imaging using an imaging device such as a CCD sensor and a color filter, and interpolating a signal value of a color different from the filter color for each pixel based on the obtained image. The present invention relates to an imaging device that generates a color image.

In an image pickup apparatus using a single-plate image pickup device such as a CCD (Charged Coupled Device), only a single spectral sensitivity can be obtained even if the image is taken as it is. Therefore, in order to obtain a color image, light reception corresponding to each pixel is obtained. In general, photographing is performed by providing a color filter of a predetermined pattern (for example, a Bayer array of R (red), G (green), and B (blue) as shown in FIG. 7) on the light receiving surface of the element. ing.
Since each pixel has only a single color component, the image obtained by this imaging is a mosaic image (checkered pattern) image (color mosaic image) corresponding to the color filter pattern. For each pixel of the color mosaic image, other color components that the pixel does not have are interpolated using surrounding pixels, so that all the pixels have all the color components (R, G, B). An image that is possessed, that is, a color image is generated.

  As an interpolation processing method described above, that is, a processing method for interpolating other color components that a pixel does not have using surrounding pixels, for example, among signal values of four pixels adjacent to a pixel to be calculated, A method of averaging the signal values of two pixels excluding the maximum value and minimum value (median interpolation), a method of averaging the signal values of four adjacent pixels (average interpolation), and the like are known. .

By the way, according to a conventionally known interpolation processing method, it is possible to prevent the occurrence of false colors at a boundary where the pixel signal value changes greatly, such as a boundary between a subject image and a background image, like a vertical stripe pattern. There is a problem that can not be.
That is, in the conventional interpolation processing method, only the signal value of the pixel adjacent to the pixel of interest is considered during the interpolation processing, and therefore, appropriate interpolation processing cannot be performed. As a result, the signal value of the pixel A false color is generated at a boundary where greatly changes.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an imaging apparatus capable of removing a false color generated in a color image generated based on a color mosaic image captured by an imaging element. Is to provide.

  In order to achieve the above object, according to an aspect of the present invention, each pixel has the plurality of colors based on a plurality of color mosaic images captured by a single-plate image sensor and arranged in a mosaic pattern for each of a plurality of color components. An imaging device that generates a color image having a component, the adjacent pixel having a color component different from a color component of a color mosaic image corresponding to the one pixel and adjacent to the one pixel. A color interpolation amount calculation unit that performs a process of calculating a color interpolation amount for interpolating color components on each pixel, and a color interpolation amount that is calculated by the color interpolation amount calculation unit is below a predetermined first threshold value Color that calculates the average value of the color interpolation amounts calculated by the color interpolation amount calculation means for a plurality of pixels within a predetermined range from the target image among the pixels corresponding to the color mosaic image of the color component of the pixel Interpolation amount average calculation means and previous Color interpolation processing means for performing color interpolation processing for each pixel based on the color interpolation amount for each pixel calculated by the color interpolation amount calculation means, and the color interpolation calculated by the color interpolation amount average calculation means When the absolute value of the difference between the average value of the amount and the color interpolation amount of the target pixel is equal to or larger than a predetermined second threshold, the color interpolation processing unit performs color interpolation processing based on the target pixel. Not performed.

  Preferably, when the absolute value of the difference between the average value of the color interpolation amount calculated by the color interpolation amount average calculating means and the color interpolation amount of the target pixel is equal to or greater than a predetermined second threshold, The interpolation processing unit sets the color interpolation amount of the target pixel as an average value of the color interpolation amounts calculated by the color interpolation amount average calculating unit.

  According to the present invention, it is possible to remove a false color generated in a color image generated based on a color mosaic image picked up by an image pickup device, so that a good image quality can be obtained.

Embodiments Hereinafter, embodiments of an imaging apparatus according to the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a diagram illustrating an overall configuration of an imaging apparatus 1 in the present embodiment. The imaging apparatus 1 is roughly composed of an optical system, a signal processing system, a recording system, a display system, and a control system.

The optical system includes a lens optical system 10 and a CCD sensor 11.
The lens optical system 10 includes a lens facing the subject, and the optical image of the subject is collected by the lens to form an image of the subject on the CCD sensor 11.
The CCD sensor 11 includes a single-plate CCD image sensor provided with a color filter such as an RGB Bayer array. The color filter pattern may be other than the RGB Bayer array.

The signal processing system is a correlated double sampling circuit (CDS: Correlated Double) for reducing noise by sampling the electrical signal output from the CCD sensor 11.
Sampling CDS 12, A / D converter 13 that converts an analog signal output from CDS 12 into a digital signal, and predetermined image processing to be described later are performed on the digital signal output from A / D converter 13. And an image processing unit 14.
The digital signal input to the image processing unit 14 is mosaic image data (checkered pattern) for each color of RGB corresponding to the color filter pattern of the CCD sensor 11 described above.

  The recording system includes a memory 15 that stores a control program executed by the control unit 16 and compressed data of the image data generated by the image processing unit 14.

  The display system includes a D / A converter 18 that converts image data processed by the image processing unit 14 and stored in a built-in image memory, and an LCD (Liquid Crystal) that functions as a finder by displaying an input image. The display unit 19 including a display) is included.

  The control system is acquired by the timing generator 17 that controls the operation timing of the CCD sensor 11 to the A / D converter 13, the operation input unit 20 for inputting a shutter operation and other commands by the user, and the CCD sensor 11. By performing interpolation processing based on the color mosaic image, the image processing unit 14 that generates a color image in which each pixel has all the color components, and the control program stored in the memory 15 are read, and the read control And a control unit 16 including a CPU (Central Processing Unit) that controls the entire imaging apparatus 1 based on a user command input from the operation input unit 20 and the like.

In the imaging apparatus 1, an optical image of a subject is incident on a CCD sensor 11 via a lens optical system 10, and is photoelectrically converted by the CCD sensor 11 into an electrical signal. The obtained electrical signal is removed of noise components by the CDS 12, digitized by the A / D converter 13, and then temporarily stored in an image memory built in the image processing unit 14.
In a normal state, an image signal is constantly overwritten at a constant frame rate in the image memory built in the image processing unit 14 by the control of the signal processing system by the timing generator 17. The image signal of the image memory built in the image processing unit 14 is converted into an analog signal by the D / A converter 18, and a corresponding image is displayed on the display unit 19.

The display unit 19 also serves as a finder for the imaging apparatus 1. After the user presses (operates) the shutter button included in the operation input unit 20, the control unit 16 holds the image signal immediately after the shutter button is pressed to the timing generator 17, that is, image processing. The signal processing system is controlled so that the image signal is not overwritten in the image memory of the unit 14.
Then, the image data held in the image memory of the image processing unit 14 is compressed by a predetermined method and recorded in the memory 15. With the operation of the imaging apparatus 1 as described above, the acquisition of one piece of image data is completed.

FIG. 2 is a block diagram illustrating an image processing operation of the image processing unit 14 of the imaging apparatus 1 according to the present embodiment.
The interpolation processing unit 141 performs a process of interpolating a color component that each pixel does not have using the color components of surrounding pixels based on the mosaic image data of each RGB color obtained by the CCD sensor 11. Thus, a color image in which all pixels have all color components (R, G, B) is generated.
The interpolation vector amount calculation unit 142 generates an interpolation vector amount CV for each pixel based on the color component generated by the interpolation processing of the interpolation processing unit 141. The interpolation vector amount CV will be described later.
The interpolation vector amount adjustment unit 143 adjusts the interpolation processing result of the interpolation processing unit 141 based on the interpolation vector amount CV for each pixel calculated by the interpolation vector amount calculation unit 142, and the color image from which the false color is removed Is generated.
Hereinafter, each process executed in the image processing unit 14 will be described.

First, the interpolation processing executed in the interpolation processing unit 141 will be described with reference to FIG.
The interpolation processing unit 141 is an interpolation process for generating a color image in which each pixel has each RGB color component based on the mosaic image data of each RGB color obtained by the CCD sensor 11. FIG. 3 illustrates an example of the interpolation processing of the interpolation processing unit 141, where (a) shows R component interpolation processing, (b) shows G component interpolation processing, and (c) shows B component interpolation. Indicates processing.

First, in step ST1b in FIG. 3B, an interpolation operation is performed on a pixel that does not display G in step ST1b based on a pixel that displays G in step ST1b, thereby setting step ST2b. .
In the interpolation calculation, for example, among the signal values of four pixels (G) adjacent to the pixel (G) to be calculated, a method of averaging the signal values of two pixels excluding the maximum value and the minimum value ( Median interpolation) and a method of averaging signal values of four adjacent pixels (G) (average interpolation). In addition, a calculation method is known by using a function corresponding to the four pixels (G) and the pixel (R, B) adjacent to the pixel (G) to be calculated.
The G signal value for all the pixels is calculated by this interpolation calculation (step ST3b).

In step ST1a of FIG. 3A, the color difference signal Cr is calculated based on the signal value of the pixel displaying R in step ST1a and the G signal value of the same pixel (step ST2b), and is set as step ST2a. .
Next, in step ST2a of FIG. 3 (a), by performing an interpolation operation on the pixels not displaying Cr in step ST2a based on the pixels displaying Cr in step ST2a, step ST3a and To do. As the interpolation calculation, the median interpolation, the average interpolation, or the like described above can be applied.
When the color difference signals Cr are calculated for all the pixels in step ST3a, the R signal values for all the pixels are calculated based on the interpolated G signal value (step ST2b) (step ST4a). .

Similarly, in step ST1c of FIG. 3C, the color difference signal Cb is calculated based on the signal value of the pixel displaying R in step ST1c and the G signal value of the same pixel (step ST2b). Let ST2c.
Next, in step ST2c of FIG. 3 (c), by performing an interpolation operation on the pixels that do not display Cb in step ST2c based on the pixels that display Cb in step ST2c, To do. As the interpolation calculation, the median interpolation, the average interpolation, or the like described above can be applied.
When the color difference signal Cb is calculated for all the pixels in step ST3c, the R signal value of all the pixels is calculated based on the interpolated G signal value (step ST2b) (step ST4c). .
The interpolation process described above is an example, and other known interpolation processes can be applied. For example, in step ST2b, not only the G signal value but also a method of changing the value to be interpolated according to the R and B signal values of the interpolation target pixel may be used.

  As described above, in the interpolation processing unit 141, the signal values of the RGB components are calculated for all the pixels, but in this state, the false color is included, and this false color is calculated as the interpolation vector amount. This is removed by the unit 142 and the interpolation vector amount adjustment unit 143.

The interpolation vector amount calculation unit 142 calculates the interpolation vector amount CV based on the interpolation processing result executed by the interpolation processing unit 141.
The interpolation vector amount CV is the size (absolute value) of a color vector generated around one pixel by the correction processing of the interpolation processing unit 141, and the interpolation vector amount calculation unit 142 calculates the interpolation vector amount for each pixel. CV is calculated. Specifically, the color difference signals Cb (= Y−B) and Cr (= Y−) of the color components generated in the adjacent pixels by interpolation processing with respect to the target pixel (target pixel) for which the interpolation vector amount CV is calculated. R) is calculated for each direction (4 directions).

FIG. 4 is a graph showing the color vector of the pixel for which the interpolation vector amount CV is calculated. The vertical axis indicates the color difference signal Cb, and the horizontal axis indicates the color difference signal Cr.
In FIG. 4, the color difference signals Cb and Cr of the color component generated by the interpolation processing are applied to the pixels adjacent to the pixel as the origin for calculation of the interpolation vector amount CV in each direction (four directions). ) Shows the state calculated every time (color vectors V1 to V4).

The interpolation vector amount CV calculated by the interpolation vector amount calculation unit 142 is the maximum value of the scalar amount calculated for each direction (four directions).
In an area where the change in color signal value is large and a false color is likely to occur, such as an image of an object boundary, for example, as shown in FIG. 4, the color vector V2 and the color vector V4 are opposite to each other, and It becomes almost the same size. Therefore, in this case, the radius R of the circle C1 shown in FIG. 4 becomes the interpolation vector amount CV (radius R = max (| V1 |, | V2 |, | V3 |, | V4 |).

The interpolation vector amount adjustment unit 143 adjusts the interpolation processing result of the interpolation processing unit 141 based on the interpolation vector amount CV for each pixel calculated by the interpolation vector amount calculation unit 142, and the color image from which the false color is removed Is generated.
In the interpolation vector amount adjusting unit 143, the false color is removed by the following steps.
In the following description, the case of the color mosaic image of the G component will be described as an example, but the same applies to the color mosaic images of other color components.

Step 1: Identification of pixels Normally, false colors are likely to occur when the boundary of an object that is a subject cuts through the pixels, and the value of the generated false color interpolation vector amount CV is small. Therefore, in order to perform the process of the following steps only with respect to a pixel with high possibility of a false color, only the pixel which is the interpolation vector amount CV below a predetermined threshold value is made into the object of the following process.
That is, in this step, target pixels for the following processing are specified.

Step 2: Calculate the average value of the surrounding interpolation vector amount CV For the G pixel specified by the processing of Step 1, the average value of the surrounding interpolation vector amount CV of the pixel is calculated.
FIG. 5 is a diagram for explaining a method of calculating the average value of the interpolation vector amount CV.
As shown in FIG. 5, among the G pixels specified by the processing in step 1, one pixel is set as the target pixel, and the same color (G in FIG. 5) in a predetermined region (region n × n) of the target pixel. ). In the example shown in FIG. 5, when the upper left of the array is the origin as shown, the target pixel is G (54) and n = 5.
The average value of the interpolation vector amount CV around the pixel of interest is as shown by the following equation (1).

CV_M = CV_T / N (1)

However,
CV_M: Average value of interpolation vector amount CV of G pixels included in region n × n CV_T: Total value of interpolation vector amount CV of G pixels included in region n × n
N: number of G pixels included in the region n × n

Note that the size of N (the range with respect to the target pixel) can be arbitrarily set according to the imaging target and the like.

Step 3: False color determination False color determination is performed by determining the following equation (2) for the target pixel. That is, if the following expression (2) is satisfied, it is determined to be a false color, and if it is not satisfied, it is determined not to be a false color.

CV_A−CV_M ≦ TH1 (2)

However,
CV_A: interpolation vector amount CV of the target pixel
TH1: False color determination threshold value.
Note that the false color determination is sequentially executed for all the pixels to be processed in step 1 while changing the target pixel.

In this step, when the difference between the average value of the interpolation vector amount CV of the target pixel and the interpolation vector amount CV around the target pixel is equal to or smaller than a predetermined false color determination threshold, a certain range including the target pixel is the object range. Since it is considered that there is little change in the color, not the boundary portion, etc., the false color removal process described later is not performed in that case.
For example, as shown in FIG. 6A, when the contour line A of the object is away from the target pixel, the interpolation vector amount CV of the target pixel and the color occupying within a predetermined range (region n × n) Since the difference from the average value of the interpolation vector amount CV of each pixel of the mosaic image (G) is small and the color change is considered to be gentle in the area A1, in this case, false color removal described later is performed. No processing is performed.

Conversely, if the difference between the average value of the interpolation vector amount CV of the pixel of interest and the interpolation vector amount CV around the pixel of interest is greater than or equal to a predetermined false color determination threshold, This means that the pixel of interest is a singular point. In such a case, it is considered that a false color has been generated by the interpolation process based on the target pixel, and a false color removal process described later is performed.
For example, as shown in FIG. 6B, when the contour line B of the object cuts the target pixel vertically, the interpolation vector amount CV of the target pixel and the color mosaic occupying within a predetermined range (region n × n). Since the difference from the average value of the interpolation vector amount CV of each pixel of the image (G) is large and the color around the pixel of interest is considered to have a high frequency change, in this case, a false color removal process described later is performed. I do.

Step 4: False color removal processing A false color removal process is performed on the pixels determined to be false colors in Step 3.
Specifically, the interpolation vector amount CV for the target pixel is set to zero. That is, adjustment is performed so as to cancel the interpolation processing executed in the interpolation processing unit 141 for the target pixel for false color removal. As a result, since the achromatic process is performed on the target pixel, the false color is removed.

  On the other hand, as the false color removal processing, the interpolation vector amount CV of the target pixel for false color removal can be set to CV_M, that is, the average value of the interpolation vector amounts CV of the pixels around the target pixel. Thereby, the color component of the target pixel becomes the same as the color component of the background portion of the target pixel, and the color component generated by the interpolation processing based on the target pixel does not stand out compared to the surrounding color components. False color is removed.

  As described above, according to the imaging apparatus 1 of the present embodiment, the lens optical system 10 that forms an optical image of a subject on the CCD sensor 11 and the optical image formed by the lens optical system 10 are arranged in a Bayer array. A CCD sensor 11 that performs photoelectric conversion according to the arrangement pattern of the color filters to generate color mosaic images of R, G, and B colors, and an A / D converter 13 that converts the generated color mosaic image into digital data, Based on the digital data of the color mosaic image of each color of R, G, B, for each pixel of the color mosaic image, other color components that the pixel does not have are interpolated using surrounding pixels. Thus, an image processing unit 14 that generates an image in which all pixels have all color components (R, G, B), that is, a color image, is processed by the correction process. An interpolation vector amount CV, which is the size of the color vector generated by the surroundings, is calculated for each pixel, and for each pixel (target pixel) having a small interpolation vector amount CV that is considered to have a high possibility of false color. If the difference is equal to or smaller than a predetermined false color determination threshold value, the false color removal process is not performed. When the false color determination threshold is exceeded, a false color removal process is performed on the target pixel. Then, as the false color removal process, the image processing unit 14 sets the interpolation vector amount CV of the target pixel to 0, or sets the interpolation vector amount CV to the average interpolation vector amount of pixels around the target pixel. Configured.

Therefore, the following effects can be obtained.
That is, when the difference between the average value of the interpolation vector amount CV of the target pixel and the interpolation vector amount CV around the target pixel is equal to or smaller than a predetermined false color determination threshold value, false color removal processing is not performed. It is possible to prevent the entire color tone from changing (shifting) by performing false color removal on these pixels.

  When the difference between the average value of the interpolation vector amount CV of the target pixel and the interpolation vector amount CV around the target pixel is equal to or larger than a predetermined false color determination threshold, the result of adjustment to cancel the interpolation process is Since the erasing process is performed, a high-quality color image from which false colors are removed can be obtained.

  When the difference between the average value of the interpolation vector amount CV of the target pixel and the interpolation vector amount CV around the target pixel is equal to or larger than a predetermined false color determination threshold, the interpolation vector amount CV of the target pixel is determined as the target pixel. By making it equal to the average interpolation vector amount of surrounding pixels, the color component generated by the target pixel becomes the same as the color component generated by the background portion of the target pixel, and the color component generated by interpolation processing based on the target pixel Is less noticeable than the surrounding color components. As a result, a high-quality color image from which the false color is removed is obtained.

It is an example of the block diagram of an imaging device. It is a block diagram of an image processing part. It is a figure which shows an example of the interpolation process performed by an image process part, (a) is an interpolation process of R component, (b) is an interpolation process of G component, (c) is an interpolation process of B component, Each is shown. It is a figure which shows the relationship between a color vector and the interpolation vector amount CV. It is a figure for demonstrating the process with respect to an attention pixel. It is a figure for demonstrating the process with respect to an attention pixel, (a) shows the case where an outline cuts the attention pixel vertically, and (b) shows the case where an outline does not exist around an attention pixel, respectively. It is a figure which shows the Bayer arrangement | sequence of the color filter provided in the CCD sensor.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Imaging device 10 ... Lens optical system 11 ... CCD sensor 12 ... CDS
DESCRIPTION OF SYMBOLS 13 ... A / D converter 14 ... Image processing part 141 ... Interpolation processing part 142 ... Interpolation vector amount calculation part 143 ... Interpolation vector amount adjustment part 15 ... Memory 16 ... Control part 17 ... Timing generator 18 ... D / A converter 19 ... Display section 20 ... Operation input section

Claims (2)

An imaging device that generates a color image in which each pixel has the plurality of color components based on a plurality of color mosaic images that are captured by a single-plate image sensor and arranged in a mosaic pattern for each of the plurality of color components. ,
A process of calculating a color interpolation amount for interpolating the color component of the one pixel with respect to an adjacent pixel adjacent to the one pixel and having a color component different from the color component of the color mosaic image corresponding to the one pixel. Color interpolation amount calculation means for each pixel;
Among the pixels corresponding to the color mosaic image of the color component of the target pixel whose color interpolation amount calculated by the color interpolation amount calculation means is equal to or less than a predetermined first threshold, a plurality of pixels within a predetermined range from the target image On the basis of the color interpolation amount average calculating means for calculating the average value of the color interpolation amounts calculated by the color interpolation amount calculating means, and the color interpolation amount for each pixel calculated by the color interpolation amount calculating means, Color interpolation processing means for performing color interpolation processing for each pixel;
Comprising
When the absolute value of the difference between the average value of the color interpolation amount calculated by the color interpolation amount average calculation unit and the color interpolation amount of the target pixel is equal to or greater than a predetermined second threshold, the color interpolation processing unit Does not perform color interpolation processing based on the pixel of interest. When the absolute value of the difference between the average value of the color interpolation amount calculated by the color interpolation amount average calculation unit and the color interpolation amount of the target pixel is equal to or greater than a predetermined second threshold, the color interpolation processing unit The imaging apparatus according to claim 1, wherein the color interpolation amount of the pixel of interest is an average value of color interpolation amounts calculated by the color interpolation amount average calculating unit.

Images